Skip to content

[Feature Request] Improve cold start latency with ServerlessLLM sllm_store #65

@drunkcoding

Description

@drunkcoding

Prerequisites

  • I have searched existing issues and reviewed documentation.

Problem Description

Model reading from disk slow, achieve only 2GB/s on 12GB/s SSD

Proposed Solution

  1. Add shm interface in sllm_store
  2. Change model loading pipeline to use state_dict

Alternatives Considered

No response

Additional Context

No response

Importance

Nice to have

Usage Statistics (Optional)

No response

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions