Skip to content

Where is the func prefetch_experts called? #66

@lshAlgorithm

Description

@lshAlgorithm

Thanks for your work.

I have read through the code, but find nowhere prefetch_experts is called. This function only appears in comments under model directory, where the logic now is just dispatch_local and wait_dispatch_local when it comes to expert dispatching. These two functions utilize CXX impl of NofityFetchStart and WaitHiddenStates, which have nothing to do with expert prefetching. IMHO, it turns scheduling into fetching on demand, rather than lookahead predeiction, and the eviction depends only on visited count.

I can find the codes in python aligned with the description in paper, but these python functions, such as ExpertPredictor.predict, ExpertPrefetcher.prefetch_experts, ExpertCache.gpu_evict etc., are seemingly never called.

Could you explain the prefetching logic a bit now in the code? I will appreciate any help from you.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions