-
Notifications
You must be signed in to change notification settings - Fork 19
Description
Thanks for your work.
I have read through the code, but find nowhere prefetch_experts is called. This function only appears in comments under model directory, where the logic now is just dispatch_local and wait_dispatch_local when it comes to expert dispatching. These two functions utilize CXX impl of NofityFetchStart and WaitHiddenStates, which have nothing to do with expert prefetching. IMHO, it turns scheduling into fetching on demand, rather than lookahead predeiction, and the eviction depends only on visited count.
I can find the codes in python aligned with the description in paper, but these python functions, such as ExpertPredictor.predict, ExpertPrefetcher.prefetch_experts, ExpertCache.gpu_evict etc., are seemingly never called.
Could you explain the prefetching logic a bit now in the code? I will appreciate any help from you.