[Kernels] Remove BatchedTritonOrDeepGemmExperts and default fallback to Triton #29929
Conversation
bnellnm
requested review from
WoosukKwon,
mgoin,
pavanimajety,
robertgshaw2-redhat,
tlrmchlsmth and
yewentao256
as code owners
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request refactors the expert selection logic by removing the BatchedTritonOrDeepGemmExperts wrapper class. The selection logic is now directly implemented in select_gemm_impl, which improves clarity. The new logic correctly handles the selection between BatchedDeepGemmExperts and BatchedTritonExperts based on compatibility and installation status of DeepGEMM. I've found a critical typo in the implementation that would cause a runtime error. Please see the specific comment for details.
vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe.py
Outdated
Show resolved
Hide resolved
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe.py
Outdated
Show resolved
Hide resolved
|
Documentation preview: https://vllm--29929.org.readthedocs.build/en/29929/ |
vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe.py
Outdated
Show resolved
Hide resolved
bnellnm
changed the title
vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe.py
Outdated
Show resolved
Hide resolved
Head branch was pushed to by a user without write access
bnellnm
requested review from
ApostaC,
DarkLight1337,
NickLucche,
aarnphm,
alexm-redhat,
heheda12345,
markmc,
njhill,
noooop,
russellb,
tjtanaa and
ywang96
as code owners
mergify
bot
added
multi-modality
Signed-off-by: Bill Nell <[email protected]>
Signed-off-by: Bill Nell <[email protected]>
Signed-off-by: Bill Nell <[email protected]>
Signed-off-by: Bill Nell <[email protected]>
Signed-off-by: Bill Nell <[email protected]>
Signed-off-by: Bill Nell <[email protected]>
Signed-off-by: Bill Nell <[email protected]>
Signed-off-by: Bill Nell <[email protected]>
Signed-off-by: Bill Nell <[email protected]>
…pressed_tensors_moe.py Co-authored-by: Tyler Michael Smith <[email protected]> Signed-off-by: bnellnm <[email protected]> Signed-off-by: Bill Nell <[email protected]>
Signed-off-by: Bill Nell <[email protected]>
bnellnm
force-pushed
the
delete-btordg
branch
from
f5fd21a to
ef85289
Compare
github-project-automation
bot
moved this from Ready
to Done
in gpt-oss Issues & Enhancements
…to Triton (vllm-project#29929) Signed-off-by: Bill Nell <[email protected]> Signed-off-by: bnellnm <[email protected]> Co-authored-by: Tyler Michael Smith <[email protected]>
…to Triton (vllm-project#29929) Signed-off-by: Bill Nell <[email protected]> Signed-off-by: bnellnm <[email protected]> Co-authored-by: Tyler Michael Smith <[email protected]> Signed-off-by: Xingyu Liu <[email protected]>
2 participants
Purpose
Remove
BatchedTritonOrDeepGemmExperts. Always try to use DeepGEMM if it is installed and the MoE layer is compatible, i.e. fp8 + 128 block quantized). If the MoE layer is compatible and DeepGEMM isn't installed, throw an error. Otherwise, use BatchedTritonExperts.Test Plan
CI
Test Result
cc @tlrmchlsmth , @varun-sundar-rabindranath
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.