Summary
The /v1/chat/completions and /tokenize endpoints allow a chat_template_kwargs request parameter that is used in the code before it is properly validated against the chat template. With the right chat_template_kwargs parameters, it is possible to block processing of the API server for long periods of time, delaying all other requests
Details
In serving_engine.py, the chat_template_kwargs are unpacked into kwargs passed to chat_utils.py apply_hf_chat_template with no validation on the keys or values in that chat_template_kwargs dict. This means they can be used to override optional parameters in the apply_hf_chat_template method, such as tokenize, changing its default from False to True.
|
request_prompt = apply_hf_chat_template( |
|
tokenizer=tokenizer, |
|
conversation=conversation, |
|
model_config=model_config, |
|
**_chat_template_kwargs, |
|
) |
|
def apply_hf_chat_template( |
|
tokenizer: Union[PreTrainedTokenizer, PreTrainedTokenizerFast], |
|
conversation: list[ConversationMessage], |
|
chat_template: Optional[str], |
|
tools: Optional[list[dict[str, Any]]], |
|
*, |
|
model_config: ModelConfig, |
|
tokenize: bool = False, # Different from HF's default |
|
**kwargs: Any, |
Both serving_chat.py and serving_tokenization.py call into this _preprocess_chat method of serving_engine.py and they both pass in chat_template_kwargs.
So, a chat_template_kwargs like {"tokenize": True} makes tokenization happen as part of applying the chat template, even though that is not expected. Tokenization is a blocking operation, and with sufficiently large input can block the API server's event loop, which blocks handling of all other requests until this tokenization is complete.
This optional tokenize parameter to apply_hf_chat_template does not appear to be used, so one option would be to just hard-code that to always be False instead of allowing it to be optionally overridden by callers. A better option may be to not pass chat_template_kwargs as unpacked kwargs but instead as a dict, and only unpack them after the logic in apply_hf_chat_template that resolves the kwargs against the chat template.
Impact
Any authenticated user can cause a denial of service to a vLLM server with Chat Completion or Tokenize requests.
Fix
#27205
russellb Coordinator
Isotr0py Remediation developer
DarkLight1337 Remediation reviewer
Summary
The /v1/chat/completions and /tokenize endpoints allow a
chat_template_kwargsrequest parameter that is used in the code before it is properly validated against the chat template. With the rightchat_template_kwargsparameters, it is possible to block processing of the API server for long periods of time, delaying all other requestsDetails
In serving_engine.py, the chat_template_kwargs are unpacked into kwargs passed to chat_utils.py
apply_hf_chat_templatewith no validation on the keys or values in that chat_template_kwargs dict. This means they can be used to override optional parameters in theapply_hf_chat_templatemethod, such astokenize, changing its default from False to True.vllm/vllm/entrypoints/openai/serving_engine.py
Lines 809 to 814 in 2a6dc67
vllm/vllm/entrypoints/chat_utils.py
Lines 1602 to 1610 in 2a6dc67
Both serving_chat.py and serving_tokenization.py call into this
_preprocess_chatmethod ofserving_engine.pyand they both pass inchat_template_kwargs.So, a
chat_template_kwargslike{"tokenize": True}makes tokenization happen as part of applying the chat template, even though that is not expected. Tokenization is a blocking operation, and with sufficiently large input can block the API server's event loop, which blocks handling of all other requests until this tokenization is complete.This optional
tokenizeparameter toapply_hf_chat_templatedoes not appear to be used, so one option would be to just hard-code that to always be False instead of allowing it to be optionally overridden by callers. A better option may be to not passchat_template_kwargsas unpacked kwargs but instead as a dict, and only unpack them after the logic inapply_hf_chat_templatethat resolves the kwargs against the chat template.Impact
Any authenticated user can cause a denial of service to a vLLM server with Chat Completion or Tokenize requests.
Fix
#27205