crashed with below error with intel/llm-scaler-vllm:1.0 #154
Description
eap-llm-server-on-arc | INFO: 127.0.0.1:48802 - "GET /health HTTP/1.1" 200 OK
eap-llm-server-on-arc | ERROR 11-08 11:01:29 [multiproc_executor.py:135] Worker proc VllmWorker-3 died unexpectedly, shutting down executor.
eap-llm-server-on-arc | ERROR 11-08 11:01:30 [core.py:502] EngineCore encountered a fatal error.
eap-llm-server-on-arc | ERROR 11-08 11:01:30 [core.py:502] Traceback (most recent call last):
eap-llm-server-on-arc | ERROR 11-08 11:01:30 [core.py:502] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 493, in run_engine_core
eap-llm-server-on-arc | ERROR 11-08 11:01:30 [core.py:502] engine_core.run_busy_loop()
eap-llm-server-on-arc | ERROR 11-08 11:01:30 [core.py:502] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 518, in run_busy_loop
eap-llm-server-on-arc | ERROR 11-08 11:01:30 [core.py:502] self._process_input_queue()
eap-llm-server-on-arc | ERROR 11-08 11:01:30 [core.py:502] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 531, in _process_input_queue
eap-llm-server-on-arc | ERROR 11-08 11:01:30 [core.py:502] self._handle_client_request(*req)
eap-llm-server-on-arc | ERROR 11-08 11:01:30 [core.py:502] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 572, in _handle_client_request
eap-llm-server-on-arc | ERROR 11-08 11:01:30 [core.py:502] raise RuntimeError("Executor failed.")
eap-llm-server-on-arc | ERROR 11-08 11:01:30 [core.py:502] RuntimeError: Executor failed.
eap-llm-server-on-arc | Process EngineCore_0:
eap-llm-server-on-arc | Traceback (most recent call last):
eap-llm-server-on-arc | File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
eap-llm-server-on-arc | self.run()
eap-llm-server-on-arc | File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
eap-llm-server-on-arc | self._target(*self._args, **self._kwargs)
eap-llm-server-on-arc | File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 504, in run_engine_core
eap-llm-server-on-arc | raise e
eap-llm-server-on-arc | File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 493, in run_engine_core
eap-llm-server-on-arc | engine_core.run_busy_loop()
eap-llm-server-on-arc | File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 518, in run_busy_loop
eap-llm-server-on-arc | self._process_input_queue()
eap-llm-server-on-arc | File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 531, in _process_input_queue
eap-llm-server-on-arc | self._handle_client_request(*req)
eap-llm-server-on-arc | File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 572, in _handle_client_request
eap-llm-server-on-arc | raise RuntimeError("Executor failed.")
eap-llm-server-on-arc | RuntimeError: Executor failed.
eap-llm-server-on-arc | ERROR 11-08 11:01:30 [async_llm.py:408] AsyncLLM output_handler failed.
eap-llm-server-on-arc | ERROR 11-08 11:01:30 [async_llm.py:408] Traceback (most recent call last):
eap-llm-server-on-arc | ERROR 11-08 11:01:30 [async_llm.py:408] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/async_llm.py", line 366, in output_handler
eap-llm-server-on-arc | ERROR 11-08 11:01:30 [async_llm.py:408] outputs = await engine_core.get_output_async()
eap-llm-server-on-arc | ERROR 11-08 11:01:30 [async_llm.py:408] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 806, in get_output_async
eap-llm-server-on-arc | ERROR 11-08 11:01:30 [async_llm.py:408] raise self._format_exception(outputs) from None
eap-llm-server-on-arc | ERROR 11-08 11:01:30 [async_llm.py:408] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
eap-llm-server-on-arc | INFO: 127.0.0.1:57446 - "GET /health HTTP/1.1" 200 OK
eap-llm-server-on-arc | INFO: Shutting down
eap-llm-server-on-arc | INFO: Waiting for application shutdown.
eap-llm-server-on-arc | INFO: Application shutdown complete.
eap-llm-server-on-arc | INFO: Finished server process [7]