QWQ-32B 出错

[csy0225]

I20251111 12:28:58.490254 215011 worker_server.cpp:72] Create worker server with device: 0
F20251111 12:28:58.504206 214989 collective_service.cpp:37] Check failed: status == HCCL_SUCCESS (1 vs. 0) HCCL get root info failed.
*** Check failure stack trace: ***
@ 0x7ccb7f google::LogMessage::SendToLog()
@ 0x7c94e2 google::LogMessage::Flush()
@ 0x7cd289 google::LogMessageFatal::~LogMessageFatal()
@ 0x29206c4 xllm::CollectiveService::CollectiveService()
@ 0xd2502c xllm::DistManager::setup_multi_node_workers()
@ 0xd25d62 xllm::DistManager::DistManager()
@ 0xc58c6f xllm::LLMEngine::setup_workers()
@ 0xc5a892 xllm::LLMEngine::LLMEngine()
@ 0x793791 xllm::Master::Master()
@ 0x782d1c xllm::LLMMaster::LLMMaster()
@ 0x792e48 xllm::create_master()
@ 0x7699bd run()
@ 0x645030 main
@ 0x7f64d9e4da10 (unknown)
@ 0x7f64d9e4dac9 __libc_start_main
@ 0x753e95 _start
@ (nil) (unknown)

[XuZhang99]

try --communication_backend=lccl when launching xllm.

[csy0225]

我用的就是这个诶，是下面这个脚本。
export PYTHON_INCLUDE_PATH="$(python3 -c 'from sysconfig import get_paths; print(get_paths()["include"])')"
export PYTHON_LIB_PATH="$(python3 -c 'from sysconfig import get_paths; print(get_paths()["include"])')"
export PYTORCH_NPU_INSTALL_PATH=/usr/local/libtorch_npu/
export PYTORCH_INSTALL_PATH="$(python3 -c 'import torch, os; print(os.path.dirname(os.path.abspath(torch.file)))')"
export LIBTORCH_ROOT="$(python3 -c 'import torch, os; print(os.path.dirname(os.path.abspath(torch.file)))')"
export LD_LIBRARY_PATH=/usr/local/libtorch_npu/lib:$LD_LIBRARY_PATH

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export ASDOPS_LOG_TO_STDOUT=1
export ASDOPS_LOG_LEVEL=ERROR
export ATB_LOG_TO_STDOUT=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export NPU_MEMORY_FRACTION=0.98
export ATB_WORKSPACE_MEM_ALLOC_ALG_TYPE=3
export ATB_WORKSPACE_MEM_ALLOC_GLOBAL=1

export OMP_NUM_THREADS=12

export HCCL_CONNECT_TIMEOUT=7200
export HCCL_OP_EXPANSION_MODE="AIV"

\rm -rf /root/atb/log/
\rm -rf /root/ascend/log/
\rm -rf core.*

MODEL_PATH="/mnt/qwq-32b/"
MASTER_NODE_ADDR="127.0.0.1:9590"
START_PORT=14830
START_DEVICE=0
LOG_DIR="log"
NNODES=4
WORLD_SIZE=4

export HCCL_IF_BASE_PORT=43439

for (( i=0; i<$NNODES; i++ ))
do
PORT=$((START_PORT + i))
DEVICE=$((START_DEVICE + i))
LOG_FILE="$LOG_DIR/node_$i.log"
./xllm/build/xllm/core/server/xllm
--model $MODEL_PATH
--port $PORT
--devices="npu:$DEVICE"
--master_node_addr=$MASTER_NODE_ADDR
--nnodes=$WORLD_SIZE
--node_rank=$i
--max_memory_utilization=0.9
--max_tokens_per_batch=20000
--max_seqs_per_batch=3000
--block_size=128
--enable_prefix_cache=false
--enable_chunked_prefill=false
--communication_backend="lccl"
--enable_schedule_overlap=true
--enable_mla=false
--dp_size=1
--ep_size=1
> $LOG_FILE 2>&1 &
done

[XuZhang99]

then try to replace lccl with hccl.

[phuole818]

@XuZhang99 @csy0225

别等了，根本没人会理你！
这操作简直给开源社区抹黑——
产品不好好做，功能不认真开发，
天天就知道搞一堆假账号刷数据！