repo:GitHub - FlashML-org/FreeToken · GitHub实践下载模型注意modelscope要升级到新版本,比如openi里面自带的版本低,下载会报错.pip install modelscope -Umodelscope download --model empero-ai/Qwen3.8-4B-DistillDownloading snapshot of empero-ai/Qwen3.8-4B-Distill (model)…2026-08-22 14:39:01,641 | INFO | modelscope_hub.download | Downloading 14 files from empero-ai/Qwen3.8-4B-DistillmasterDownloading: 100%|███████████████████████████| 14/14 [15:3000:00, 66.49s/file]✓ Snapshot ready at /home/skywalk/.cache/modelscope/models/empero-ai--Qwen3.8-4B-Distill/snapshots/master安装FreeTokenpip install uv uv venv uv pip install freetoken[accel]uv 安装的时候如果慢就加上清华源-i https://pypi.tuna.tsinghua.edu.cn/simple启动ft serve --model ~/models/Qwen3.6-35B-A3B # ft serve --model /home/skywalk/.cache/modelscope/models/empero-ai--Qwen3.8-4B-Distill/snapshots/master/可惜啊,这个显卡太老了,还是跑不起来(root) rootfb98:~# ft serve --model /home/skywalk/.cache/modelscope/models/empero-ai--Qwen3.8-4B-Distill/snapshots/master/ [2026-08-22|15:56:13] INFO Parsed arguments: ServerArgs(model_path/home/skywalk/.cache/modelscope/models/empero-ai--Qwen3.8-4B-Distill/snapshots/master/, tp_infoDistributedInfo(rank0, size1), dtypetorch.bfloat16, max_running_req4, attention_backendauto, moe_backendauto, nvfp4_backendtriton, expert_loadauto, moe_cache_size0, moe_cache_rateNone, moe_cache_autoFalse, kv_reserve_tokens8192, moe_cache_policylru, moe_prefill_overlapTrue, moe_prefill_hit_d2dFalse, moe_collect_statsFalse, moe_cpu_threads0, moe_cpu_layersNone, moe_hybrid_max_fetch-1, cuda_graph_bsNone, cuda_graph_max_bsNone, page_size1, memory_ratio0.9, linear_state_cache_ratio2.0, swa_full_tokens_ratio0.2, swa_num_pages_overrideNone, distributed_timeout60.0, use_dummy_weightFalse, use_pyncclTrue, max_seq_len_overrideNone, num_page_overrideNone, num_token_overrideNone, max_extend_tokens8192, cache_typeradix, offline_modeFalse, decode_log_interval40, special_token_ckptFalse, _unique_suffix.pid5761, server_host127.0.0.1, server_port1919, num_tokenizer0, silent_outputFalse, shell_modeFalse, served_model_namemaster, tool_call_parserqwen3_coder, reasoning_parserqwen3, sampling_defaultsmodel, max_output_tokensNone, enable_cache_reportFalse, cors_originstauri://localhost,http://tauri.localhost,http://localhost:1420) [2026-08-22|15:56:13|FrontendAPI] INFO Default sampling config (sourceframework): temperature0.0, top_k-1, top_p1.0 INFO: Started server process [5761] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://127.0.0.1:1919 (Press CTRLC to quit) /root/.venv/lib/python3.10/site-packages/freetoken/engine/engine.py:967: FutureWarning: torch.cuda._set_allocator_settings is deprecated. Use torch._C._accelerator_setAllocatorSettings instead. torch.cuda.memory._set_allocator_settings(expandable_segments:True) [2026-08-22|15:56:19|core|rank0] INFO Enabled expandable_segments (override via PYTORCH_ALLOC_CONF) /root/.venv/lib/python3.10/site-packages/torch/cuda/__init__.py:180: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 304: OS call failed or operation not supported on this OS (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:119.) return torch._C._cuda_getDeviceCount() 0 [2026-08-22|15:56:19|core|rank0] INFO Auto-selected attention backend: fi [2026-08-22|15:56:19|core|rank0] INFO Resolved config: attention_backendfi, cache_typehybrid_radix, page_size1 [2026-08-22|15:56:19|FrontendAPI] ERROR Backend supervisor: RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 304: OS call failed or operation not supported on this OS Process freetoken-TP0-scheduler: Traceback (most recent call last): File /usr/lib/python3.10/multiprocessing/process.py, line 314, in _bootstrap self.run() File /usr/lib/python3.10/multiprocessing/process.py, line 108, in run self._target(*self._args, **self._kwargs) File /root/.venv/lib/python3.10/site-packages/freetoken/server/launch.py, line 74, in _run_scheduler scheduler Scheduler(args) File /root/.venv/lib/python3.10/site-packages/freetoken/scheduler/scheduler.py, line 64, in __init__ self.engine Engine(config) File /root/.venv/lib/python3.10/site-packages/freetoken/engine/engine.py, line 300, in __init__ torch.cuda.set_device(self.device) File /root/.venv/lib/python3.10/site-packages/torch/cuda/__init__.py, line 638, in set_device torch._C._cuda_setDevice(device) File /root/.venv/lib/python3.10/site-packages/torch/cuda/__init__.py, line 478, in _lazy_init torch._C._cuda_init() RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 304: OS call failed or operation not supported on this OS [2026-08-22|15:56:29|FrontendAPI] ERROR Backend worker is gone and cannot be restarted; stopping the API server INFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [5761] /usr/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown warnings.warn(resource_tracker: There appear to be %d 在openi尝试设置加速镜像export HF_ENDPOINThttps://hf-mirror.com启动ft serve --model empero-ai/Qwen3.8-4B-Distill使用先前下载的模型,模型下载到:/mnt/workspace/.cache/modelscope/models/empero-ai--Qwen3.8-4B-Distill/snapshots/masterft serve --model /mnt/workspace/.cache/modelscope/models/empero-ai--Qwen3.8-4B-Distill/snapshots/master调试碰到报错启动模型报错,使用命令:ft serve --model /mnt/workspace/.cache/modelscope/models/empero-ai--Qwen3.8-4B-Distill/snapshots/master,报错信息:: /usr/local/cuda/bin/nvcc --generate-dependencies-with-compile -MF /root/.cache/flashinfer/0.6.18/80/cached_ops/batch_prefill_with_kv_cache_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_qk_256_head_dim_vo_256_posenc_0_use_swa_False_use_logits_cap_False_f16qk_False/batch_prefill_with_kv_cache_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_qk_256_head_dim_vo_256_posenc_0_use_swa_False_use_logits_cap_False_f16qk_False_batch_prefill_jit_binding.cuda.o.d -DPy_LIMITED_API0x03090000 -D_GLIBCXX_USE_CXX11_ABI1 -I/root/.venv/lib/python3.10/site-packages/flashinfer/data/cccl/cub -I/root/.venv/lib/python3.10/site-packages/flashinfer/data/cccl/libcudacxx/include -I/root/.venv/lib/python3.10/site-packages/flashinfer/data/cccl/thrust -isystem /opt/conda/include/python3.10 -isystem /usr/local/cuda/include -isystem /root/.venv/lib/python3.10/site-packages/tvm_ffi/include -isystem /root/.venv/lib/python3.10/site-packages/tvm_ffi/include -isystem /root/.venv/lib/python3.10/site-packages/flashinfer/data/include -isystem /root/.venv/lib/python3.10/site-packages/flashinfer/data/csrc -isystem /root/.venv/lib/python3.10/site-packages/flashinfer/data/cutlass/include -isystem /root/.venv/lib/python3.10/site-packages/flashinfer/data/cutlass/tools/util/include -isystem /root/.venv/lib/python3.10/site-packages/flashinfer/data/spdlog/include --compiler-options-fPIC --expt-relaxed-constexpr -gencodearchcompute_80,codesm_80 -DFLASHINFER_ENABLE_FP8_E8M0 -DFLASHINFER_ENABLE_FP4_E2M1 -stdc17 --threads1 -use_fast_math -Xfatbin-compress-all --compress-modesize -DFLASHINFER_ENABLE_F16 -DFLASHINFER_ENABLE_BF16 -DFLASHINFER_ENABLE_FP8_E4M3 -DFLASHINFER_ENABLE_FP8_E5M2 -DNDEBUG -O3 -c /root/.cache/flashinfer/0.6.18/80/generated/batch_prefill_with_kv_cache_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_qk_256_head_dim_vo_256_posenc_0_use_swa_False_use_logits_cap_False_f16qk_False/batch_prefill_jit_binding.cu -o /root/.cache/flashinfer/0.6.18/80/cached_ops/batch_prefill_with_kv_cache_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_qk_256_head_dim_vo_256_posenc_0_use_swa_False_use_logits_cap_False_f16qk_False/batch_prefill_with_kv_cache_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_qk_256_head_dim_vo_256_posenc_0_use_swa_False_use_logits_cap_False_f16qk_False_batch_prefill_jit_binding.cuda.onvcc fatal : Unknown option --compress-modesizeninja: build stopped: subcommand failed.[2026-08-30|00:52:01|FrontendAPI] ERROR Backend worker is gone and cannot be restarted; stopping the API serverINFO: Shutting downINFO: Waiting for application shutdown.INFO: Application shutdown complete.INFO: Finished server process [1751]已终止(root) rootc85df02bb3364d1ba93b8f3bdb8fc6ea-task0-0:~# /opt/conda/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdownwarnings.warn(resource_tracker: There appear to be %d 说是cuda版本低,问题我这就是cuda13啊! 但是nvcc确实低rootc85df02bb3364d1ba93b8f3bdb8fc6ea-task0-0:~# nvcc --versionnvcc: NVIDIA (R) Cuda compiler driverCopyright (c) 2005-2023 NVIDIA CorporationBuilt on Mon_Apr__3_17:16:06_PDT_2023Cuda compilation tools, release 12.1, V12.1.105Build cuda_12.1.r12.1/compiler.32688072_0升级所有的pip install pip -U uv pip install uv -U uv pip install freetoken[accel] -U还是不行!