NodeBB

Misère … # /root/vllm-env/bin/vllm serve nvidia/Qwen3.6-35B-A3B-NVFP4 --port 8002 --gpu-memory-utilization 0.95 --max-model-len 180000 --max-num-seqs 1 --kv-cache-dtype fp8 --enable-chunked-prefill --max-num-batched-tokens 4096 --attention-backend flashinfer --reasoning-parser qwen3 WARNING 08-07 13:29:38 [cuda.py:959] Detected different devices in the system: NVIDIA GeForce RTX 3060 Laptop GPU, NVIDIA GeForce RTX 5060 Ti. Please make sure to set `CUDA_DEVICE_ORDER=PCI_BUS_ID` to avoid unexpected behavior. (APIServer pid=1833126) INFO 08-07 13:29:49 [api_utils.py:345] (APIServer pid=1833126) INFO 08-07 13:29:49 [api_utils.py:345] █ █ █▄ ▄█ (APIServer pid=1833126) INFO 08-07 13:29:49 [api_utils.py:345] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.26.0 (APIServer pid=1833126) INFO 08-07 13:29:49 [api_utils.py:345] █▄█▀ █ █ █ █ model nvidia/Qwen3.6-35B-A3B-NVFP4 (APIServer pid=1833126) INFO 08-07 13:29:49 [api_utils.py:345] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ (APIServer pid=1833126) INFO 08-07 13:29:49 [api_utils.py:345] (APIServer pid=1833126) INFO 08-07 13:29:49 [api_utils.py:273] non-default args: {'model_tag': 'nvidia/Qwen3.6-35B-A3B-NVFP4', 'port': 8002, 'model': 'nvidia/Qwen3.6-35B-A3B-NVFP4', 'max_model_len': 180000, 'attention_backend': 'flashinfer', 'reasoning_parser': 'qwen3', 'gpu_memory_utilization': 0.95, 'kv_cache_dtype': 'fp8', 'max_num_batched_tokens': 4096, 'max_num_seqs': 1, 'enable_chunked_prefill': True} (APIServer pid=1833126) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 58.1k/58.1k [00:00<00:00, 93.7MB/s] preprocessor_config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 390/390 [00:00<00:00, 2.29MB/s] (APIServer pid=1833126) INFO 08-07 13:30:04 [model.py:623] Resolved architecture: Qwen3_5MoeForConditionalGeneration (APIServer pid=1833126) INFO 08-07 13:30:04 [model.py:1788] Using max model len 180000 (APIServer pid=1833126) INFO 08-07 13:30:05 [cache.py:285] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor (APIServer pid=1833126) INFO 08-07 13:30:05 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=4096. (APIServer pid=1833126) WARNING 08-07 13:30:05 [modelopt.py:385] Detected ModelOpt fp8 checkpoint (quant_algo=FP8). Please note that the format is experimental and could change. (APIServer pid=1833126) WARNING 08-07 13:30:05 [modelopt.py:1034] Detected ModelOpt NVFP4 checkpoint (quant_algo=NVFP4). Please note that the format is experimental and could change in future. (APIServer pid=1833126) WARNING 08-07 13:30:05 [modelopt.py:1034] Detected ModelOpt NVFP4 checkpoint (quant_algo=W4A16_NVFP4). Please note that the format is experimental and could change in future. (APIServer pid=1833126) WARNING 08-07 13:30:05 [modelopt.py:1707] Detected ModelOpt MXFP8 checkpoint. Please note that the format is experimental and could change in future. (APIServer pid=1833126) INFO 08-07 13:30:05 [vllm.py:1109] Asynchronous scheduling is enabled. (APIServer pid=1833126) INFO 08-07 13:30:05 [kernel.py:295] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']) tokenizer_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16.7k/16.7k [00:00<00:00, 26.1MB/s] vocab.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6.72M/6.72M [00:00<00:00, 32.2MB/s] tokenizer.json: downloading bytes: ████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5.67MB, 532kB/s tokenizer.json: reconstructing file: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12.8MB / 12.8MB, 1.22MB/s chat_template.jinja: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7.76k/7.76k [00:00<00:00, 23.5MB/s] generation_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 202/202 [00:00<00:00, 998kB/s] video_preprocessor_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 385/385 [00:00<00:00, 1.41MB/s] (APIServer pid=1833126) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`. WARNING 08-07 13:30:34 [cuda.py:959] Detected different devices in the system: NVIDIA GeForce RTX 3060 Laptop GPU, NVIDIA GeForce RTX 5060 Ti. Please make sure to set `CUDA_DEVICE_ORDER=PCI_BUS_ID` to avoid unexpected behavior. (EngineCore pid=1833533) INFO 08-07 13:30:41 [core.py:116] Initializing a V1 LLM engine (v0.26.0) with config: model='nvidia/Qwen3.6-35B-A3B-NVFP4', speculative_config=None, tokenizer='nvidia/Qwen3.6-35B-A3B-NVFP4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=180000, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=modelopt_mixed, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=fp8, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=nvidia/Qwen3.6-35B-A3B-NVFP4, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 2, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto') (EngineCore pid=1833533) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. (EngineCore pid=1833533) INFO 08-07 13:30:45 [parallel_state.py:1615] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.3.2.87:52967 backend=nccl (EngineCore pid=1833533) INFO 08-07 13:30:45 [parallel_state.py:1946] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A (EngineCore pid=1833533) Failed to get device capability: SM 12.x requires CUDA >= 12.9. (EngineCore pid=1833533) Failed to get device capability: SM 12.x requires CUDA >= 12.9. (EngineCore pid=1833533) INFO 08-07 13:30:49 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. (EngineCore pid=1833533) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`. (EngineCore pid=1833533) INFO 08-07 13:31:05 [gpu_model_runner.py:5250] Starting to load model nvidia/Qwen3.6-35B-A3B-NVFP4... (EngineCore pid=1833533) INFO 08-07 13:31:05 [cuda.py:541] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention (EngineCore pid=1833533) INFO 08-07 13:31:05 [mm_encoder_attention.py:373] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention. (EngineCore pid=1833533) INFO 08-07 13:31:05 [__init__.py:635] Selected MarlinFP8ScaledMMLinearKernel for ModelOptFp8LinearMethod (EngineCore pid=1833533) INFO 08-07 13:31:05 [qwen_gdn_linear_attn.py:150] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128). (EngineCore pid=1833533) INFO 08-07 13:31:05 [nvfp4.py:285] Using 'MARLIN' NvFp4 MoE backend out of potential backends: ['FLASHINFER_TRTLLM', 'FLASHINFER_CUTEDSL', 'FLASHINFER_CUTEDSL_BATCHED', 'FLASHINFER_CUTLASS', 'VLLM_CUTLASS', 'MARLIN', 'HUMMING', 'EMULATION']. (EngineCore pid=1833533) INFO 08-07 13:31:05 [cuda.py:422] Using AttentionBackendEnum.FLASHINFER backend. (EngineCore pid=1833533) ERROR 08-07 13:31:07 [gpu_model_runner.py:5345] Failed to load model - not enough GPU memory. Try lowering --gpu-memory-utilization to free memory for weights, increasing --tensor-parallel-size, or using --quantization. See https://docs.vllm.ai/en/latest/configuration/conserving_memory/ for more tips. (original error: CUDA out of memory. Tried to allocate 256.00 MiB. GPU 0 has a total capacity of 15.52 GiB of which 68.62 MiB is free. Including non-PyTorch memory, this process has 15.44 GiB memory in use. Of the allocated memory 15.15 GiB is allocated by PyTorch, and 78.45 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf))