-
Installation de vllm : https://oneuptime.com/blog/post/2026-03-02-how-to-install-and-configure-vllm-on-ubuntu/view
Step 1: Verify GPU Setup
# nvidia-smi Fri Aug 7 11:33:15 2026 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 610.43.03 KMD Version: 610.43.03 CUDA UMD Version: 13.3 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce RTX 3060 ... Off | 00000000:01:00.0 Off | N/A | | N/A 46C P8 15W / 115W | 0MiB / 6144MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA GeForce RTX 5060 Ti Off | 00000000:05:00.0 Off | N/A | | 0% 45C P8 5W / 180W | 0MiB / 16311MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+ # nvcc --version || nvidia-smi | grep "CUDA Version" nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2023 NVIDIA Corporation Built on Fri_Jan__6_16:45:21_PST_2023 Cuda compilation tools, release 12.0, V12.0.140 Build cuda_12.0.r12.0/compiler.32267302_0Step 2: Create Python Environment
# Install Python, venv, curl, and uv sudo apt-get update && sudo apt-get install -y python3 python3-pip python3-venv curl curl -LsSf https://astral.sh/uv/install.sh | sh source "$HOME/.local/bin/env" # Create a dedicated venv uv venv --python 3.12 --seed --managed-python ~/vllm-env source ~/vllm-env/bin/activate # Upgrade pip pip install --upgrade pipStep 3: Install vLLM
# Install vLLM with a PyTorch backend selected for your CUDA driver uv pip install vllm --torch-backend=auto # Verify installation python3 -c "import vllm; print(vllm.__version__)"Step 4: Download a Model
# Install huggingface_hub for model downloading pip install huggingface_hub # Pre-download a model (optional - vLLM downloads on first use) python3 -c " from huggingface_hub import snapshot_download snapshot_download( repo_id='meta-llama/Meta-Llama-3-8B-Instruct', # 8B model, needs 16GB VRAM local_dir='/models/llama-3.2-1b-instruct' ) "Je vais devoir ajouter dans mon config.xml :
host: "127.0.0.1" port: 8080 idle_ttl: 300 models: # Modèle : Nvidia Qwen 3.6 35B NVFP4 poussé à 180k de contexte - name: "nvidia/Qwen3.6-35B-A3B-NVFP4" command: > bash -c "source ~/vllm-env/bin/activate && vllm serve nvidia/Qwen3.6-35B-A3B-NVFP4 --port 8002 --gpu-memory-utilization 0.95 --max-model-len 180000 --max-num-seqs 1 --kv-cache-dtype fp8 --enable-chunked-prefill --max-num-batched-tokens 4096 --gpu-memory-buffer-kv-cache-space 4 --attention-backend flashinfer --reasoning-parser qwen3" ready_url: "http://127.0.0" upstream_url: "http://127.0.0"A suivre.
-
Premier test :
# vllm serve nvidia/Qwen3.6-35B-A3B-NVFP4 --port 8002 --gpu-memory-utilization 0.95 --max-model-len 180000 --max-num-seqs 1 --kv-cache-dtype fp8 --enable-chunked-prefill --max-num-batched-tokens 4096 --gpu-memory-buffer-kv-cache-space 4 --attention-backend flashinfer --reasoning-parser qwen3 WARNING 08-07 12:57:18 [cuda.py:959] Detected different devices in the system: NVIDIA GeForce RTX 3060 Laptop GPU, NVIDIA GeForce RTX 5060 Ti. Please make sure to set `CUDA_DEVICE_ORDER=PCI_BUS_ID` to avoid unexpected behavior. usage: vllm [-h] [-v] {chat,complete,serve,launch,bench,collect-env,run-batch} ... vllm: error: unrecognized arguments: --gpu-memory-buffer-kv-cache-space 4 -
Misère …
# /root/vllm-env/bin/vllm serve nvidia/Qwen3.6-35B-A3B-NVFP4 --port 8002 --gpu-memory-utilization 0.95 --max-model-len 180000 --max-num-seqs 1 --kv-cache-dtype fp8 --enable-chunked-prefill --max-num-batched-tokens 4096 --attention-backend flashinfer --reasoning-parser qwen3 WARNING 08-07 13:29:38 [cuda.py:959] Detected different devices in the system: NVIDIA GeForce RTX 3060 Laptop GPU, NVIDIA GeForce RTX 5060 Ti. Please make sure to set `CUDA_DEVICE_ORDER=PCI_BUS_ID` to avoid unexpected behavior. (APIServer pid=1833126) INFO 08-07 13:29:49 [api_utils.py:345] (APIServer pid=1833126) INFO 08-07 13:29:49 [api_utils.py:345] █ █ █▄ ▄█ (APIServer pid=1833126) INFO 08-07 13:29:49 [api_utils.py:345] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.26.0 (APIServer pid=1833126) INFO 08-07 13:29:49 [api_utils.py:345] █▄█▀ █ █ █ █ model nvidia/Qwen3.6-35B-A3B-NVFP4 (APIServer pid=1833126) INFO 08-07 13:29:49 [api_utils.py:345] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ (APIServer pid=1833126) INFO 08-07 13:29:49 [api_utils.py:345] (APIServer pid=1833126) INFO 08-07 13:29:49 [api_utils.py:273] non-default args: {'model_tag': 'nvidia/Qwen3.6-35B-A3B-NVFP4', 'port': 8002, 'model': 'nvidia/Qwen3.6-35B-A3B-NVFP4', 'max_model_len': 180000, 'attention_backend': 'flashinfer', 'reasoning_parser': 'qwen3', 'gpu_memory_utilization': 0.95, 'kv_cache_dtype': 'fp8', 'max_num_batched_tokens': 4096, 'max_num_seqs': 1, 'enable_chunked_prefill': True} (APIServer pid=1833126) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 58.1k/58.1k [00:00<00:00, 93.7MB/s] preprocessor_config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 390/390 [00:00<00:00, 2.29MB/s] (APIServer pid=1833126) INFO 08-07 13:30:04 [model.py:623] Resolved architecture: Qwen3_5MoeForConditionalGeneration (APIServer pid=1833126) INFO 08-07 13:30:04 [model.py:1788] Using max model len 180000 (APIServer pid=1833126) INFO 08-07 13:30:05 [cache.py:285] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor (APIServer pid=1833126) INFO 08-07 13:30:05 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=4096. (APIServer pid=1833126) WARNING 08-07 13:30:05 [modelopt.py:385] Detected ModelOpt fp8 checkpoint (quant_algo=FP8). Please note that the format is experimental and could change. (APIServer pid=1833126) WARNING 08-07 13:30:05 [modelopt.py:1034] Detected ModelOpt NVFP4 checkpoint (quant_algo=NVFP4). Please note that the format is experimental and could change in future. (APIServer pid=1833126) WARNING 08-07 13:30:05 [modelopt.py:1034] Detected ModelOpt NVFP4 checkpoint (quant_algo=W4A16_NVFP4). Please note that the format is experimental and could change in future. (APIServer pid=1833126) WARNING 08-07 13:30:05 [modelopt.py:1707] Detected ModelOpt MXFP8 checkpoint. Please note that the format is experimental and could change in future. (APIServer pid=1833126) INFO 08-07 13:30:05 [vllm.py:1109] Asynchronous scheduling is enabled. (APIServer pid=1833126) INFO 08-07 13:30:05 [kernel.py:295] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']) tokenizer_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16.7k/16.7k [00:00<00:00, 26.1MB/s] vocab.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6.72M/6.72M [00:00<00:00, 32.2MB/s] tokenizer.json: downloading bytes: ████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5.67MB, 532kB/s tokenizer.json: reconstructing file: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12.8MB / 12.8MB, 1.22MB/s chat_template.jinja: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7.76k/7.76k [00:00<00:00, 23.5MB/s] generation_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 202/202 [00:00<00:00, 998kB/s] video_preprocessor_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 385/385 [00:00<00:00, 1.41MB/s] (APIServer pid=1833126) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`. WARNING 08-07 13:30:34 [cuda.py:959] Detected different devices in the system: NVIDIA GeForce RTX 3060 Laptop GPU, NVIDIA GeForce RTX 5060 Ti. Please make sure to set `CUDA_DEVICE_ORDER=PCI_BUS_ID` to avoid unexpected behavior. (EngineCore pid=1833533) INFO 08-07 13:30:41 [core.py:116] Initializing a V1 LLM engine (v0.26.0) with config: model='nvidia/Qwen3.6-35B-A3B-NVFP4', speculative_config=None, tokenizer='nvidia/Qwen3.6-35B-A3B-NVFP4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=180000, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=modelopt_mixed, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=fp8, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=nvidia/Qwen3.6-35B-A3B-NVFP4, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 2, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto') (EngineCore pid=1833533) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. (EngineCore pid=1833533) INFO 08-07 13:30:45 [parallel_state.py:1615] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.3.2.87:52967 backend=nccl (EngineCore pid=1833533) INFO 08-07 13:30:45 [parallel_state.py:1946] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A (EngineCore pid=1833533) Failed to get device capability: SM 12.x requires CUDA >= 12.9. (EngineCore pid=1833533) Failed to get device capability: SM 12.x requires CUDA >= 12.9. (EngineCore pid=1833533) INFO 08-07 13:30:49 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. (EngineCore pid=1833533) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`. (EngineCore pid=1833533) INFO 08-07 13:31:05 [gpu_model_runner.py:5250] Starting to load model nvidia/Qwen3.6-35B-A3B-NVFP4... (EngineCore pid=1833533) INFO 08-07 13:31:05 [cuda.py:541] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention (EngineCore pid=1833533) INFO 08-07 13:31:05 [mm_encoder_attention.py:373] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention. (EngineCore pid=1833533) INFO 08-07 13:31:05 [__init__.py:635] Selected MarlinFP8ScaledMMLinearKernel for ModelOptFp8LinearMethod (EngineCore pid=1833533) INFO 08-07 13:31:05 [qwen_gdn_linear_attn.py:150] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128). (EngineCore pid=1833533) INFO 08-07 13:31:05 [nvfp4.py:285] Using 'MARLIN' NvFp4 MoE backend out of potential backends: ['FLASHINFER_TRTLLM', 'FLASHINFER_CUTEDSL', 'FLASHINFER_CUTEDSL_BATCHED', 'FLASHINFER_CUTLASS', 'VLLM_CUTLASS', 'MARLIN', 'HUMMING', 'EMULATION']. (EngineCore pid=1833533) INFO 08-07 13:31:05 [cuda.py:422] Using AttentionBackendEnum.FLASHINFER backend. (EngineCore pid=1833533) ERROR 08-07 13:31:07 [gpu_model_runner.py:5345] Failed to load model - not enough GPU memory. Try lowering --gpu-memory-utilization to free memory for weights, increasing --tensor-parallel-size, or using --quantization. See https://docs.vllm.ai/en/latest/configuration/conserving_memory/ for more tips. (original error: CUDA out of memory. Tried to allocate 256.00 MiB. GPU 0 has a total capacity of 15.52 GiB of which 68.62 MiB is free. Including non-PyTorch memory, this process has 15.44 GiB memory in use. Of the allocated memory 15.15 GiB is allocated by PyTorch, and 78.45 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)) -
F farias a déplacé ce sujet de Linux sur
Bonjour ! Vous semblez intéressé par cette conversation, mais vous n’avez pas encore de compte.
Marre de refaire défiler les mêmes messages ? Créez un compte pour retrouver votre position, recevoir des notifications des nouvelles réponses, sauvegarder vos favoris et voter pour les messages que vous appréciez.
Grâce à votre participation, ce message peut devenir encore meilleur 💗
S'inscrire Se connecter