<![CDATA[Tuxedo 17 : Installation vllm]]>Installation de vllm : https://oneuptime.com/blog/post/2026-03-02-how-to-install-and-configure-vllm-on-ubuntu/view

Step 1: Verify GPU Setup

# nvidia-smi
Fri Aug  7 11:33:15 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 610.43.03              KMD Version: 610.43.03     CUDA UMD Version: 13.3     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3060 ...    Off |   00000000:01:00.0 Off |                  N/A |
| N/A   46C    P8             15W /  115W |       0MiB /   6144MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 5060 Ti     Off |   00000000:05:00.0 Off |                  N/A |
|  0%   45C    P8              5W /  180W |       0MiB /  16311MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

# nvcc --version || nvidia-smi | grep "CUDA Version"
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Fri_Jan__6_16:45:21_PST_2023
Cuda compilation tools, release 12.0, V12.0.140
Build cuda_12.0.r12.0/compiler.32267302_0

Step 2: Create Python Environment

# Install Python, venv, curl, and uv
sudo apt-get update && sudo apt-get install -y python3 python3-pip python3-venv curl
curl -LsSf https://astral.sh/uv/install.sh | sh
source "$HOME/.local/bin/env"

# Create a dedicated venv
uv venv --python 3.12 --seed --managed-python ~/vllm-env
source ~/vllm-env/bin/activate

# Upgrade pip
pip install --upgrade pip

Step 3: Install vLLM

# Install vLLM with a PyTorch backend selected for your CUDA driver
uv pip install vllm --torch-backend=auto

# Verify installation
python3 -c "import vllm; print(vllm.__version__)"

Step 4: Download a Model

# Install huggingface_hub for model downloading
pip install huggingface_hub

# Pre-download a model (optional - vLLM downloads on first use)
python3 -c "
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id='meta-llama/Meta-Llama-3-8B-Instruct',  # 8B model, needs 16GB VRAM
    local_dir='/models/llama-3.2-1b-instruct'
)
"

Je vais devoir ajouter dans mon config.xml :

host: "127.0.0.1"
port: 8080
idle_ttl: 300 

models:
  # Modèle : Nvidia Qwen 3.6 35B NVFP4 poussé à 180k de contexte
  - name: "nvidia/Qwen3.6-35B-A3B-NVFP4"
    command: >
      bash -c "source ~/vllm-env/bin/activate && vllm serve nvidia/Qwen3.6-35B-A3B-NVFP4 --port 8002 --gpu-memory-utilization 0.95 --max-model-len 180000 --max-num-seqs 1 --kv-cache-dtype fp8 --enable-chunked-prefill --max-num-batched-tokens 4096 --gpu-memory-buffer-kv-cache-space 4 --attention-backend flashinfer --reasoning-parser qwen3"
    ready_url: "http://127.0.0"
    upstream_url: "http://127.0.0"

A suivre.

]]>https://lemmy.cyber-neurones.org/topic/390/tuxedo-17-installation-vllmRSS for NodeMon, 21 Sep 2026 07:06:49 GMTFri, 07 Aug 2026 09:59:18 GMT60<![CDATA[Reply to Tuxedo 17 : Installation vllm on Fri, 07 Aug 2026 11:32:13 GMT]]>Misère …

# /root/vllm-env/bin/vllm serve nvidia/Qwen3.6-35B-A3B-NVFP4 --port 8002 --gpu-memory-utilization 0.95 --max-model-len 180000 --max-num-seqs 1 --kv-cache-dtype fp8 --enable-chunked-prefill --max-num-batched-tokens 4096 --attention-backend flashinfer --reasoning-parser qwen3

WARNING 08-07 13:29:38 [cuda.py:959] Detected different devices in the system: NVIDIA GeForce RTX 3060 Laptop GPU, NVIDIA GeForce RTX 5060 Ti. Please make sure to set `CUDA_DEVICE_ORDER=PCI_BUS_ID` to avoid unexpected behavior.
(APIServer pid=1833126) INFO 08-07 13:29:49 [api_utils.py:345] 
(APIServer pid=1833126) INFO 08-07 13:29:49 [api_utils.py:345]        █     █     █▄   ▄█
(APIServer pid=1833126) INFO 08-07 13:29:49 [api_utils.py:345]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.26.0
(APIServer pid=1833126) INFO 08-07 13:29:49 [api_utils.py:345]   █▄█▀ █     █     █     █  model   nvidia/Qwen3.6-35B-A3B-NVFP4
(APIServer pid=1833126) INFO 08-07 13:29:49 [api_utils.py:345]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=1833126) INFO 08-07 13:29:49 [api_utils.py:345] 
(APIServer pid=1833126) INFO 08-07 13:29:49 [api_utils.py:273] non-default args: {'model_tag': 'nvidia/Qwen3.6-35B-A3B-NVFP4', 'port': 8002, 'model': 'nvidia/Qwen3.6-35B-A3B-NVFP4', 'max_model_len': 180000, 'attention_backend': 'flashinfer', 'reasoning_parser': 'qwen3', 'gpu_memory_utilization': 0.95, 'kv_cache_dtype': 'fp8', 'max_num_batched_tokens': 4096, 'max_num_seqs': 1, 'enable_chunked_prefill': True}
(APIServer pid=1833126) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 58.1k/58.1k [00:00<00:00, 93.7MB/s]
preprocessor_config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 390/390 [00:00<00:00, 2.29MB/s]
(APIServer pid=1833126) INFO 08-07 13:30:04 [model.py:623] Resolved architecture: Qwen3_5MoeForConditionalGeneration
(APIServer pid=1833126) INFO 08-07 13:30:04 [model.py:1788] Using max model len 180000
(APIServer pid=1833126) INFO 08-07 13:30:05 [cache.py:285] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor
(APIServer pid=1833126) INFO 08-07 13:30:05 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=4096.
(APIServer pid=1833126) WARNING 08-07 13:30:05 [modelopt.py:385] Detected ModelOpt fp8 checkpoint (quant_algo=FP8). Please note that the format is experimental and could change.
(APIServer pid=1833126) WARNING 08-07 13:30:05 [modelopt.py:1034] Detected ModelOpt NVFP4 checkpoint (quant_algo=NVFP4). Please note that the format is experimental and could change in future.
(APIServer pid=1833126) WARNING 08-07 13:30:05 [modelopt.py:1034] Detected ModelOpt NVFP4 checkpoint (quant_algo=W4A16_NVFP4). Please note that the format is experimental and could change in future.
(APIServer pid=1833126) WARNING 08-07 13:30:05 [modelopt.py:1707] Detected ModelOpt MXFP8 checkpoint. Please note that the format is experimental and could change in future.
(APIServer pid=1833126) INFO 08-07 13:30:05 [vllm.py:1109] Asynchronous scheduling is enabled.
(APIServer pid=1833126) INFO 08-07 13:30:05 [kernel.py:295] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
tokenizer_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16.7k/16.7k [00:00<00:00, 26.1MB/s]
vocab.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6.72M/6.72M [00:00<00:00, 32.2MB/s]
tokenizer.json: downloading bytes: ████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5.67MB,  532kB/s  
tokenizer.json: reconstructing file: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12.8MB / 12.8MB, 1.22MB/s  
chat_template.jinja: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7.76k/7.76k [00:00<00:00, 23.5MB/s]
generation_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 202/202 [00:00<00:00, 998kB/s]
video_preprocessor_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 385/385 [00:00<00:00, 1.41MB/s]
(APIServer pid=1833126) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
WARNING 08-07 13:30:34 [cuda.py:959] Detected different devices in the system: NVIDIA GeForce RTX 3060 Laptop GPU, NVIDIA GeForce RTX 5060 Ti. Please make sure to set `CUDA_DEVICE_ORDER=PCI_BUS_ID` to avoid unexpected behavior.
(EngineCore pid=1833533) INFO 08-07 13:30:41 [core.py:116] Initializing a V1 LLM engine (v0.26.0) with config: model='nvidia/Qwen3.6-35B-A3B-NVFP4', speculative_config=None, tokenizer='nvidia/Qwen3.6-35B-A3B-NVFP4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=180000, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=modelopt_mixed, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=fp8, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=nvidia/Qwen3.6-35B-A3B-NVFP4, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 2, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
(EngineCore pid=1833533) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
(EngineCore pid=1833533) INFO 08-07 13:30:45 [parallel_state.py:1615] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.3.2.87:52967 backend=nccl
(EngineCore pid=1833533) INFO 08-07 13:30:45 [parallel_state.py:1946] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(EngineCore pid=1833533) Failed to get device capability: SM 12.x requires CUDA >= 12.9.
(EngineCore pid=1833533) Failed to get device capability: SM 12.x requires CUDA >= 12.9.
(EngineCore pid=1833533) INFO 08-07 13:30:49 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=1833533) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(EngineCore pid=1833533) INFO 08-07 13:31:05 [gpu_model_runner.py:5250] Starting to load model nvidia/Qwen3.6-35B-A3B-NVFP4...
(EngineCore pid=1833533) INFO 08-07 13:31:05 [cuda.py:541] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(EngineCore pid=1833533) INFO 08-07 13:31:05 [mm_encoder_attention.py:373] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(EngineCore pid=1833533) INFO 08-07 13:31:05 [__init__.py:635] Selected MarlinFP8ScaledMMLinearKernel for ModelOptFp8LinearMethod
(EngineCore pid=1833533) INFO 08-07 13:31:05 [qwen_gdn_linear_attn.py:150] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128).
(EngineCore pid=1833533) INFO 08-07 13:31:05 [nvfp4.py:285] Using 'MARLIN' NvFp4 MoE backend out of potential backends: ['FLASHINFER_TRTLLM', 'FLASHINFER_CUTEDSL', 'FLASHINFER_CUTEDSL_BATCHED', 'FLASHINFER_CUTLASS', 'VLLM_CUTLASS', 'MARLIN', 'HUMMING', 'EMULATION'].
(EngineCore pid=1833533) INFO 08-07 13:31:05 [cuda.py:422] Using AttentionBackendEnum.FLASHINFER backend.
(EngineCore pid=1833533) ERROR 08-07 13:31:07 [gpu_model_runner.py:5345] Failed to load model - not enough GPU memory. Try lowering --gpu-memory-utilization to free memory for weights, increasing --tensor-parallel-size, or using --quantization. See https://docs.vllm.ai/en/latest/configuration/conserving_memory/ for more tips. (original error: CUDA out of memory. Tried to allocate 256.00 MiB. GPU 0 has a total capacity of 15.52 GiB of which 68.62 MiB is free. Including non-PyTorch memory, this process has 15.44 GiB memory in use. Of the allocated memory 15.15 GiB is allocated by PyTorch, and 78.45 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf))
]]>https://lemmy.cyber-neurones.org/post/1202https://lemmy.cyber-neurones.org/post/1202Fri, 07 Aug 2026 11:32:13 GMT<![CDATA[Reply to Tuxedo 17 : Installation vllm on Fri, 07 Aug 2026 10:58:33 GMT]]>Premier test :

# vllm serve nvidia/Qwen3.6-35B-A3B-NVFP4 --port 8002 --gpu-memory-utilization 0.95 --max-model-len 180000 --max-num-seqs 1 --kv-cache-dtype fp8 --enable-chunked-prefill --max-num-batched-tokens 4096 --gpu-memory-buffer-kv-cache-space 4 --attention-backend flashinfer --reasoning-parser qwen3
WARNING 08-07 12:57:18 [cuda.py:959] Detected different devices in the system: NVIDIA GeForce RTX 3060 Laptop GPU, NVIDIA GeForce RTX 5060 Ti. Please make sure to set `CUDA_DEVICE_ORDER=PCI_BUS_ID` to avoid unexpected behavior.
usage: vllm [-h] [-v] {chat,complete,serve,launch,bench,collect-env,run-batch} ...
vllm: error: unrecognized arguments: --gpu-memory-buffer-kv-cache-space 4
]]>https://lemmy.cyber-neurones.org/post/1201https://lemmy.cyber-neurones.org/post/1201Fri, 07 Aug 2026 10:58:33 GMT<![CDATA[Reply to Tuxedo 17 : Installation vllm on Fri, 07 Aug 2026 10:06:47 GMT]]>Chargement :

# hf download nvidia/Qwen3.6-35B-A3B-NVFP4 --local-dir /models/
]]>https://lemmy.cyber-neurones.org/post/1199https://lemmy.cyber-neurones.org/post/1199Fri, 07 Aug 2026 10:06:47 GMT