llama.cpp avec Vulkan

farias

Petit test :

# make install
# ldconfig -v
#  llama-bench -m  /models/qwen2.5-1.5b-instruct-q4_k_m.gguf
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = Quadro M5000 (NVIDIA) | uma: 0 | fp16: 0 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: none
ggml_vulkan: 1 = Quadro M4000 (NVIDIA) | uma: 0 | fp16: 0 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| qwen2 1.5B Q4_K - Medium       | 934.69 MiB |     1.54 B | Vulkan     |  -1 |           pp512 |         53.48 ± 0.42 |
| qwen2 1.5B Q4_K - Medium       | 934.69 MiB |     1.54 B | Vulkan     |  -1 |           tg128 |         63.55 ± 0.73 |

build: 5fd2dc2c4 (9721)

farias

Arret de openwebui :

# systemctl stop openwebui
# systemctl disable openwebui
Removed "/etc/systemd/system/multi-user.target.wants/openwebui.service".

Arret de ollama :

# systemctl stop ollama
# systemctl disable ollama
Removed "/etc/systemd/system/default.target.wants/ollama.service".

farias

Test en ligne de commande :

# llama-server -m /models/qwen2.5-1.5b-instruct-q4_k_m.gguf --host 0.0.0.0

farias

Mon fichier service :

# systemctl status llama-server
● llama-server.service - Llama Server
     Loaded: loaded (/etc/systemd/system/llama-server.service; disabled; preset: enabled)
     Active: active (running) since Fri 2026-06-19 17:27:42 UTC; 29s ago
   Main PID: 37413 (llama-server)
      Tasks: 41 (limit: 94224)
     Memory: 91.7M (peak: 91.7M)
        CPU: 3.103s
     CGroup: /system.slice/llama-server.service
             └─37413 /usr/local/bin/llama-server --model /models/qwen2.5-1.5b-instruct-q4_k_m.gguf --host 0.0.0.0 --port 8080

juin 19 17:27:42 XXXX systemd[1]: Started llama-server.service - Llama Server.
# cat /etc/systemd/system/llama-server.service
[Unit]
Description=Llama Server
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/home/XXXX/llama.cpp
Environment="NVM_BIN=/root/.nvm/versions/node/v26.3.1/bin"
Environment="LD_LIBRARY_PATH=:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64"
Environment="VULKAN_VERSION=1.4.350.1"
ExecStart=/usr/local/bin/llama-server \
  --model /models/qwen2.5-1.5b-instruct-q4_k_m.gguf \
  --host 0.0.0.0 --port 8080
Restart=on-failure
RestartSec=5s
StandardOutput=file:/tmp/llama-server.stdout.log
StandardError=file:/tmp/llama-server.stderr.log

[Install]
WantedBy=multi-user.target

farias

Le meilleur modèle semble être https://huggingface.co/Qwen/Qwen3.5-2B pour mes cartes.

farias

https://huggingface.co/unsloth/Qwen3.5-2B-GGUF/resolve/main/Qwen3.5-2B-Q4_0.gguf?download=true

farias

Chargement MMproj :

# curl --output /models/mmproj-BF16.gguf https://huggingface.co/unsloth/Qwen2.5-Omni-7B-GGUF/resolve/main/mmproj-BF16.gguf?download=true 
# ls -l /models/mmproj-BF16.gguf
-rw-r--r-- 1 root root 986 juin  28 16:53 /models/mmproj-BF16.gguf

farias

MMproj pour Quen 3.5 :
https://huggingface.co/unsloth/Qwen3.5-2B-GGUF/blob/main/mmproj-F32.gguf

farias

Nouvelle commande de lancement :

ExecStart=/usr/local/bin/llama-server \
  --model /models/Qwen3.5-2B-Q4_0.gguf \
  --mmproj  /models/mmproj-F32.gguf \
  --host 0.0.0.0 --port 8080

farias

Petit test : 40.81 t/s

NodeBB

llama.cpp avec Vulkan