llama.cpp avec Vulkan

farias

Test de la configuration :

# npx --no node-llama-cpp inspect gpu
OS: Ubuntu 24.04.4 LTS (x64)
Node: 26.3.1 (x64)

node-llama-cpp: 3.18.1
Prebuilt binaries: b8390

ggml_cuda_init: failed to initialize CUDA: CUDA driver version is insufficient for CUDA runtime version
CUDA: available
ggml_cuda_init: failed to initialize CUDA: CUDA driver version is insufficient for CUDA runtime version
Vulkan: available

CUDA devices: Quadro M5000, Quadro M4000
CUDA used VRAM: 0.85% (138.88MB/15.86GB)
CUDA free VRAM: 99.14% (15.73GB/15.86GB)

Vulkan devices: Quadro M5000, Quadro M4000
Vulkan used VRAM: 1.76% (298.13MB/16.48GB)
Vulkan free VRAM: 98.23% (16.19GB/16.48GB)

CPU model: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
Math cores: 32
Used RAM: 16.49% (2.99GB/18.14GB)
Free RAM: 83.5% (15.15GB/18.14GB)
Used swap: 0% (0B/4GB)
Max swap size: 4GB
mmap: supported

farias

Lancement de l’installation :

sudo apt update
sudo apt install curl wget xz-utils

export VULKAN_VERSION="$(curl -fsSL https://vulkan.lunarg.com/sdk/latest/linux.txt)"                                                                                      

echo "Downloading Vulkan SDK version ${VULKAN_VERSION}"
curl --progress-bar "https://sdk.lunarg.com/sdk/download/${VULKAN_VERSION}/linux/vulkan_sdk.tar.xz" -o "/opt/vulkan-sdk.tar.xz"

echo "Installing Vulkan SDK to /opt/vulkan-sdk"
rm -rf "/opt/vulkan-sdk" && mkdir -p "/opt/vulkan-sdk"
tar -Jxf "/opt/vulkan-sdk.tar.xz" --strip-components=1 -C "/opt/vulkan-sdk"
rm -f "/opt/vulkan-sdk.tar.xz"

echo "Adding Vulkan SDK environment variables to shell profiles"
([ ! -f "$HOME/.bashrc" ] || grep -qxF "source /opt/vulkan-sdk/setup-env.sh" "$HOME/.bashrc") || (echo "source /opt/vulkan-sdk/setup-env.sh" >> "$HOME/.bashrc")
([ ! -f "$HOME/.zshrc" ] || grep -qxF "source /opt/vulkan-sdk/setup-env.sh" "$HOME/.zshrc") || (echo "source /opt/vulkan-sdk/setup-env.sh" >> "$HOME/.zshrc")
source /opt/vulkan-sdk/setup-env.sh

farias

Actuellement on est :

Downloading Vulkan SDK version 1.4.350.1

farias

Ajout pour la compilation :

# sudo apt install libvulkan-dev vulkan-tools glslang-tools  cmake build-essential git

farias

Test :

# vulkaninfo | grep -i "deviceName"
'DISPLAY' environment variable not set... skipping surface info
error: XDG_RUNTIME_DIR is invalid or not set in the environment.
	deviceName        = Quadro M5000
	deviceName        = Quadro M4000
	deviceName        = llvmpipe (LLVM 20.1.2, 256 bits)

farias

Tentative de build :

# cmake -B build -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release
-- The C compiler identification is GNU 13.3.0
-- The CXX compiler identification is GNU 13.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMAKE_BUILD_TYPE=Release
-- Found Git: /usr/bin/git (found version "2.43.0") 
-- The ASM compiler identification is GNU
-- Found assembler: /usr/bin/cc
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Warning: ccache not found - consider installing it for faster compilation or disable this warning with GGML_CCACHE=OFF
-- CMAKE_SYSTEM_PROCESSOR: x86_64
-- GGML_SYSTEM_ARCH: x86
-- Found OpenMP_C: -fopenmp (found version "4.5") 
-- Found OpenMP_CXX: -fopenmp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5")  
-- Including CPU backend
-- x86 detected
-- Adding CPU backend variant ggml-cpu: -march=native 
CMake Error at /usr/share/cmake-3.28/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Vulkan (missing: glslc) (found version "1.3.275")
Call Stack (most recent call first):
  /usr/share/cmake-3.28/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-3.28/Modules/FindVulkan.cmake:600 (find_package_handle_standard_args)
  ggml/src/ggml-vulkan/CMakeLists.txt:9 (find_package)


-- Configuring incomplete, errors occurred!

farias

Installation de SDK Vulkan :

# wget -qO- https://packages.lunarg.com/lunarg-signing-key-pub.asc | sudo tee /etc/apt/trusted.gpg.d/lunarg.asc
# sudo wget -qO /etc/apt/sources.list.d/lunarg-vulkan-jammy.list http://packages.lunarg.com/vulkan/lunarg-vulkan-jammy.list
# sudo apt update
# sudo apt install vulkan-sdk

Mais erreur :

# sudo apt install vulkan-sdk
Lecture des listes de paquets... Fait
Construction de l'arbre des dépendances... Fait
Lecture des informations d'état... Fait      
Certains paquets ne peuvent être installés. Ceci peut signifier
que vous avez demandé l'impossible, ou bien, si vous utilisez
la distribution unstable, que certains paquets n'ont pas encore
été créés ou ne sont pas sortis d'Incoming.
L'information suivante devrait vous aider à résoudre la situation : 

Les paquets suivants contiennent des dépendances non satisfaites :
 crashdiagnosticlayer : Dépend: libyaml-cpp0.7 (>= 0.7.0) mais il n'est pas installable
E: Impossible de corriger les problèmes, des paquets défectueux sont en mode « garder en l'état ».

farias

La boulette j’ai pas pris la bonne version… on recommance :

rm  /etc/apt/sources.list.d/lunarg-vulkan-jammy.list 
wget -qO- https://packages.lunarg.com/lunarg-signing-key-pub.asc | sudo tee /etc/apt/trusted.gpg.d/lunarg.asc
sudo wget -qO /etc/apt/sources.list.d/lunarg-vulkan-noble.list http://packages.lunarg.com/vulkan/lunarg-vulkan-noble.list
sudo apt update
sudo apt install vulkan-sdk

farias

Nouveau build :

#  cmake -B build -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release
CMAKE_BUILD_TYPE=Release
-- Warning: ccache not found - consider installing it for faster compilation or disable this warning with GGML_CCACHE=OFF
-- CMAKE_SYSTEM_PROCESSOR: x86_64
-- GGML_SYSTEM_ARCH: x86
-- Including CPU backend
-- x86 detected
-- Adding CPU backend variant ggml-cpu: -march=native 
-- Found Vulkan: /usr/lib/x86_64-linux-gnu/libvulkan.so (found version "1.4.313") found components: glslc glslangValidator 
-- Vulkan found
-- GL_KHR_cooperative_matrix supported by glslc
-- GL_NV_cooperative_matrix2 supported by glslc
-- GL_NV_cooperative_matrix_decode_vector not supported by glslc
-- GL_EXT_integer_dot_product supported by glslc
-- GL_EXT_bfloat16 supported by glslc
-- Including Vulkan backend
-- ggml version: 0.15.2
-- ggml commit:  5fd2dc2c4
-- Found OpenSSL: /usr/lib/x86_64-linux-gnu/libcrypto.so (found version "3.0.13")  
-- Performing Test OPENSSL_VERSION_SUPPORTED
-- Performing Test OPENSSL_VERSION_SUPPORTED - Success
-- OpenSSL found: 3.0.13
-- Generating embedded license file for target: llama-app
-- Configuring done (5.0s)
-- Generating done (0.6s)

farias

La commande pour le build :

# cmake --build build --config Release -j

farias

Petit test :

# make install
# ldconfig -v
#  llama-bench -m  /models/qwen2.5-1.5b-instruct-q4_k_m.gguf
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = Quadro M5000 (NVIDIA) | uma: 0 | fp16: 0 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: none
ggml_vulkan: 1 = Quadro M4000 (NVIDIA) | uma: 0 | fp16: 0 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| qwen2 1.5B Q4_K - Medium       | 934.69 MiB |     1.54 B | Vulkan     |  -1 |           pp512 |         53.48 ± 0.42 |
| qwen2 1.5B Q4_K - Medium       | 934.69 MiB |     1.54 B | Vulkan     |  -1 |           tg128 |         63.55 ± 0.73 |

build: 5fd2dc2c4 (9721)

farias

Arret de openwebui :

# systemctl stop openwebui
# systemctl disable openwebui
Removed "/etc/systemd/system/multi-user.target.wants/openwebui.service".

Arret de ollama :

# systemctl stop ollama
# systemctl disable ollama
Removed "/etc/systemd/system/default.target.wants/ollama.service".

farias

Test en ligne de commande :

# llama-server -m /models/qwen2.5-1.5b-instruct-q4_k_m.gguf --host 0.0.0.0

farias

Mon fichier service :

# systemctl status llama-server
● llama-server.service - Llama Server
     Loaded: loaded (/etc/systemd/system/llama-server.service; disabled; preset: enabled)
     Active: active (running) since Fri 2026-06-19 17:27:42 UTC; 29s ago
   Main PID: 37413 (llama-server)
      Tasks: 41 (limit: 94224)
     Memory: 91.7M (peak: 91.7M)
        CPU: 3.103s
     CGroup: /system.slice/llama-server.service
             └─37413 /usr/local/bin/llama-server --model /models/qwen2.5-1.5b-instruct-q4_k_m.gguf --host 0.0.0.0 --port 8080

juin 19 17:27:42 jellyfin systemd[1]: Started llama-server.service - Llama Server.
root@jellyfin:/home/arias/llama.cpp/build# cat /etc/systemd/system/llama-server.service
[Unit]
Description=Llama Server
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/home/XXXX/llama.cpp
Environment="NVM_BIN=/root/.nvm/versions/node/v26.3.1/bin"
Environment="LD_LIBRARY_PATH=:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64"
Environment="VULKAN_VERSION=1.4.350.1"
ExecStart=/usr/local/bin/llama-server \
  --model /models/qwen2.5-1.5b-instruct-q4_k_m.gguf \
  --host 0.0.0.0 --port 8080
Restart=on-failure
RestartSec=5s
StandardOutput=file:/tmp/llama-server.stdout.log
StandardError=file:/tmp/llama-server.stderr.log

[Install]
WantedBy=multi-user.target

farias

Le meilleur modèle semble être https://huggingface.co/Qwen/Qwen3.5-2B pour mes cartes.

farias

https://huggingface.co/unsloth/Qwen3.5-2B-GGUF/resolve/main/Qwen3.5-2B-Q4_0.gguf?download=true

NodeBB

llama.cpp avec Vulkan