标签： LLM

Fedora 43 源码编译安装 llama.cpp 教程（支持 CUDA 显卡加速）

在 Linux 环境下彻底释放显卡的高效推理潜能，源码编译 llama.cpp 带有 CUDA 加速的版本是最佳选择。本文将手把手带你在最新的 Fedora 43 系统（基于 CUDA 13.2 环境）中，从配置环境变量开始，一步步完成源码构建，打造完美契合你显卡算力的私有化大模型部署神器。

1. 配置 CUDA 环境变量

在编译之前，我们需要将 CUDA 的编译器 nvcc 和相关动态库路径写入系统的环境变量中，确保 CMake 能够准确识别。

# 1. 将 CUDA 路径写入全局 shell 配置中
cat >> ~/.bashrc << 'EOF'

# NVIDIA CUDA Toolkit
export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
EOF

# 2. 使环境变量立即生效
source ~/.bashrc

# 3. 验证环境变量是否生效
nvcc --version
which nvcc

提示：如果 nvcc –version 正确打印出了 CUDA 的版本号，说明路径配置成功，可以进行下一步。

2. 获取源码与依赖安装

确保系统里安装了基本的构建工具（cmake 和 gcc-c++）。

# 安装基础编译依赖
sudo dnf install -y cmake gcc-c++ git

# 进入你的开发目录并克隆官方仓库
mkdir -p ~/Dev && cd ~/Dev
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

3. 利用 CMake 执行 CUDA 加速编译

为了让 llama.cpp 在运行时获得最佳性能，我们通过 CMake 指定启用 CUDA，并且将算力精准锁定在 86（对应 RTX 30 系列）。

# 1. 安全起见，先清理可能存在的旧构建缓存
rm -rf build

# 2. 配置 CMake 编译选项
cmake -B build \
      -DGGML_CUDA=ON \
      -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_CUDA_ARCHITECTURES=86 \
      -DGGML_CUDA_F16=ON \
      -DGGML_NATIVE=OFF

# 3. 启动多核并行编译
cmake --build build -j$(nproc)

编译参数解析

-DGGML_CUDA=ON：核心开关，开启 NVIDIA GPU 的 CUDA 计算后端。
-DCMAKE_BUILD_TYPE=Release：开启高级编译器优化，大幅提升运行期推理速度。
-DCMAKE_CUDA_ARCHITECTURES=86：极其关键！ 指定 GPU 架构。RTX 30 系列（如 3060 Ti、3070、3090）填写 86；如果你使用的是 RTX 40 系列，请改为 89。
-DGGML_NATIVE=OFF：关闭纯本地 CPU 强绑定优化，提升编译出的二进制文件在不同 CPU 架构间的兼容性。

4. 验证编译结果

编译完成后，可执行文件会被输出到 build/bin/ 目录下。我们需要验证程序能正常跑起来，且 GPU 加速已成功启用。

4.1 验证基础版本信息

./build/bin/llama-cli --version

如果输出显示了相应的版本号与 Git Commit 标识，说明程序本身编译无误。

4.2 核心验证：检查 CUDA 参数是否就绪

./build/bin/llama-cli -h | grep -E "ngl|gpu|cuda|flash"

运行后如果在输出中看到了如下关键控制参数，说明你的显卡已经可以整装待发了：

  -ngl N, --n-gpu-layers N  number of layers to store in VRAM
  --sm N, --split-mode N    how to split the model across multiple GPUs
  --flash-attn              enable Flash Attention

特别是 -ngl（–n-gpu-layers），这是我们在运行大模型时，用来指定将多少层大模型放进显存（VRAM）中计算的核心参数。

5. 部署大模型服务（llama-server）

编译成功后，llama.cpp 提供了一个非常现代且支持 OpenAI 兼容接口的 Web 服务端程序，叫做 llama-server。我们可以直接用它把 GGUF 模型挂载为 API 服务。

# 启动 API 服务端
./build/bin/llama-server \
  -m /path/to/your-model.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  -ngl 99 \
  --ctx-size 4096

核心启动参数调优说明：

–host 0.0.0.0 和 –port 8080：绑定所有网络接口并监听 8080 端口，不仅本地能调，局域网内的其他设备（如 HomeLab 中的 OpenWebUI）也能直接跨机器访问。
-ngl 99：告诉程序把模型的全部层（写 99 足够覆盖绝大多数 7B/14B 模型）都完全卸载到的显存中。如果模型太大导致显存溢出（OOM），可以适当调低该数值（如 24 或 32）。
–ctx-size 4096：分配 4096 长度的上下文窗口。

启动后如何玩转？

原生 Web 界面：直接在浏览器打开 http://localhost:8080，即可使用 llama.cpp 内置的轻量化聊天面板。
对接第三方客户端：该服务完美兼容 OpenAI API 标准，接口地址为 http://localhost:8080/v1，可无缝接入 OpenWebUI、Cherry Studio 或 NextChat 等主流前端。

编译过程

# 配置 CMake 编译选项
liqixin@fedora:~/Dev/llama.cpp$ cmake -B build -DGGML_CUDA=ON \
      -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_CUDA_ARCHITECTURES=86 \
      -DGGML_CUDA_F16=ON \
      -DGGML_NATIVE=OFF
CMAKE_BUILD_TYPE=Release
-- Warning: ccache not found - consider installing it for faster compilation or disable this warning with GGML_CCACHE=OFF
-- CMAKE_SYSTEM_PROCESSOR: x86_64
-- GGML_SYSTEM_ARCH: x86
-- Including CPU backend
-- x86 detected
-- Adding CPU backend variant ggml-cpu: -msse4.2;-mf16c;-mfma;-mbmi2;-mavx;-mavx2 GGML_SSE42;GGML_F16C;GGML_FMA;GGML_BMI2;GGML_AVX;GGML_AVX2
-- CUDA Toolkit found
-- The CUDA compiler identification is NVIDIA 13.2.78 with host compiler GNU 15.2.1
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Using CMAKE_CUDA_ARCHITECTURES=86 CMAKE_CUDA_ARCHITECTURES_NATIVE=86-real
-- Could NOT find NCCL (missing: NCCL_LIBRARY NCCL_INCLUDE_DIR) 
-- Warning: NCCL not found, performance for multiple CUDA GPUs will be suboptimal
-- CUDA host compiler is GNU 15.2.1
-- Including CUDA backend
-- ggml version: 0.12.0
-- ggml commit:  4f0e43da6
-- Found OpenSSL: /usr/lib64/libcrypto.so (found version "3.5.4")
-- Performing Test OPENSSL_VERSION_SUPPORTED
-- Performing Test OPENSSL_VERSION_SUPPORTED - Success
-- OpenSSL found: 3.5.4
-- UI: derived HF_UI_VERSION=b9282
-- UI: embedded with source: provisioned
-- Generating embedded license file for target: llama-common
-- Configuring done (2.2s)
-- Generating done (0.1s)
-- Build files have been written to: /home/liqixin/Dev/llama.cpp/build
# 执行编译（Build）
liqixin@fedora:~/Dev/llama.cpp$ cmake --build build -j$(nproc)
[  0%] Building/provisioning UI assets (npm build -> HF Bucket fallback)
[  1%] Building C object examples/gguf-hash/CMakeFiles/xxhash.dir/deps/xxhash/xxhash.c.o
[  1%] Building CXX object ggml/src/CMakeFiles/ggml-base.dir/ggml-backend.cpp.o
[  1%] Building CXX object common/CMakeFiles/llama-common-base.dir/build-info.cpp.o
[  1%] Building C object ggml/src/CMakeFiles/ggml-base.dir/ggml.c.o
[  2%] Building C object ggml/src/CMakeFiles/ggml-base.dir/ggml-alloc.c.o
[  2%] Building C object examples/gguf-hash/CMakeFiles/sha256.dir/deps/sha256/sha256.c.o
[  2%] Building CXX object tools/mtmd/CMakeFiles/llama-llava-cli.dir/deprecation-warning.cpp.o
[  2%] Building CXX object ggml/src/CMakeFiles/ggml-base.dir/ggml.cpp.o
[  2%] Building CXX object ggml/src/CMakeFiles/ggml-base.dir/ggml-backend-meta.cpp.o
[  2%] Building CXX object tools/mtmd/CMakeFiles/llama-minicpmv-cli.dir/deprecation-warning.cpp.o
[  2%] Building CXX object ggml/src/CMakeFiles/ggml-base.dir/ggml-opt.cpp.o
[  3%] Building CXX object ggml/src/CMakeFiles/ggml-base.dir/ggml-threading.cpp.o
[  3%] Building C object ggml/src/CMakeFiles/ggml-base.dir/ggml-quants.c.o
[  3%] Building CXX object vendor/cpp-httplib/CMakeFiles/cpp-httplib.dir/httplib.cpp.o
[  3%] Building CXX object tools/mtmd/CMakeFiles/llama-gemma3-cli.dir/deprecation-warning.cpp.o
[  3%] Building CXX object ggml/src/CMakeFiles/ggml-base.dir/gguf.cpp.o
-- UI: npm not found, skipping npm build and trying HF Bucket download
-- UI: downloading assets from version: https://huggingface.co/buckets/ggml-org/llama-ui/resolve/b9282
[  3%] Building C object examples/gguf-hash/CMakeFiles/sha1.dir/deps/sha1/sha1.c.o
[  3%] Building CXX object tools/mtmd/CMakeFiles/llama-qwen2vl-cli.dir/deprecation-warning.cpp.o
[  3%] Built target sha1
[  3%] Linking CXX executable ../../bin/llama-llava-cli
[  3%] Linking CXX executable ../../bin/llama-gemma3-cli
[  3%] Linking CXX executable ../../bin/llama-qwen2vl-cli
[  3%] Built target sha256
[  3%] Linking CXX static library libllama-common-base.a
[  3%] Linking CXX executable ../../bin/llama-minicpmv-cli
[  3%] Built target llama-llava-cli
[  3%] Built target llama-common-base
[  3%] Built target llama-gemma3-cli
[  3%] Built target llama-qwen2vl-cli
[  3%] Built target llama-minicpmv-cli
-- UI: failed to download index.html from version: "HTTP response code said error"
-- UI: downloading assets from latest: https://huggingface.co/buckets/ggml-org/llama-ui/resolve/latest
[  3%] Built target xxhash
-- UI: downloaded index.html
[  3%] Linking CXX shared library ../../bin/libggml-base.so
[  3%] Built target ggml-base
[  4%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/hbm.cpp.o
[  4%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/repack.cpp.o
[  4%] Building C object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/quants.c.o
[  4%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/amx/amx.cpp.o
[  4%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/traits.cpp.o
[  4%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/binary-ops.cpp.o
[  4%] Building C object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/ggml-cpu.c.o
[  5%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/unary-ops.cpp.o
[  4%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/vec.cpp.o
[  5%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/ggml-cpu.cpp.o
[  5%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/ops.cpp.o
[  5%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/llamafile/sgemm.cpp.o
[  5%] Building C object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/arch/x86/quants.c.o
[  5%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/arch/x86/repack.cpp.o
[  5%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/allreduce.cu.o
[  5%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/acc.cu.o
[  5%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/amx/mmq.cpp.o
[  5%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/add-id.cu.o
[  5%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/arange.cu.o
-- UI: downloaded bundle.js
[  5%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/argmax.cu.o
[  5%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/argsort.cu.o
[  6%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/binbcast.cu.o
[  6%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/clamp.cu.o
[  6%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/concat.cu.o
[  6%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/conv-transpose-1d.cu.o
[  6%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/conv2d-dw.cu.o
[  6%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/conv2d-transpose.cu.o
-- UI: downloaded bundle.css
[  6%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/conv2d.cu.o
[  7%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/convert.cu.o
[  7%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/count-equal.cu.o
[  7%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/cpy.cu.o
[  7%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/cross-entropy-loss.cu.o
[  7%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/cumsum.cu.o
[  7%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/diag.cu.o
[  8%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/diagmask.cu.o
-- UI: downloaded loading.html
[  8%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/fattn-tile.cu.o
[  8%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/fattn-wmma-f16.cu.o
[  8%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/fattn.cu.o
[  8%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/fill.cu.o
-- UI: verifying checksums...
-- UI: all checksums verified
-- UI: provisioning complete
[  8%] Generating loading.html.hpp
[  8%] Generating bundle.css.hpp
[  8%] Generating bundle.js.hpp
[  8%] Generating index.html.hpp
[  8%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/getrows.cu.o
[  8%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/gated_delta_net.cu.o
[  9%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/ggml-cuda.cu.o
[  9%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/gla.cu.o
[  9%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/im2col.cu.o
[  9%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/mean.cu.o
[  9%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/mmf.cu.o
[ 10%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/mmid.cu.o
[ 10%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/mmq.cu.o
[ 10%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/mmvf.cu.o
[ 10%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/mmvq.cu.o
[ 10%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/norm.cu.o
[ 11%] Linking CXX shared library ../../bin/libggml-cpu.so
[ 11%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/opt-step-adamw.cu.o
[ 11%] Built target ggml-cpu
[ 11%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/opt-step-sgd.cu.o
[ 11%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/out-prod.cu.o
[ 12%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/pad.cu.o
[ 12%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/pad_reflect_1d.cu.o
[ 13%] Building CXX object tools/ui/CMakeFiles/llama-ui.dir/ui.cpp.o
[ 13%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/pool2d.cu.o
[ 13%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/quantize.cu.o
[ 13%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/roll.cu.o
[ 13%] Linking CXX static library libllama-ui.a
[ 13%] Built target llama-ui
[ 13%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/rope.cu.o
[ 14%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/scale.cu.o
[ 14%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/set-rows.cu.o
[ 14%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/set.cu.o
[ 14%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/snake.cu.o
[ 14%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/softcap.cu.o
[ 14%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/softmax.cu.o
[ 14%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/solve_tri.cu.o
[ 15%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/ssm-conv.cu.o
[ 15%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/ssm-scan.cu.o
[ 15%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/sum.cu.o
[ 15%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/sumrows.cu.o
[ 15%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/top-k.cu.o
[ 15%] Linking CXX static library libcpp-httplib.a
[ 15%] Built target cpp-httplib
[ 15%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/topk-moe.cu.o
[ 15%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/tri.cu.o
[ 16%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/tsembd.cu.o
[ 16%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/unary.cu.o
[ 16%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/upscale.cu.o
[ 16%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/wkv.cu.o
[ 16%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-tile-instance-dkq112-dv112.cu.o
[ 16%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-tile-instance-dkq128-dv128.cu.o
[ 17%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-tile-instance-dkq192-dv128.cu.o
[ 17%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-tile-instance-dkq256-dv256.cu.o
[ 17%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-tile-instance-dkq320-dv256.cu.o
[ 17%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-tile-instance-dkq40-dv40.cu.o
[ 17%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-tile-instance-dkq512-dv512.cu.o
[ 17%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-tile-instance-dkq576-dv512.cu.o
[ 17%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-tile-instance-dkq64-dv64.cu.o
[ 18%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-tile-instance-dkq72-dv72.cu.o
[ 18%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-tile-instance-dkq80-dv80.cu.o
[ 18%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-tile-instance-dkq96-dv96.cu.o
[ 18%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-mma-f16-instance-ncols1_1-ncols2_16.cu.o
[ 18%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-mma-f16-instance-ncols1_1-ncols2_32.cu.o
[ 18%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-mma-f16-instance-ncols1_1-ncols2_8.cu.o
[ 19%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-mma-f16-instance-ncols1_16-ncols2_1.cu.o
[ 19%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-mma-f16-instance-ncols1_16-ncols2_2.cu.o
[ 19%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-mma-f16-instance-ncols1_16-ncols2_4.cu.o
[ 19%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-mma-f16-instance-ncols1_2-ncols2_16.cu.o
[ 19%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-mma-f16-instance-ncols1_2-ncols2_32.cu.o
[ 19%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-mma-f16-instance-ncols1_2-ncols2_4.cu.o
[ 19%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-mma-f16-instance-ncols1_2-ncols2_8.cu.o
[ 20%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-mma-f16-instance-ncols1_32-ncols2_1.cu.o
[ 20%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-mma-f16-instance-ncols1_32-ncols2_2.cu.o
[ 20%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-mma-f16-instance-ncols1_4-ncols2_16.cu.o
[ 20%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-mma-f16-instance-ncols1_4-ncols2_2.cu.o
[ 20%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-mma-f16-instance-ncols1_4-ncols2_4.cu.o
[ 20%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-mma-f16-instance-ncols1_4-ncols2_8.cu.o
[ 20%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-mma-f16-instance-ncols1_64-ncols2_1.cu.o
[ 21%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-mma-f16-instance-ncols1_8-ncols2_1.cu.o
[ 21%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-mma-f16-instance-ncols1_8-ncols2_2.cu.o
[ 21%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-mma-f16-instance-ncols1_8-ncols2_4.cu.o
[ 21%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-mma-f16-instance-ncols1_8-ncols2_8.cu.o
[ 21%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmq-instance-iq1_s.cu.o
[ 21%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmq-instance-iq2_s.cu.o
[ 22%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmq-instance-iq2_xs.cu.o
[ 22%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmq-instance-iq2_xxs.cu.o
[ 22%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmq-instance-iq3_s.cu.o
[ 22%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmq-instance-iq3_xxs.cu.o
[ 22%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmq-instance-iq4_nl.cu.o
[ 22%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmq-instance-iq4_xs.cu.o
[ 22%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmq-instance-mxfp4.cu.o
[ 23%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmq-instance-nvfp4.cu.o
[ 23%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmq-instance-q1_0.cu.o
[ 23%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmq-instance-q2_k.cu.o
[ 23%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmq-instance-q3_k.cu.o
[ 23%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmq-instance-q4_0.cu.o
[ 23%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmq-instance-q4_1.cu.o
[ 24%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmq-instance-q4_k.cu.o
[ 24%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmq-instance-q5_0.cu.o
[ 24%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmq-instance-q5_1.cu.o
[ 24%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmq-instance-q5_k.cu.o
[ 24%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmq-instance-q6_k.cu.o
[ 24%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmq-instance-q8_0.cu.o
[ 24%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmf-instance-ncols_1.cu.o
[ 25%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmf-instance-ncols_10.cu.o
[ 25%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmf-instance-ncols_11.cu.o
[ 25%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmf-instance-ncols_12.cu.o
[ 25%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmf-instance-ncols_13.cu.o
[ 25%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmf-instance-ncols_14.cu.o
[ 25%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmf-instance-ncols_15.cu.o
[ 26%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmf-instance-ncols_16.cu.o
[ 26%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmf-instance-ncols_2.cu.o
[ 26%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmf-instance-ncols_3.cu.o
[ 26%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmf-instance-ncols_4.cu.o
[ 26%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmf-instance-ncols_5.cu.o
[ 26%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmf-instance-ncols_6.cu.o
[ 26%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmf-instance-ncols_7.cu.o
[ 27%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmf-instance-ncols_8.cu.o
[ 27%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmf-instance-ncols_9.cu.o
[ 27%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-vec-instance-f16-f16.cu.o
[ 27%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-vec-instance-q4_0-q4_0.cu.o
[ 27%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-vec-instance-q8_0-q8_0.cu.o
[ 27%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-vec-instance-bf16-bf16.cu.o
[ 27%] Linking CUDA shared library ../../../bin/libggml-cuda.so
[ 27%] Built target ggml-cuda
[ 27%] Building CXX object ggml/src/CMakeFiles/ggml.dir/ggml-backend-dl.cpp.o
[ 27%] Building CXX object ggml/src/CMakeFiles/ggml.dir/ggml-backend-reg.cpp.o
[ 28%] Linking CXX shared library ../../bin/libggml.so
[ 28%] Built target ggml
[ 28%] Building CXX object examples/gguf/CMakeFiles/llama-gguf.dir/gguf.cpp.o
[ 28%] Building CXX object examples/gguf-hash/CMakeFiles/llama-gguf-hash.dir/gguf-hash.cpp.o
[ 28%] Building CXX object src/CMakeFiles/llama.dir/llama.cpp.o
[ 28%] Building CXX object src/CMakeFiles/llama.dir/llama-batch.cpp.o
[ 29%] Building CXX object src/CMakeFiles/llama.dir/llama-context.cpp.o
[ 29%] Building CXX object src/CMakeFiles/llama.dir/llama-chat.cpp.o
[ 29%] Building CXX object src/CMakeFiles/llama.dir/llama-adapter.cpp.o
[ 29%] Building CXX object src/CMakeFiles/llama.dir/llama-grammar.cpp.o
[ 29%] Building CXX object src/CMakeFiles/llama.dir/llama-arch.cpp.o
[ 29%] Building CXX object src/CMakeFiles/llama.dir/llama-graph.cpp.o
[ 29%] Building CXX object src/CMakeFiles/llama.dir/llama-impl.cpp.o
[ 29%] Building CXX object src/CMakeFiles/llama.dir/llama-kv-cache.cpp.o
[ 30%] Building CXX object src/CMakeFiles/llama.dir/llama-kv-cache-iswa.cpp.o
[ 30%] Building CXX object src/CMakeFiles/llama.dir/llama-io.cpp.o
[ 30%] Building CXX object src/CMakeFiles/llama.dir/llama-hparams.cpp.o
[ 30%] Building CXX object src/CMakeFiles/llama.dir/llama-memory.cpp.o
[ 31%] Building CXX object src/CMakeFiles/llama.dir/llama-memory-recurrent.cpp.o
[ 31%] Building CXX object src/CMakeFiles/llama.dir/llama-cparams.cpp.o
[ 31%] Building CXX object src/CMakeFiles/llama.dir/llama-memory-hybrid-iswa.cpp.o
[ 31%] Building CXX object src/CMakeFiles/llama.dir/llama-memory-hybrid.cpp.o
[ 31%] Building CXX object src/CMakeFiles/llama.dir/llama-mmap.cpp.o
[ 31%] Building CXX object src/CMakeFiles/llama.dir/llama-model-loader.cpp.o
[ 31%] Building CXX object src/CMakeFiles/llama.dir/llama-model-saver.cpp.o
[ 31%] Building CXX object src/CMakeFiles/llama.dir/llama-model.cpp.o
[ 31%] Building CXX object src/CMakeFiles/llama.dir/llama-quant.cpp.o
[ 31%] Linking CXX executable ../../bin/llama-gguf-hash
[ 31%] Building CXX object src/CMakeFiles/llama.dir/llama-sampler.cpp.o
[ 32%] Linking CXX executable ../../bin/llama-gguf
[ 33%] Building CXX object src/CMakeFiles/llama.dir/llama-vocab.cpp.o
[ 33%] Built target llama-gguf-hash
[ 33%] Building CXX object src/CMakeFiles/llama.dir/unicode-data.cpp.o
[ 33%] Built target llama-gguf
[ 33%] Building CXX object src/CMakeFiles/llama.dir/unicode.cpp.o
[ 33%] Building CXX object src/CMakeFiles/llama.dir/models/afmoe.cpp.o
[ 33%] Building CXX object src/CMakeFiles/llama.dir/models/apertus.cpp.o
[ 33%] Building CXX object src/CMakeFiles/llama.dir/models/arcee.cpp.o
[ 34%] Building CXX object src/CMakeFiles/llama.dir/models/arctic.cpp.o
[ 34%] Building CXX object src/CMakeFiles/llama.dir/models/arwkv7.cpp.o
[ 34%] Building CXX object src/CMakeFiles/llama.dir/models/baichuan.cpp.o
[ 34%] Building CXX object src/CMakeFiles/llama.dir/models/bailingmoe.cpp.o
[ 34%] Building CXX object src/CMakeFiles/llama.dir/models/bailingmoe2.cpp.o
[ 34%] Building CXX object src/CMakeFiles/llama.dir/models/bert.cpp.o
[ 34%] Building CXX object src/CMakeFiles/llama.dir/models/bitnet.cpp.o
[ 35%] Building CXX object src/CMakeFiles/llama.dir/models/bloom.cpp.o
[ 35%] Building CXX object src/CMakeFiles/llama.dir/models/chameleon.cpp.o
[ 35%] Building CXX object src/CMakeFiles/llama.dir/models/chatglm.cpp.o
[ 35%] Building CXX object src/CMakeFiles/llama.dir/models/codeshell.cpp.o
[ 35%] Building CXX object src/CMakeFiles/llama.dir/models/cogvlm.cpp.o
[ 35%] Building CXX object src/CMakeFiles/llama.dir/models/cohere2.cpp.o
[ 35%] Building CXX object src/CMakeFiles/llama.dir/models/command-r.cpp.o
[ 36%] Building CXX object src/CMakeFiles/llama.dir/models/dbrx.cpp.o
[ 36%] Building CXX object src/CMakeFiles/llama.dir/models/deci.cpp.o
[ 36%] Building CXX object src/CMakeFiles/llama.dir/models/deepseek.cpp.o
[ 36%] Building CXX object src/CMakeFiles/llama.dir/models/deepseek2.cpp.o
[ 36%] Building CXX object src/CMakeFiles/llama.dir/models/deepseek2ocr.cpp.o
[ 36%] Building CXX object src/CMakeFiles/llama.dir/models/delta-net-base.cpp.o
[ 37%] Building CXX object src/CMakeFiles/llama.dir/models/dots1.cpp.o
[ 37%] Building CXX object src/CMakeFiles/llama.dir/models/dream.cpp.o
[ 37%] Building CXX object src/CMakeFiles/llama.dir/models/ernie4-5-moe.cpp.o
[ 37%] Building CXX object src/CMakeFiles/llama.dir/models/ernie4-5.cpp.o
[ 37%] Building CXX object src/CMakeFiles/llama.dir/models/eurobert.cpp.o
[ 37%] Building CXX object src/CMakeFiles/llama.dir/models/exaone-moe.cpp.o
[ 38%] Building CXX object src/CMakeFiles/llama.dir/models/exaone.cpp.o
[ 38%] Building CXX object src/CMakeFiles/llama.dir/models/exaone4.cpp.o
[ 38%] Building CXX object src/CMakeFiles/llama.dir/models/falcon-h1.cpp.o
[ 38%] Building CXX object src/CMakeFiles/llama.dir/models/falcon.cpp.o
[ 38%] Building CXX object src/CMakeFiles/llama.dir/models/gemma-embedding.cpp.o
[ 38%] Building CXX object src/CMakeFiles/llama.dir/models/gemma.cpp.o
[ 38%] Building CXX object src/CMakeFiles/llama.dir/models/gemma2.cpp.o
[ 39%] Building CXX object src/CMakeFiles/llama.dir/models/gemma3.cpp.o
[ 39%] Building CXX object src/CMakeFiles/llama.dir/models/gemma3n.cpp.o
[ 39%] Building CXX object src/CMakeFiles/llama.dir/models/gemma4.cpp.o
[ 39%] Building CXX object src/CMakeFiles/llama.dir/models/glm-dsa.cpp.o
[ 39%] Building CXX object src/CMakeFiles/llama.dir/models/glm4-moe.cpp.o
[ 39%] Building CXX object src/CMakeFiles/llama.dir/models/glm4.cpp.o
[ 39%] Building CXX object src/CMakeFiles/llama.dir/models/gpt2.cpp.o
[ 40%] Building CXX object src/CMakeFiles/llama.dir/models/gptneox.cpp.o
[ 40%] Building CXX object src/CMakeFiles/llama.dir/models/granite-hybrid.cpp.o
[ 40%] Building CXX object src/CMakeFiles/llama.dir/models/granite-moe.cpp.o
[ 40%] Building CXX object src/CMakeFiles/llama.dir/models/granite.cpp.o
[ 40%] Building CXX object src/CMakeFiles/llama.dir/models/grok.cpp.o
[ 40%] Building CXX object src/CMakeFiles/llama.dir/models/grovemoe.cpp.o
[ 40%] Building CXX object src/CMakeFiles/llama.dir/models/hunyuan-dense.cpp.o
[ 41%] Building CXX object src/CMakeFiles/llama.dir/models/hunyuan-moe.cpp.o
[ 41%] Building CXX object src/CMakeFiles/llama.dir/models/hunyuan-vl.cpp.o
[ 41%] Building CXX object src/CMakeFiles/llama.dir/models/internlm2.cpp.o
[ 41%] Building CXX object src/CMakeFiles/llama.dir/models/jais.cpp.o
[ 41%] Building CXX object src/CMakeFiles/llama.dir/models/jais2.cpp.o
[ 41%] Building CXX object src/CMakeFiles/llama.dir/models/jamba.cpp.o
[ 42%] Building CXX object src/CMakeFiles/llama.dir/models/jina-bert-v2.cpp.o
[ 42%] Building CXX object src/CMakeFiles/llama.dir/models/jina-bert-v3.cpp.o
[ 42%] Building CXX object src/CMakeFiles/llama.dir/models/kimi-linear.cpp.o
[ 42%] Building CXX object src/CMakeFiles/llama.dir/models/lfm2.cpp.o
[ 42%] Building CXX object src/CMakeFiles/llama.dir/models/lfm2moe.cpp.o
[ 42%] Building CXX object src/CMakeFiles/llama.dir/models/llada-moe.cpp.o
[ 42%] Building CXX object src/CMakeFiles/llama.dir/models/llada.cpp.o
[ 43%] Building CXX object src/CMakeFiles/llama.dir/models/llama-embed.cpp.o
[ 43%] Building CXX object src/CMakeFiles/llama.dir/models/llama.cpp.o
[ 43%] Building CXX object src/CMakeFiles/llama.dir/models/llama4.cpp.o
[ 43%] Building CXX object src/CMakeFiles/llama.dir/models/maincoder.cpp.o
[ 43%] Building CXX object src/CMakeFiles/llama.dir/models/mamba-base.cpp.o
[ 43%] Building CXX object src/CMakeFiles/llama.dir/models/mamba.cpp.o
[ 44%] Building CXX object src/CMakeFiles/llama.dir/models/mamba2.cpp.o
[ 44%] Building CXX object src/CMakeFiles/llama.dir/models/mimo2.cpp.o
[ 44%] Building CXX object src/CMakeFiles/llama.dir/models/minicpm.cpp.o
[ 44%] Building CXX object src/CMakeFiles/llama.dir/models/minicpm3.cpp.o
[ 44%] Building CXX object src/CMakeFiles/llama.dir/models/minimax-m2.cpp.o
[ 44%] Building CXX object src/CMakeFiles/llama.dir/models/mistral3.cpp.o
[ 44%] Building CXX object src/CMakeFiles/llama.dir/models/mistral4.cpp.o
[ 45%] Building CXX object src/CMakeFiles/llama.dir/models/modern-bert.cpp.o
[ 45%] Building CXX object src/CMakeFiles/llama.dir/models/mpt.cpp.o
[ 45%] Building CXX object src/CMakeFiles/llama.dir/models/nemotron-h-moe.cpp.o
[ 45%] Building CXX object src/CMakeFiles/llama.dir/models/nemotron-h.cpp.o
[ 45%] Building CXX object src/CMakeFiles/llama.dir/models/nemotron.cpp.o
[ 45%] Building CXX object src/CMakeFiles/llama.dir/models/neo-bert.cpp.o
[ 46%] Building CXX object src/CMakeFiles/llama.dir/models/nomic-bert-moe.cpp.o
[ 46%] Building CXX object src/CMakeFiles/llama.dir/models/nomic-bert.cpp.o
[ 46%] Building CXX object src/CMakeFiles/llama.dir/models/olmo.cpp.o
[ 46%] Building CXX object src/CMakeFiles/llama.dir/models/olmo2.cpp.o
[ 46%] Building CXX object src/CMakeFiles/llama.dir/models/olmoe.cpp.o
[ 46%] Building CXX object src/CMakeFiles/llama.dir/models/openai-moe.cpp.o
[ 46%] Building CXX object src/CMakeFiles/llama.dir/models/openelm.cpp.o
[ 47%] Building CXX object src/CMakeFiles/llama.dir/models/orion.cpp.o
[ 47%] Building CXX object src/CMakeFiles/llama.dir/models/paddleocr.cpp.o
[ 47%] Building CXX object src/CMakeFiles/llama.dir/models/pangu-embed.cpp.o
[ 47%] Building CXX object src/CMakeFiles/llama.dir/models/phi2.cpp.o
[ 47%] Building CXX object src/CMakeFiles/llama.dir/models/phi3.cpp.o
[ 47%] Building CXX object src/CMakeFiles/llama.dir/models/phimoe.cpp.o
[ 47%] Building CXX object src/CMakeFiles/llama.dir/models/plamo.cpp.o
[ 48%] Building CXX object src/CMakeFiles/llama.dir/models/plamo2.cpp.o
[ 48%] Building CXX object src/CMakeFiles/llama.dir/models/plamo3.cpp.o
[ 48%] Building CXX object src/CMakeFiles/llama.dir/models/plm.cpp.o
[ 48%] Building CXX object src/CMakeFiles/llama.dir/models/qwen.cpp.o
[ 48%] Building CXX object src/CMakeFiles/llama.dir/models/qwen2.cpp.o
[ 48%] Building CXX object src/CMakeFiles/llama.dir/models/qwen2moe.cpp.o
[ 49%] Building CXX object src/CMakeFiles/llama.dir/models/qwen2vl.cpp.o
[ 49%] Building CXX object src/CMakeFiles/llama.dir/models/qwen3.cpp.o
[ 49%] Building CXX object src/CMakeFiles/llama.dir/models/qwen35.cpp.o
[ 49%] Building CXX object src/CMakeFiles/llama.dir/models/qwen35moe.cpp.o
[ 49%] Building CXX object src/CMakeFiles/llama.dir/models/qwen3moe.cpp.o
[ 49%] Building CXX object src/CMakeFiles/llama.dir/models/qwen3next.cpp.o
[ 49%] Building CXX object src/CMakeFiles/llama.dir/models/qwen3vl.cpp.o
[ 50%] Building CXX object src/CMakeFiles/llama.dir/models/qwen3vlmoe.cpp.o
[ 50%] Building CXX object src/CMakeFiles/llama.dir/models/refact.cpp.o
[ 50%] Building CXX object src/CMakeFiles/llama.dir/models/rnd1.cpp.o
[ 50%] Building CXX object src/CMakeFiles/llama.dir/models/rwkv6-base.cpp.o
[ 50%] Building CXX object src/CMakeFiles/llama.dir/models/rwkv6.cpp.o
[ 50%] Building CXX object src/CMakeFiles/llama.dir/models/rwkv6qwen2.cpp.o
[ 51%] Building CXX object src/CMakeFiles/llama.dir/models/rwkv7-base.cpp.o
[ 51%] Building CXX object src/CMakeFiles/llama.dir/models/rwkv7.cpp.o
[ 51%] Building CXX object src/CMakeFiles/llama.dir/models/seed-oss.cpp.o
[ 51%] Building CXX object src/CMakeFiles/llama.dir/models/smallthinker.cpp.o
[ 51%] Building CXX object src/CMakeFiles/llama.dir/models/smollm3.cpp.o
[ 51%] Building CXX object src/CMakeFiles/llama.dir/models/stablelm.cpp.o
[ 51%] Building CXX object src/CMakeFiles/llama.dir/models/starcoder.cpp.o
[ 52%] Building CXX object src/CMakeFiles/llama.dir/models/starcoder2.cpp.o
[ 52%] Building CXX object src/CMakeFiles/llama.dir/models/step35.cpp.o
[ 52%] Building CXX object src/CMakeFiles/llama.dir/models/t5.cpp.o
[ 52%] Building CXX object src/CMakeFiles/llama.dir/models/wavtokenizer-dec.cpp.o
[ 52%] Building CXX object src/CMakeFiles/llama.dir/models/t5encoder.cpp.o
[ 52%] Building CXX object src/CMakeFiles/llama.dir/models/xverse.cpp.o
[ 53%] Linking CXX shared library ../bin/libllama.so
[ 53%] Built target llama
[ 54%] Building CXX object examples/simple/CMakeFiles/llama-simple.dir/simple.cpp.o
[ 54%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/mtmd-audio.cpp.o
[ 54%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/mtmd-image.cpp.o
[ 55%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/clip.cpp.o
[ 55%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/mtmd-helper.cpp.o
[ 55%] Building CXX object examples/simple-chat/CMakeFiles/llama-simple-chat.dir/simple-chat.cpp.o
[ 55%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/gemma4a.cpp.o
[ 55%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/mtmd.cpp.o
[ 55%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/dotsocr.cpp.o
[ 56%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/glm4v.cpp.o
[ 56%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/gemma4v.cpp.o
[ 56%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/internvl.cpp.o
[ 56%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/cogvlm.cpp.o
[ 56%] Building C object tests/CMakeFiles/test-c.dir/test-c.c.o
[ 56%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/hunyuanvl.cpp.o
[ 56%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/granite-speech.cpp.o
[ 56%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/kimivl.cpp.o
[ 56%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/conformer.cpp.o
[ 56%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/kimik25.cpp.o
[ 56%] Building CXX object common/CMakeFiles/llama-common.dir/arg.cpp.o
[ 57%] Linking C executable ../bin/test-c
[ 57%] Built target test-c
[ 57%] Building CXX object common/CMakeFiles/llama-common.dir/chat-auto-parser-generator.cpp.o
[ 57%] Linking CXX executable ../../bin/llama-simple
[ 57%] Built target llama-simple
[ 58%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/nemotron-v2-vl.cpp.o
[ 58%] Linking CXX executable ../../bin/llama-simple-chat
[ 58%] Built target llama-simple-chat
[ 58%] Building CXX object common/CMakeFiles/llama-common.dir/chat-auto-parser-helpers.cpp.o
[ 58%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/llama4.cpp.o
[ 59%] Building CXX object common/CMakeFiles/llama-common.dir/chat-diff-analyzer.cpp.o
[ 59%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/llava.cpp.o
[ 59%] Building CXX object common/CMakeFiles/llama-common.dir/chat-peg-parser.cpp.o
[ 59%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/minicpmv.cpp.o
[ 59%] Building CXX object common/CMakeFiles/llama-common.dir/chat.cpp.o
[ 59%] Building CXX object common/CMakeFiles/llama-common.dir/common.cpp.o
[ 59%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/paddleocr.cpp.o
[ 59%] Building CXX object common/CMakeFiles/llama-common.dir/console.cpp.o
[ 59%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/pixtral.cpp.o
[ 59%] Building CXX object common/CMakeFiles/llama-common.dir/debug.cpp.o
[ 59%] Building CXX object common/CMakeFiles/llama-common.dir/download.cpp.o
[ 60%] Building CXX object common/CMakeFiles/llama-common.dir/fit.cpp.o
[ 60%] Building CXX object common/CMakeFiles/llama-common.dir/hf-cache.cpp.o
[ 60%] Building CXX object common/CMakeFiles/llama-common.dir/json-partial.cpp.o
[ 60%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/qwen2vl.cpp.o
[ 60%] Building CXX object common/CMakeFiles/llama-common.dir/json-schema-to-grammar.cpp.o
[ 61%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/qwen3vl.cpp.o
[ 61%] Building CXX object common/CMakeFiles/llama-common.dir/llguidance.cpp.o
[ 61%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/mimovl.cpp.o
[ 61%] Building CXX object common/CMakeFiles/llama-common.dir/log.cpp.o
[ 62%] Building CXX object common/CMakeFiles/llama-common.dir/ngram-cache.cpp.o
[ 62%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/qwen3a.cpp.o
[ 62%] Building CXX object common/CMakeFiles/llama-common.dir/ngram-map.cpp.o
[ 62%] Building CXX object common/CMakeFiles/llama-common.dir/ngram-mod.cpp.o
[ 62%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/step3vl.cpp.o
[ 62%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/siglip.cpp.o
[ 62%] Building CXX object common/CMakeFiles/llama-common.dir/peg-parser.cpp.o
[ 62%] Building CXX object common/CMakeFiles/llama-common.dir/preset.cpp.o
[ 62%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/whisper-enc.cpp.o
[ 63%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/deepseekocr.cpp.o
[ 63%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/mobilenetv5.cpp.o
[ 63%] Building CXX object common/CMakeFiles/llama-common.dir/regex-partial.cpp.o
[ 63%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/youtuvl.cpp.o
[ 63%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/yasa2.cpp.o
[ 63%] Building CXX object common/CMakeFiles/llama-common.dir/reasoning-budget.cpp.o
[ 64%] Building CXX object common/CMakeFiles/llama-common.dir/sampling.cpp.o
[ 64%] Building CXX object common/CMakeFiles/llama-common.dir/speculative.cpp.o
[ 64%] Building CXX object common/CMakeFiles/llama-common.dir/unicode.cpp.o
[ 64%] Building CXX object common/CMakeFiles/llama-common.dir/jinja/lexer.cpp.o
[ 64%] Building CXX object common/CMakeFiles/llama-common.dir/jinja/parser.cpp.o
[ 64%] Building CXX object common/CMakeFiles/llama-common.dir/jinja/runtime.cpp.o
[ 64%] Building CXX object common/CMakeFiles/llama-common.dir/jinja/value.cpp.o
[ 65%] Building CXX object common/CMakeFiles/llama-common.dir/jinja/string.cpp.o
[ 65%] Building CXX object common/CMakeFiles/llama-common.dir/jinja/caps.cpp.o
[ 65%] Building CXX object common/CMakeFiles/llama-common.dir/__/license.cpp.o
[ 65%] Linking CXX shared library ../../bin/libmtmd.so
[ 65%] Built target mtmd
[ 65%] Linking CXX shared library ../bin/libllama-common.so
[ 65%] Built target llama-common
[ 66%] Building CXX object tests/CMakeFiles/test-tokenizer-0.dir/test-tokenizer-0.cpp.o
[ 66%] Building CXX object tests/CMakeFiles/test-grammar-parser.dir/test-grammar-parser.cpp.o
[ 66%] Building CXX object tests/CMakeFiles/test-gbnf-validator.dir/test-gbnf-validator.cpp.o
[ 66%] Building CXX object tools/server/CMakeFiles/server-context.dir/server-chat.cpp.o
[ 66%] Building CXX object tests/CMakeFiles/test-llama-grammar.dir/test-llama-grammar.cpp.o
[ 67%] Building CXX object tests/CMakeFiles/test-json-schema-to-grammar.dir/test-json-schema-to-grammar.cpp.o
[ 68%] Building CXX object tests/CMakeFiles/test-quantize-stats.dir/test-quantize-stats.cpp.o
[ 68%] Building CXX object tests/CMakeFiles/test-reasoning-budget.dir/test-reasoning-budget.cpp.o
[ 69%] Building CXX object tests/CMakeFiles/test-chat-peg-parser.dir/test-chat-peg-parser.cpp.o
[ 70%] Building CXX object tests/CMakeFiles/test-jinja.dir/test-jinja.cpp.o
[ 70%] Building CXX object tests/CMakeFiles/test-grammar-integration.dir/test-grammar-integration.cpp.o
[ 70%] Building CXX object tests/CMakeFiles/test-log.dir/test-log.cpp.o
[ 70%] Building CXX object tests/CMakeFiles/test-tokenizer-1-bpe.dir/test-tokenizer-1-bpe.cpp.o
[ 70%] Building CXX object tests/CMakeFiles/test-json-partial.dir/test-json-partial.cpp.o
[ 70%] Building CXX object tests/CMakeFiles/test-llama-archs.dir/test-llama-archs.cpp.o
[ 70%] Building CXX object tests/CMakeFiles/test-sampling.dir/test-sampling.cpp.o
[ 70%] Building CXX object tests/CMakeFiles/test-chat-auto-parser.dir/test-chat-auto-parser.cpp.o
[ 70%] Building CXX object tests/CMakeFiles/test-tokenizer-1-spm.dir/test-tokenizer-1-spm.cpp.o
[ 70%] Building CXX object tests/CMakeFiles/test-peg-parser.dir/test-peg-parser.cpp.o
[ 70%] Building CXX object tests/CMakeFiles/test-chat-template.dir/test-chat-template.cpp.o
[ 70%] Building CXX object tests/CMakeFiles/test-log.dir/get-model.cpp.o
[ 70%] Linking CXX executable ../bin/test-log
[ 70%] Built target test-log
[ 70%] Building CXX object tests/CMakeFiles/test-json-partial.dir/get-model.cpp.o
[ 70%] Building CXX object tests/CMakeFiles/test-reasoning-budget.dir/get-model.cpp.o
[ 70%] Building CXX object tests/CMakeFiles/test-llama-archs.dir/get-model.cpp.o
[ 70%] Linking CXX executable ../bin/test-reasoning-budget
[ 70%] Building CXX object tests/CMakeFiles/test-chat-peg-parser.dir/peg-parser/simple-tokenize.cpp.o
[ 70%] Linking CXX executable ../bin/test-gbnf-validator
[ 70%] Built target test-reasoning-budget
[ 70%] Building CXX object tests/CMakeFiles/test-grammar-integration.dir/get-model.cpp.o
[ 70%] Linking CXX executable ../bin/test-tokenizer-1-bpe
[ 70%] Built target test-gbnf-validator
[ 70%] Building CXX object tests/CMakeFiles/test-json-schema-to-grammar.dir/get-model.cpp.o
[ 70%] Building CXX object tests/CMakeFiles/test-chat-template.dir/get-model.cpp.o
[ 70%] Building CXX object tools/server/CMakeFiles/server-context.dir/server-task.cpp.o
[ 70%] Building CXX object tests/CMakeFiles/test-sampling.dir/get-model.cpp.o
[ 71%] Building CXX object tests/CMakeFiles/test-llama-grammar.dir/get-model.cpp.o
[ 71%] Linking CXX executable ../bin/test-tokenizer-1-spm
[ 71%] Building CXX object tests/CMakeFiles/test-chat-auto-parser.dir/get-model.cpp.o
[ 71%] Building CXX object tests/CMakeFiles/test-peg-parser.dir/peg-parser/simple-tokenize.cpp.o
[ 71%] Built target test-tokenizer-1-bpe
[ 71%] Building CXX object tests/CMakeFiles/test-jinja.dir/get-model.cpp.o
[ 72%] Building CXX object tools/server/CMakeFiles/server-context.dir/server-queue.cpp.o
[ 72%] Building CXX object tests/CMakeFiles/test-grammar-parser.dir/get-model.cpp.o
[ 72%] Building CXX object tests/CMakeFiles/test-chat-peg-parser.dir/get-model.cpp.o
[ 72%] Linking CXX executable ../bin/test-grammar-parser
[ 72%] Built target test-tokenizer-1-spm
[ 72%] Building CXX object tools/server/CMakeFiles/server-context.dir/server-common.cpp.o
[ 72%] Building CXX object tests/CMakeFiles/test-regex-partial.dir/test-regex-partial.cpp.o
[ 72%] Built target test-grammar-parser
[ 72%] Building CXX object tools/server/CMakeFiles/server-context.dir/server-context.cpp.o
[ 72%] Building CXX object tests/CMakeFiles/test-regex-partial.dir/get-model.cpp.o
[ 72%] Linking CXX executable ../bin/test-llama-grammar
[ 72%] Building CXX object tests/CMakeFiles/test-thread-safety.dir/test-thread-safety.cpp.o
[ 72%] Linking CXX executable ../bin/test-tokenizer-0
[ 72%] Linking CXX executable ../bin/test-sampling
[ 72%] Built target test-llama-grammar
[ 72%] Building CXX object tests/CMakeFiles/test-peg-parser.dir/peg-parser/test-basic.cpp.o
[ 72%] Built target test-tokenizer-0
[ 72%] Building CXX object tools/server/CMakeFiles/server-context.dir/server-tools.cpp.o
[ 72%] Built target test-sampling
[ 72%] Building CXX object tests/CMakeFiles/test-thread-safety.dir/get-model.cpp.o
[ 72%] Building CXX object tests/CMakeFiles/test-peg-parser.dir/peg-parser/test-gbnf-generation.cpp.o
[ 72%] Linking CXX executable ../bin/test-llama-archs
[ 72%] Built target test-llama-archs
[ 72%] Building CXX object tests/CMakeFiles/test-arg-parser.dir/test-arg-parser.cpp.o
[ 72%] Linking CXX executable ../bin/test-json-partial
[ 72%] Linking CXX executable ../bin/test-regex-partial
[ 72%] Built target test-json-partial
[ 72%] Building CXX object tests/CMakeFiles/test-peg-parser.dir/peg-parser/test-json-parser.cpp.o
[ 72%] Linking CXX executable ../bin/test-thread-safety
[ 72%] Built target test-regex-partial
[ 72%] Building CXX object tests/CMakeFiles/test-opt.dir/test-opt.cpp.o
[ 72%] Built target test-thread-safety
[ 72%] Building CXX object tests/CMakeFiles/test-arg-parser.dir/get-model.cpp.o
[ 73%] Building CXX object tests/CMakeFiles/test-peg-parser.dir/peg-parser/test-json-serialization.cpp.o
[ 73%] Linking CXX executable ../bin/test-arg-parser
[ 73%] Built target test-arg-parser
[ 73%] Building CXX object tests/CMakeFiles/test-opt.dir/get-model.cpp.o
[ 73%] Building CXX object tests/CMakeFiles/test-peg-parser.dir/peg-parser/test-python-dict-parser.cpp.o
[ 74%] Linking CXX executable ../bin/test-opt
[ 74%] Built target test-opt
[ 74%] Building CXX object tests/CMakeFiles/test-peg-parser.dir/peg-parser/test-unicode.cpp.o
[ 74%] Linking CXX executable ../bin/test-grammar-integration
[ 74%] Linking CXX executable ../bin/test-quantize-stats
[ 74%] Built target test-grammar-integration
[ 74%] Building CXX object tests/CMakeFiles/test-peg-parser.dir/get-model.cpp.o
[ 74%] Built target test-quantize-stats
[ 74%] Building CXX object tests/CMakeFiles/test-gguf.dir/test-gguf.cpp.o
[ 74%] Building CXX object tests/CMakeFiles/test-gguf.dir/get-model.cpp.o
[ 74%] Building CXX object tests/CMakeFiles/test-backend-ops.dir/test-backend-ops.cpp.o
[ 74%] Building CXX object tests/CMakeFiles/test-model-load-cancel.dir/test-model-load-cancel.cpp.o
[ 75%] Building CXX object tests/CMakeFiles/test-model-load-cancel.dir/get-model.cpp.o
[ 75%] Linking CXX executable ../bin/test-model-load-cancel
[ 75%] Built target test-model-load-cancel
[ 75%] Building CXX object tests/CMakeFiles/test-backend-ops.dir/get-model.cpp.o
[ 76%] Building CXX object tests/CMakeFiles/test-autorelease.dir/test-autorelease.cpp.o
[ 76%] Building CXX object tests/CMakeFiles/test-autorelease.dir/get-model.cpp.o
[ 76%] Linking CXX executable ../bin/test-autorelease
[ 76%] Building CXX object tests/CMakeFiles/test-backend-sampler.dir/test-backend-sampler.cpp.o
[ 76%] Built target test-autorelease
[ 76%] Building CXX object tests/CMakeFiles/test-state-restore-fragmented.dir/test-state-restore-fragmented.cpp.o
[ 76%] Building CXX object tests/CMakeFiles/test-recurrent-state-rollback.dir/test-recurrent-state-rollback.cpp.o
[ 76%] Linking CXX executable ../bin/test-gguf
[ 76%] Built target test-gguf
[ 77%] Building CXX object tests/CMakeFiles/test-backend-sampler.dir/get-model.cpp.o
[ 77%] Building CXX object tests/CMakeFiles/test-save-load-state.dir/test-save-load-state.cpp.o
[ 77%] Building CXX object tests/CMakeFiles/test-state-restore-fragmented.dir/get-model.cpp.o
[ 78%] Linking CXX executable ../bin/test-chat-template
[ 78%] Linking CXX executable ../bin/test-state-restore-fragmented
[ 78%] Linking CXX executable ../bin/test-json-schema-to-grammar
[ 78%] Built target test-chat-template
[ 78%] Built target test-state-restore-fragmented
[ 78%] Building CXX object tests/CMakeFiles/test-save-load-state.dir/get-model.cpp.o
[ 79%] Building CXX object tests/CMakeFiles/test-recurrent-state-rollback.dir/get-model.cpp.o
[ 79%] Built target test-json-schema-to-grammar
[ 79%] Building CXX object tests/CMakeFiles/test-quantize-perf.dir/test-quantize-perf.cpp.o
[ 79%] Building CXX object tests/CMakeFiles/test-barrier.dir/test-barrier.cpp.o
[ 79%] Building CXX object tests/CMakeFiles/test-quantize-fns.dir/test-quantize-fns.cpp.o
[ 79%] Linking CXX executable ../bin/test-recurrent-state-rollback
[ 79%] Building CXX object tests/CMakeFiles/test-quantize-perf.dir/get-model.cpp.o
[ 79%] Building CXX object tests/CMakeFiles/test-rope.dir/test-rope.cpp.o
[ 79%] Built target test-recurrent-state-rollback
[ 79%] Building CXX object tests/CMakeFiles/test-barrier.dir/get-model.cpp.o
[ 79%] Building CXX object tests/CMakeFiles/test-rope.dir/get-model.cpp.o
[ 79%] Building CXX object tests/CMakeFiles/test-quantize-fns.dir/get-model.cpp.o
[ 79%] Building C object tests/CMakeFiles/test-mtmd-c-api.dir/test-mtmd-c-api.c.o
[ 79%] Building CXX object tests/CMakeFiles/test-mtmd-c-api.dir/get-model.cpp.o
[ 79%] Linking CXX executable ../bin/test-mtmd-c-api
[ 79%] Linking CXX executable ../bin/test-barrier
[ 79%] Linking CXX executable ../bin/test-quantize-fns
[ 80%] Linking CXX executable ../bin/test-save-load-state
[ 81%] Linking CXX executable ../bin/test-rope
[ 81%] Built target test-mtmd-c-api
[ 82%] Building CXX object tests/CMakeFiles/gguf-model-data.dir/gguf-model-data.cpp.o
[ 82%] Built target test-barrier
[ 82%] Built target test-quantize-fns
[ 82%] Building CXX object tests/CMakeFiles/test-alloc.dir/test-alloc.cpp.o
[ 82%] Building CXX object examples/batched/CMakeFiles/llama-batched.dir/batched.cpp.o
[ 82%] Built target test-rope
[ 82%] Building CXX object tests/CMakeFiles/test-alloc.dir/get-model.cpp.o
[ 82%] Built target test-save-load-state
[ 82%] Building CXX object examples/debug/CMakeFiles/llama-debug.dir/debug.cpp.o
[ 82%] Building CXX object examples/embedding/CMakeFiles/llama-embedding.dir/embedding.cpp.o
[ 82%] Linking CXX executable ../bin/test-backend-sampler
[ 82%] Built target test-backend-sampler
[ 82%] Building CXX object examples/eval-callback/CMakeFiles/llama-eval-callback.dir/eval-callback.cpp.o
[ 82%] Building CXX object examples/idle/CMakeFiles/llama-idle.dir/idle.cpp.o
[ 82%] Building CXX object examples/lookahead/CMakeFiles/llama-lookahead.dir/lookahead.cpp.o
[ 83%] Building CXX object examples/lookup/CMakeFiles/llama-lookup.dir/lookup.cpp.o
[ 83%] Linking CXX executable ../bin/test-quantize-perf
[ 83%] Linking CXX executable ../bin/test-alloc
[ 83%] Built target test-quantize-perf
[ 83%] Building CXX object examples/lookup/CMakeFiles/llama-lookup-create.dir/lookup-create.cpp.o
[ 83%] Built target test-alloc
[ 83%] Building CXX object examples/lookup/CMakeFiles/llama-lookup-merge.dir/lookup-merge.cpp.o
[ 83%] Linking CXX executable ../../bin/llama-batched
[ 83%] Built target llama-batched
[ 83%] Building CXX object examples/lookup/CMakeFiles/llama-lookup-stats.dir/lookup-stats.cpp.o
[ 83%] Building CXX object examples/parallel/CMakeFiles/llama-parallel.dir/parallel.cpp.o
[ 83%] Linking CXX executable ../../bin/llama-eval-callback
[ 83%] Linking CXX executable ../bin/test-peg-parser
[ 83%] Built target llama-eval-callback
[ 83%] Linking CXX executable ../../bin/llama-embedding
[ 83%] Building CXX object examples/passkey/CMakeFiles/llama-passkey.dir/passkey.cpp.o
[ 83%] Linking CXX executable ../../bin/llama-idle
[ 83%] Built target test-peg-parser
[ 83%] Building CXX object examples/retrieval/CMakeFiles/llama-retrieval.dir/retrieval.cpp.o
[ 83%] Linking CXX executable ../../bin/llama-lookup-merge
[ 83%] Built target llama-embedding
[ 83%] Built target llama-idle
[ 83%] Building CXX object examples/speculative/CMakeFiles/llama-speculative.dir/speculative.cpp.o
[ 84%] Building CXX object examples/speculative-simple/CMakeFiles/llama-speculative-simple.dir/speculative-simple.cpp.o
[ 84%] Built target llama-lookup-merge
[ 84%] Building CXX object examples/gen-docs/CMakeFiles/llama-gen-docs.dir/gen-docs.cpp.o
[ 84%] Linking CXX executable ../../bin/llama-lookup
[ 84%] Linking CXX executable ../../bin/llama-lookahead
[ 84%] Built target llama-lookup
[ 84%] Building CXX object examples/training/CMakeFiles/llama-finetune.dir/finetune.cpp.o
[ 84%] Linking CXX executable ../../bin/llama-lookup-create
[ 84%] Built target llama-lookahead
[ 84%] Building CXX object examples/diffusion/CMakeFiles/llama-diffusion.dir/diffusion.cpp.o
[ 84%] Built target llama-lookup-create
[ 84%] Building CXX object examples/convert-llama2c-to-ggml/CMakeFiles/llama-convert-llama2c-to-ggml.dir/convert-llama2c-to-ggml.cpp.o
[ 85%] Linking CXX executable ../../bin/llama-lookup-stats
[ 85%] Built target llama-lookup-stats
[ 85%] Building CXX object pocs/vdot/CMakeFiles/llama-vdot.dir/vdot.cpp.o
[ 85%] Linking CXX executable ../../bin/llama-parallel
[ 85%] Linking CXX executable ../../bin/llama-passkey
[ 85%] Built target llama-parallel
[ 85%] Built target llama-passkey
[ 85%] Building CXX object pocs/vdot/CMakeFiles/llama-q8dot.dir/q8dot.cpp.o
[ 86%] Building CXX object tools/batched-bench/CMakeFiles/llama-batched-bench-impl.dir/batched-bench.cpp.o
[ 86%] Linking CXX executable ../../bin/llama-gen-docs
[ 86%] Built target llama-gen-docs
[ 86%] Building CXX object tools/gguf-split/CMakeFiles/llama-gguf-split.dir/gguf-split.cpp.o
[ 86%] Linking CXX executable ../../bin/llama-finetune
[ 87%] Linking CXX static library libllama-diffusion.a
[ 87%] Linking CXX executable ../../bin/llama-vdot
[ 87%] Built target llama-diffusion
[ 88%] Building CXX object tools/imatrix/CMakeFiles/llama-imatrix.dir/imatrix.cpp.o
[ 89%] Linking CXX executable ../../bin/llama-retrieval
[ 89%] Built target llama-finetune
[ 89%] Building CXX object tools/llama-bench/CMakeFiles/llama-bench-impl.dir/llama-bench.cpp.o
[ 89%] Built target llama-vdot
[ 89%] Building CXX object tools/completion/CMakeFiles/llama-completion-impl.dir/completion.cpp.o
[ 89%] Linking CXX executable ../../bin/llama-q8dot
[ 89%] Built target llama-retrieval
[ 90%] Building CXX object tools/perplexity/CMakeFiles/llama-perplexity-impl.dir/perplexity.cpp.o
[ 90%] Building CXX object tools/quantize/CMakeFiles/llama-quantize-impl.dir/quantize.cpp.o
[ 90%] Built target llama-q8dot
[ 90%] Building CXX object tools/tokenize/CMakeFiles/llama-tokenize.dir/tokenize.cpp.o
[ 90%] Linking CXX executable ../../bin/llama-speculative-simple
[ 90%] Built target llama-speculative-simple
[ 90%] Building CXX object tools/parser/CMakeFiles/llama-debug-template-parser.dir/debug-template-parser.cpp.o
[ 90%] Linking CXX executable ../../bin/llama-convert-llama2c-to-ggml
[ 90%] Built target llama-convert-llama2c-to-ggml
[ 90%] Linking CXX shared library ../../bin/libllama-batched-bench-impl.so
[ 90%] Building CXX object tools/parser/CMakeFiles/llama-template-analysis.dir/template-analysis.cpp.o
[ 90%] Linking CXX executable ../../bin/llama-gguf-split
[ 90%] Built target llama-batched-bench-impl
[ 90%] Building CXX object tools/tts/CMakeFiles/llama-tts.dir/tts.cpp.o
[ 90%] Built target llama-gguf-split
[ 90%] Linking CXX executable ../../bin/llama-tokenize
[ 90%] Building CXX object tools/mtmd/CMakeFiles/llama-mtmd-cli.dir/mtmd-cli.cpp.o
[ 90%] Built target llama-tokenize
[ 90%] Building CXX object tools/mtmd/CMakeFiles/llama-mtmd-debug.dir/debug/mtmd-debug.cpp.o
[ 90%] Linking CXX executable ../../bin/llama-debug
[ 90%] Built target llama-debug
[ 90%] Building CXX object tools/cvector-generator/CMakeFiles/llama-cvector-generator.dir/cvector-generator.cpp.o
[ 90%] Linking CXX executable ../../bin/llama-speculative
[ 90%] Built target llama-speculative
[ 91%] Building CXX object tools/export-lora/CMakeFiles/llama-export-lora.dir/export-lora.cpp.o
[ 92%] Linking CXX executable ../../bin/llama-mtmd-debug
[ 92%] Built target llama-mtmd-debug
[ 93%] Linking CXX shared library ../../bin/libllama-quantize-impl.so
[ 94%] Building CXX object tools/fit-params/CMakeFiles/llama-fit-params-impl.dir/fit-params.cpp.o
[ 94%] Built target llama-quantize-impl
[ 94%] Building CXX object tools/results/CMakeFiles/llama-results.dir/results.cpp.o
[ 94%] Linking CXX static library libgguf-model-data.a
[ 94%] Built target gguf-model-data
[ 94%] Building CXX object examples/diffusion/CMakeFiles/llama-diffusion-cli.dir/diffusion-cli.cpp.o
[ 94%] Linking CXX executable ../../bin/llama-export-lora
[ 95%] Linking CXX executable ../../bin/llama-cvector-generator
[ 95%] Building CXX object tools/batched-bench/CMakeFiles/llama-batched-bench.dir/main.cpp.o
[ 95%] Linking CXX executable ../../bin/llama-mtmd-cli
[ 95%] Linking CXX executable ../../bin/llama-batched-bench
[ 95%] Built target llama-export-lora
[ 95%] Building CXX object tools/quantize/CMakeFiles/llama-quantize.dir/main.cpp.o
[ 95%] Built target llama-cvector-generator
[ 95%] Building CXX object tests/CMakeFiles/test-gguf-model-data.dir/test-gguf-model-data.cpp.o
[ 95%] Linking CXX executable ../../bin/llama-quantize
[ 95%] Built target llama-mtmd-cli
[ 95%] Built target llama-batched-bench
[ 95%] Building CXX object tests/CMakeFiles/test-quant-type-selection.dir/test-quant-type-selection.cpp.o
[ 95%] Building CXX object tests/CMakeFiles/export-graph-ops.dir/export-graph-ops.cpp.o
[ 95%] Linking CXX shared library ../../bin/libllama-fit-params-impl.so
[ 95%] Built target llama-quantize
[ 95%] Building CXX object tests/CMakeFiles/test-quant-type-selection.dir/get-model.cpp.o
[ 95%] Linking CXX executable ../bin/test-chat-auto-parser
[ 95%] Built target llama-fit-params-impl
[ 95%] Building CXX object tools/fit-params/CMakeFiles/llama-fit-params.dir/main.cpp.o
[ 95%] Linking CXX executable ../../bin/llama-fit-params
[ 95%] Linking CXX shared library ../../bin/libllama-completion-impl.so
[ 95%] Built target test-chat-auto-parser
[ 96%] Linking CXX executable ../bin/test-gguf-model-data
[ 96%] Linking CXX executable ../../bin/llama-results
[ 96%] Built target llama-fit-params
[ 96%] Built target llama-completion-impl
[ 97%] Building CXX object tools/completion/CMakeFiles/llama-completion.dir/main.cpp.o
[ 97%] Linking CXX executable ../../bin/llama-completion
[ 97%] Built target llama-results
[ 97%] Built target test-gguf-model-data
[ 97%] Built target llama-completion
[ 97%] Linking CXX shared library ../../bin/libllama-perplexity-impl.so
[ 97%] Linking CXX executable ../../bin/llama-diffusion-cli
[ 97%] Linking CXX executable ../bin/export-graph-ops
[ 97%] Built target llama-perplexity-impl
[ 97%] Building CXX object tools/perplexity/CMakeFiles/llama-perplexity.dir/main.cpp.o
[ 97%] Linking CXX executable ../../bin/llama-perplexity
[ 98%] Linking CXX executable ../bin/test-quant-type-selection
[ 98%] Built target llama-diffusion-cli
[ 98%] Built target export-graph-ops
[ 98%] Built target test-quant-type-selection
[ 98%] Built target llama-perplexity
[ 98%] Linking CXX executable ../bin/test-jinja
[ 98%] Linking CXX executable ../../bin/llama-debug-template-parser
[ 98%] Built target test-jinja
[ 98%] Built target llama-debug-template-parser
[ 98%] Linking CXX executable ../../bin/llama-template-analysis
[ 98%] Linking CXX executable ../bin/test-chat-peg-parser
[ 98%] Built target llama-template-analysis
[ 98%] Built target test-chat-peg-parser
[ 98%] Linking CXX executable ../../bin/llama-imatrix
[ 98%] Built target llama-imatrix
[ 98%] Linking CXX static library libserver-context.a
[ 98%] Built target server-context
[ 98%] Building CXX object tools/server/CMakeFiles/llama-server-impl.dir/server-models.cpp.o
[ 98%] Building CXX object tools/server/CMakeFiles/llama-server-impl.dir/server.cpp.o
[ 98%] Building CXX object tools/server/CMakeFiles/llama-server-impl.dir/server-http.cpp.o
[ 98%] Building CXX object tools/cli/CMakeFiles/llama-cli-impl.dir/cli.cpp.o
[ 98%] Building CXX object tests/CMakeFiles/test-chat.dir/get-model.cpp.o
[ 98%] Building CXX object tests/CMakeFiles/test-chat.dir/test-chat.cpp.o
[ 99%] Linking CXX executable ../../bin/llama-tts
[ 99%] Linking CXX shared library ../../bin/libllama-bench-impl.so
[ 99%] Built target llama-tts
[ 99%] Built target llama-bench-impl
[ 99%] Building CXX object tools/llama-bench/CMakeFiles/llama-bench.dir/main.cpp.o
[ 99%] Linking CXX executable ../../bin/llama-bench
[ 99%] Built target llama-bench
[ 99%] Linking CXX executable ../bin/test-backend-ops
[ 99%] Built target test-backend-ops
[ 99%] Linking CXX shared library ../../bin/libllama-cli-impl.so
[ 99%] Built target llama-cli-impl
[ 99%] Building CXX object tools/cli/CMakeFiles/llama-cli.dir/main.cpp.o
[100%] Linking CXX executable ../../bin/llama-cli
[100%] Built target llama-cli
[100%] Linking CXX shared library ../../bin/libllama-server-impl.so
[100%] Built target llama-server-impl
[100%] Building CXX object tools/server/CMakeFiles/llama-server.dir/main.cpp.o
[100%] Building CXX object app/CMakeFiles/llama-app.dir/llama.cpp.o
[100%] Linking CXX executable ../../bin/llama-server
[100%] Built target llama-server
[100%] Linking CXX executable ../bin/llama
[100%] Built target llama-app
[100%] Linking CXX executable ../bin/test-chat
[100%] Built target test-chat
# 查看版本
liqixin@fedora:~/Dev/llama.cpp$ ./build/bin/llama-cli --version
version: 9282 (4f0e43da6)
built with GNU 15.2.1 for Linux x86_64
# 查看是否支持 GPU
liqixin@fedora:~/Dev/llama.cpp$ ./build/bin/llama-cli -h | grep -E "ngl|gpu|cuda|flash"
-fa,   --flash-attn [on|off|auto]       set Flash Attention use ('on', 'off', or 'auto', default: 'auto')
-ngl,  --gpu-layers, --n-gpu-layers N   max. number of layers to store in VRAM, either an exact number,
-mg,   --main-gpu INDEX                 the GPU to use for the model (with split-mode = none), or for
                                        single value is broadcast across all devices, default: 1024
--spec-draft-ngl, -ngld, --gpu-layers-draft, --n-gpu-layers-draft N
-st,   --single-turn                    run conversation for a single turn only, then exit when done
liqixin@fedora:~/Dev/llama.cpp$

2026年5月22日

Token 预算是 AI 系统真正的硬约束
Token 预算是大语言模型（LLM）系统中的核心硬约束，直接限制每次交互可使用的 Token 数量，包括输入、输出及工具调用。它不仅影响计算成本和延迟，也决定系统可用性与任务成功率。有效管理 Token 预算依赖上下文压缩（Summarization、关键状态保留、递归编码）、动态分配策略以及实时消耗监控。在多轮对话、RAG 系统和 LLM Agent 架构中，预算驱动设计可保证信息完整性、生成稳定性和成本可控性。长期优化 Token 使用效率是系统可扩展性、可控性和可靠性提升的关键。

概念

Token Budget（Token 预算）：在大语言模型（LLM, Large Language Model）系统中，Token 预算指每次模型交互允许消耗的最大 Token 数量，包括输入 Prompt 和模型输出。Token 预算是对计算资源、延迟和成本的直接约束，而非单纯的文本长度限制。

Token Cost：Token 的计算成本由模型架构、序列长度和硬件资源决定。长序列不仅增加推理时间，也会线性或超线性地增加 GPU 显存占用和运行成本。

Context Compression（上下文压缩）：在 Token 预算限制下，为保持任务信息完整性，需要对历史上下文进行策略性压缩，包括摘要（Summarization）、选择性保留关键状态和递归编码。

Token-Budget-Driven Design（Token 预算驱动设计）：系统架构需以 Token 预算为硬约束来规划任务拆分、状态管理、工具调用和多轮交互流程。

运作机制
1. Token 消耗结构分析
- Input Tokens：用户输入、系统指令、历史上下文。
- Output Tokens：模型生成的响应。
- Overhead Tokens：控制符、特殊标记和工具接口调用占用的 Token。
1. 预算分配策略
- 静态分配（Static Allocation）：为 Prompt、上下文和输出分别预设 Token 上限。适用于确定性任务或单轮交互。
- 动态分配（Dynamic Allocation）：根据任务复杂度、上下文信息密度或工具调用需求动态调整 Token 上限。适用于多轮任务或 Agent 系统。
1. 上下文压缩方法
- 摘要压缩（Summarization）：将多轮历史对话或文档内容生成简短表示。
- 关键状态保留（Key-State Retention）：只保留对任务决策至关重要的信息。
- 递归编码（Recursive Encoding）：将长序列内容编码成向量或标记化形式，以减少 Token 占用。
1. Token 消耗监控与反馈
- 实时跟踪输入、输出和工具调用的 Token 消耗。
- 根据预算消耗自动触发上下文压缩或任务拆分机制。
- 对多轮系统提供 Token 使用报告，用于优化系统设计和提示工程（Prompt Engineering）。
实际应用
1. 多轮对话系统
- 确保对话状态管理在 Token 预算内，防止长轮次交互导致上下文截断或生成失败。
- 对重要信息使用递归摘要，实现长期对话记忆。
1. RAG 系统（Retrieval-Augmented Generation）
- 限制检索内容长度，使向模型提供的文档片段不会超过预算。
- 在检索-生成闭环中动态分配 Token，保证生成质量和上下文完整性。
1. LLM Agent 架构
- 工具调用前评估输出 Token 需求，避免因生成过长导致预算溢出。
- 复杂任务拆解为子任务，每个子任务对应独立 Token 预算管理，确保整体系统稳定。
1. 成本控制与可扩展性
- Token 预算直接对应云计算成本，可用于实时估算系统运行成本。
- 对大规模部署和高并发场景，Token 预算是保证系统稳定性和可预测性的核心指标。
深刻的见解/反思
- Token 预算是系统设计的根约束：与模型参数、算法优化相比，Token 预算直接影响系统的可用性、响应时间和成本。任何忽略 Token 预算的系统设计必然导致性能退化或任务失败。
- 设计思路应“预算优先”：在多轮交互、复杂任务或 Agent 系统中，先规划 Token 预算，再设计上下文管理、工具调用和生成策略，比先生成内容再裁剪更高效。
- Token 使用效率是长期优化核心：通过上下文压缩、关键状态保留和动态分配，系统可以在固定预算下提高信息利用率和任务成功率。
- Token 预算驱动的架构演化：随着任务复杂性增加，系统设计会趋向模块化、可递归的上下文管理结构，实现“预算可控 + 信息完整”的最佳平衡。
- Token 与 AI 可控性关联：预算限制迫使系统选择信息优先级、明确工具调用边界，从而提升生成内容的可靠性和决策可解释性。
2026年1月7日
在 VS Code编辑器中使用 LM Studio API + Continue 打造智能开发环境
在日常开发中大语言模型（LLM）已经成为许多开发者的得力助手。LM Studio 提供了本地运行大模型的能力，而 Continue 插件则能在 VS Code 中无缝调用这些模型，带来智能补全、对话和代码生成体验。通过 Continue 插件接入 LM Studio 的 API，我们可以让本地或局域网部署的 AI 模型无缝集成到开发环境中。

这种组合不仅能提供媲美云端的 AI 开发体验，还能保障数据隐私、不依赖外网。在代码补全、错误排查、文档生成等场景中，都能显著提升效率。

一、准备工作
1. 安装 LM Studio
- 前往 LM Studio 官网下载并安装。
- 启动后，根据需求下载模型（如 gpt-oss-20b、Qwen 等）。
1. 安装 VS Code 与 Continue 插件
- 打开 VS Code → Extensions → 搜索 Continue 并安装。
- 安装完成后，左侧工具栏会出现 Continue 图标。
二、启动 LM Studio API 服务
1. 打开 LM Studio，进入 开发者模式。
2. 加载所需模型（例如 gpt-oss-20b）。
3. 启动 API 服务，默认会监听本地或局域网 IP 上的指定端口，例如：
```
http://192.168.1.100:1234/v1
```
三、配置 Continue 插件

Continue 的配置文件默认路径如下：
- Windows: C:\Users\<用户名>\.continue\config.yaml
- macOS/Linux: ~/.continue/config.yaml
示例配置：
```
name: Local Agent
version: 1.0.0
schema: v1
models:
  - name: gpt-oss-20b
    provider: lmstudio
    model: openai/gpt-oss-20b
    apiBase: http://192.168.1.100:1234/v1/
```
⚠️ 注意：如果需要局域网访问，请将 localhost 替换为实际 IP 地址。

四、开始使用
1. 打开 VS Code，点击左侧 Continue 图标。
2. 在对话框中选择你配置好的模型（如 gpt-oss-20b）。
3. 输入问题，即可与模型对话，体验智能补全、调试和代码生成。
2025年9月23日
使用 llama.cpp 打造你的专属本地大语言模型服务
是否曾想过拥有一个完全私有、无需联网、响应迅速的个人 AI 助手？大语言模型（LLM）在各种任务中的表现令人惊艳，但它们通常依赖强大的 GPU 和云计算资源。llama.cpp 的出现，为本地运行 LLM 提供了一种全新可能：无需昂贵显卡，也无需连接互联网，我们也能在个人电脑上畅快使用大模型。

llama.cpp 是一个由 Georgi Gerganov 开发的开源项目，它彻底改变了我们在本地设备上运行大型语言模型 (LLM) 的方式。它以 C/C++ 编写，并针对 CPU 进行了高度优化，使得许多原本需要强大 GPU 才能运行的复杂模型，现在也能在个人电脑上流畅运行，包括 macOS、Windows 和 Linux 系统。

llama.cpp 提供了两种主要的使用方式
- 命令行界面 (CLI)
- Server Web UI。
一、命令行界面（CLI）：极简高效的交互方式

CLI (Command Line Interface) 是 llama.cpp 最基础也是最核心的交互方式。它让你能直接通过终端命令来加载模型并进行文本生成。

特点与优势
- 直接控制： CLI 提供了对模型运行参数最直接的控制。你可以精确调整上下文长度、温度、重复惩罚等参数，以微调模型的输出行为。
- 资源效率： 作为 CPU 优化项目，CLI 模式通常具有非常高的资源效率，尤其是在内存和处理速度方面。
- 脚本化友好： 对于开发者或高级用户来说，CLI 命令可以轻松集成到 shell 脚本中，实现自动化任务或批处理。
- 快速测试： 如果你只想快速测试一个模型或进行简单的交互，CLI 是最快的入门方式。
典型使用场景
- 模型功能与性能基准测试： 快速加载不同模型版本，评估其在特定任务上的表现和资源消耗。
- 自动化文本生成： 在 CI/CD 流程中集成 LLM 的文本生成能力，例如自动生成代码注释、测试用例或文档草稿。
- 特定任务微调前的数据生成： 利用模型生成大量特定格式的文本数据，用于后续的数据集构建或模型训练。
- 离线批量处理： 对大量输入数据进行 LLM 推理，无需网络连接或复杂的服务器架构。
如何使用
1. 加载模型并进行交互式对话：
```
./llama-cli -m path/to/your/model.gguf -i
```
llama-cli是主程序，-m 指定模型路径，-i 启用交互模式
1. 一次性生成文本：
```
./llama-cli -m path/to/your/model.gguf -p "请给我讲一个关于未来的短故事。" -n 256
```
-p 提供初始提示，-n指定生成最大 token 数量

二、Server Web UI：图形化体验与本地 API 服务

Server Web UI 是 llama.cpp 提供的另一种高级功能，它将模型封装成一个本地 HTTP 服务器，并允许通过 Web 浏览器或其他应用程序与之交互。这种方式为用户提供了更友好、更直观的图形界面，并扩展了模型的应用范围。

特点与优势
- 用户友好界面： 无需命令行知识，普通用户也能通过直观的网页界面与模型进行对话，就像使用 ChatGPT 一样。
- API 接口： 服务器提供与 OpenAI API 兼容的 RESTful API，这意味着许多为 OpenAI 模型设计的第三方工具、应用程序和前端界面可以直接连接到你的本地 llama.cpp 服务器。
- 可扩展性： 通过 API，你可以轻松地将本地 LLM 集成到自己的应用程序、聊天机器人或自动化流程中，而无需深入了解 llama.cpp 的底层实现。
- 多用户或多会话支持： 服务器模式通常可以更好地管理并发请求，适合同时处理多个用户或多个独立的对话会话。
典型使用场景
- 个人 AI 助理： 搭建一个私有的本地聊天机器人，用于日常交流、写作辅助、知识查询等。
- 开发者工具： 作为本地 LLM 后端，供开发中的应用程序调用，用于测试、原型设计等。
- 桌面应用集成： 任何需要 LLM 能力的桌面应用都可以通过连接到本地 llama-server 来获取支持。
如何使用
1. 启动 llama-server：
```
./llama-server -m path/to/your/model.gguf
```
默认情况下，服务器会在 http://127.0.0.1:8080 上启动。
1. 通过 Web 浏览器访问：
  服务器启动后，你可以在浏览器中打开 http://127.0.0.1:8080 来访问 llama.cpp 提供的基本 Web UI。
2025年6月30日

Ollama 与 LangChain构建本地化且强大的 LLM 生态系统

在大型语言模型（LLM）的快速发展中，Ollama 将继续简化本地部署流程，帮助开发者更加便捷地运行 LLM，而 LangChain 将扩展更多数据源和功能模块，帮助开发者构建更加丰富、复杂的 AI 应用。

1. LangChain：构建 LLM 应用的强大框架

LangChain 是一个用于构建 LLM 驱动的应用程序的开发框架，它提供了一系列强大的工具和组件，使得开发者能够轻松构建出复杂的、数据驱动的应用。LangChain 支持与各种外部数据源、API 和工具的集成，极大地扩展了 LLM 的应用场景。

LangChain 的核心功能包括多种提示管理、链式调用、数据增强等，能够帮助开发者构建更加智能、灵活的 AI 应用，如智能问答、聊天机器人、文档分析系统等。

LangChain 的特点

丰富的组件与工具：LangChain 提供了提示模板（Prompt Templates）、链式调用（Chains）、智能体（Agents）等多种组件，帮助开发者实现复杂的功能。
数据集成：LangChain 可以与多种数据源集成，包括文件、数据库、向量数据库、Web API 等，扩展了 LLM 的应用范围。
应用开发能力：开发者可以通过 LangChain 构建复杂的应用，如问答系统、聊天机器人、数据增强应用等，为实际业务提供智能支持。
会话记忆管理：LangChain 支持多轮对话和会话记忆，使得开发者能够创建更加自然的对话流和用户交互体验。

2. Ollama：简化本地 LLM 部署

Ollama 是一个工具平台，旨在简化本地 LLM 的部署过程。它为用户提供了一个简单、直观的方式，让开发者能够快速在本地计算机上运行各种开源的 LLM，无需复杂的配置或环境搭建。通过 Ollama，开发者可以轻松下载、安装并运行大多数流行的 LLM，从而降低了 LLM 使用的技术门槛。

Ollama 的特点

简洁易用的命令行工具：通过简短的命令即可下载并启动 LLM，极大简化了模型的部署流程。
本地运行，隐私保障：Ollama 允许用户在本地计算机上运行模型，避免了将数据上传至云端的隐私风险。
灵活的模型管理：用户可以方便地下载、管理并切换多个不同的 LLM，例如 LLaMA、Mistral、Gemma 等，满足不同应用需求。
跨平台支持：Ollama 支持 macOS、Linux 和部分 Windows 系统，确保各种开发环境都能顺利使用。

3. Ollama与LangChain的协同作用

虽然 Ollama 和 LangChain 各自有独立的功能，但它们实际上是互为补充的工具，可以在多个开发场景中协同工作，帮助开发者更高效地利用 LLM 技术。

例如，开发者可以在本地使用 Ollama 运行 LLAMA 模型，然后结合 LangChain 构建一个智能问答系统，该系统能够与外部数据源（如本地文件或数据库）交互。Ollama 提供了本地模型的运行环境，而 LangChain 则帮助开发者设计和实现复杂的应用逻辑。

4. 应用场景

场景	Ollama 角色	LangChain 角色
本地文档问答系统	提供本地运行的 LLM 推理能力	管理文档解析、嵌入、问答流程
离线智能助理	本地运行模型（如LLAMA）	处理会话管理、记忆、工具调用
多模型测试与对比	快速切换多个 LLM 进行测试	提供统一接口进行多模型测试

通过将 Ollama 与 LangChain 结合，开发者不仅能够在本地高效运行 LLM，还能够快速构建复杂的 LLM 应用，充分发挥 LLM 的能力。

2025年4月13日

基于LangChain和Ollama的本地文档问答系统

使用 LangChain 框架和 Ollama 开源大模型搭建一个问答系统，根据提供的文档内容回答用户提出的问题。

系统主要由以下几个部分组成：

文档加载模块: 负责从指定的路径加载文档。
文档切分模块: 将加载的文档按照指定的大小和重叠度进行切分，以便 LLM 处理。
LLM 加载模块: 负责加载指定的 Ollama 模型。
问答链创建模块: 使用 LangChain 表达式语言 (LCEL) 构建问答链，将用户问题和文档内容输入到 LLM 中，并解析 LLM 的输出。
提问模块: 接收用户输入的问题，调用问答链生成答案，并返回给用户。
重试机制: 使用 Tenacity 库，当提问模块出现错误时，自动进行重试，以提高系统的稳定性。

1. 创建一个虚拟环境，名字为langqa

conda create -n langqa python=3.12

2. 激活虚拟环境

conda activate langqa

3. 安装依赖

pip install langchain langchain_community langchain-ollama ollama tenacity

4. 部署代码

替换为你自己的文件、模型和地址

文件：tcpdump.1-4.4.0.txt
模型：llama3.1:8b
OLLAMA ：http://localhost:11434

配置完成后使用命令行进行运行对话

python langqa.py

代码

"""
Author: Qixinlee
Description: 基于LangChain和Ollama的本地文档问答系统
""" 

import os
from langchain_community.document_loaders import TextLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_ollama import OllamaLLM
from langchain.prompts import PromptTemplate
import ollama
from tenacity import retry, stop_after_attempt, wait_random_exponential
from langchain_core.runnables import chain
from langchain_core.output_parsers import StrOutputParser

# 常量定义
DOCUMENT_PATH = os.path.join(os.path.dirname(__file__), "tcpdump.1-4.4.0.txt")
CHUNK_SIZE = 1000
CHUNK_OVERLAP = 100
LLM_MODEL_NAME = "llama3.1:8b"
OLLAMA_HOST = "http://localhost:11434"

# 1. 设置 Ollama 连接信息
os.environ["OLLAMA_HOST"] = OLLAMA_HOST
ollama.DEFAULT_BASE_URL = OLLAMA_HOST

# 2. 加载文档
def load_document(document_path: str):
    try:
        loader = TextLoader(document_path)
        documents = loader.load()
        print(f"Successfully loaded document from {document_path}")
        return documents
    except Exception as e:
        print(f"Error loading document: {e}")
        exit()

# 3. 切分文档
def split_document(documents, chunk_size: int, chunk_overlap: int):
    text_splitter = RecursiveCharacterTextSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap)
    texts = text_splitter.split_documents(documents)
    return texts

# 4. 加载 LLM
def load_llm(llm_model_name: str):
    try:
        llm = OllamaLLM(model=llm_model_name)
        print(f"Successfully loaded LLM model: {llm_model_name}")
        return llm
    except Exception as e:
        print(f"Error loading LLM model: {e}")
        exit()

# 5. 创建问答链 (使用 LangChain 表达式语言)
def create_qa_chain(llm):
    try:
        template = """你是一个问答机器人。请根据以下已知信息，用简洁和专业的中文回答用户的问题。
        如果无法从中得到答案，请清晰地输出 “根据已知信息无法回答该问题”。

        已知信息:
        {context}

        问题: {question}
        答案:"""
        QA_CHAIN_PROMPT = PromptTemplate(
            input_variables=["context", "question"], template=template
        )

        # 使用 Runnable 序列
        qa_chain = QA_CHAIN_PROMPT | llm | StrOutputParser()

        print("Successfully created QA chain.")
        return qa_chain
    except Exception as e:
        print(f"Error creating QA chain: {e}")
        exit()

# 6. 提问 (使用 Tenacity 实现重试机制)
@retry(stop=stop_after_attempt(3), wait=wait_random_exponential(multiplier=1, min=4, max=10))
def generate_answer(qa_chain, document_content: str, query: str):
    try:
        result = qa_chain.invoke({"context": document_content, "question": query})
        print("Successfully generated answer.")
        return result
    except Exception as e:
        print(f"Error during QA: {e}")
        raise

# 主程序
if __name__ == "__main__":

    # 1. 加载文档
    documents = load_document(DOCUMENT_PATH)

    # 2. 切分文档
    texts = split_document(documents, CHUNK_SIZE, CHUNK_OVERLAP)

    # 3. 加载 LLM (使用常量)
    llm = load_llm(LLM_MODEL_NAME)

    # 4. 将所有文本块连接成一个字符串
    document_content = "\n".join([text.page_content for text in texts])

    # 5. 创建问答链
    qa_chain = create_qa_chain(llm)

    # 6. 循环提问
    while True:
        query = input("请输入你的问题 (输入 'exit' 退出): ")
        if query.lower() == "exit":
            break

        try:
            result = generate_answer(qa_chain, document_content, query)
            print("Question (中文):", query)
            print("Answer (中文):", result)

        except Exception as e:
            print(f"Failed to generate answer: {e}")
            exit()

2025年4月12日

标签： LLM

Fedora 43 源码编译安装 llama.cpp 教程（支持 CUDA 显卡加速）

1. 配置 CUDA 环境变量

2. 获取源码与依赖安装

3. 利用 CMake 执行 CUDA 加速编译

编译参数解析

4. 验证编译结果

4.1 验证基础版本信息

4.2 核心验证：检查 CUDA 参数是否就绪

5. 部署大模型服务（llama-server）

核心启动参数调优说明：

启动后如何玩转？

编译过程

Token 预算是 AI 系统真正的硬约束

概念

运作机制

实际应用

深刻的见解/反思

在 VS Code编辑器中使用 LM Studio API + Continue 打造智能开发环境

一、准备工作

二、启动 LM Studio API 服务

三、配置 Continue 插件

四、开始使用

使用 llama.cpp 打造你的专属本地大语言模型服务

一、命令行界面（CLI）：极简高效的交互方式

特点与优势

典型使用场景

如何使用

二、Server Web UI：图形化体验与本地 API 服务