From download
to your first Kimi response.

Before the large download.

These instructions target the tested Linux/NVIDIA setup: one RTX PRO 6000 Blackwell with 96 GB VRAM and 70 GiB of host RAM available to the Linux environment. These are tested conditions, not a minimum-hardware claim.

• Software: Git, a C/C++ compiler and make, Python 3 with venv/pip, and a compatible NVIDIA driver and CUDA toolkit. On Ubuntu, the basic packages are git build-essential python3-venv python3-pip. Use NVIDIA’s Linux instructions for the driver/toolkit combination.
• NVMe space: the source checkpoint is about 1.56 TB and the extracted experts add about 1.45 TB. Allow at least 3.5 TB of free space for this example workflow, including resident tensors, the spine and working headroom. This is a planning allowance, not a measured minimum; replicas need more space.
• Filesystem: put the model on a local Linux filesystem that supports sparse files and direct reads, such as ext4 or XFS. The spine has a large logical size with an unallocated expert region. Avoid exFAT for this workflow.

nvidia-smi
nvcc --version

If either command fails, finish the driver/toolkit setup first. The engine build below targets Blackwell; its compiler must support sm_120a.

STEP 01

Get and build the Kimi engine.

Open a Bash terminal on your Linux machine. This downloads our integrated engine at the release used by the published November 7 measurements. It builds a program called ds4 inside the repository.

git clone --branch k31-cuda-qwen-20261107b --depth 1 \
  https://github.com/argonautlabsai/argodrive-ds4-kimi-k3.1.git
cd argodrive-ds4-kimi-k3.1
make ds4 CUDA_ARCH=sm_120

Create a Python environment for the download and conversion tools. Keep this terminal open and stay in the repository for the remaining steps.

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install huggingface_hub numpy

Git may report “detached HEAD” because the command selects a release tag. That is expected. The optional Qwen drafter needs additional dependencies; the first run below leaves it off.

STEP 02

Download the weights. Prepare them once.

The model comes from Moonshot’s official Kimi-K3.1-Pro repository. The engine and model are separate downloads. This converter expects the original safetensors checkpoint, not an unrelated GGUF quantization. Review the model’s license on its page.

Choose your model directory

Replace /mnt/nvme/kimi-k31 below with a writable folder on your mounted NVMe storage. Use a fresh directory for K3.1, not an existing K3 extraction. KIMI_DATA is just a shortcut used by the commands in this guide.

export KIMI_DATA=/mnt/nvme/kimi-k31
mkdir -p "$KIMI_DATA"
df -h "$KIMI_DATA"

Download the complete checkpoint

The Hugging Face CLI shows progress. The first command previews the download; the second downloads it. Both pin the model revision inspected for this guide. Wait for the full download to finish before extracting.

export KIMI_MODEL_REV=d9e99b0792a5e93b78ea9bea62a68167e36ddd9e
hf download moonshotai/Kimi-K3.1-Pro \
  --revision "$KIMI_MODEL_REV" \
  --local-dir "$KIMI_DATA/checkpoint" --dry-run

hf download moonshotai/Kimi-K3.1-Pro \
  --revision "$KIMI_MODEL_REV" \
  --local-dir "$KIMI_DATA/checkpoint"

Download time depends on your connection. If interrupted, rerun the same download command. If Hugging Face asks you to authenticate, run hf auth login in your terminal.

Extract the experts and resident tensors

Run the extractor included with the engine. It copies the expert data into files the reader can load individually, and separates the tensors that stay resident. This adds a second large copy of the weights; it does not delete the checkpoint.

python tools/kimi-k3/k31_extract.py \
  --src "$KIMI_DATA/checkpoint" \
  --out-experts "$KIMI_DATA/experts" \
  --out-resident "$KIMI_DATA/resident" \
  --out-meta "$KIMI_DATA/meta" \
  --workers 8

Expect 82,432 expert files, each 17,547,264 bytes. Resolve any incomplete-shard or bad-size messages before continuing. The extractor can skip already present files when rerun.

python - <<'PY'
import os
from pathlib import Path
folder = Path(os.environ["KIMI_DATA"]) / "experts"
bad = 0
for layer in range(1, 93):
    for expert in range(896):
        path = folder / f"L{layer}-E{expert}.bin"
        if not path.is_file() or path.stat().st_size != 17547264:
            bad += 1
print(f"Expected 82432 experts; missing or wrong size: {bad}")
raise SystemExit(1 if bad else 0)
PY

Create the spine file

The spine holds the non-expert weights and records the expert directory. Pass the resident/tensors subfolder, not the parent resident folder. Keep the recorded expert path in place after conversion.

python tools/kimi-k3/k3_spine_to_gguf.py \
  --meta "$KIMI_DATA/meta" \
  --src "$KIMI_DATA/resident/tensors" \
  --out "$KIMI_DATA/k3-spine.gguf" \
  --primary-experts "$KIMI_DATA/experts" \
  --name "Kimi K3.1 Pro"

The resulting GGUF is sparse: its apparent size is larger than the disk space it actually occupies. Use du -h "$KIMI_DATA/k3-spine.gguf" to inspect allocated space. Keep the source files until you have a working run and a recovery plan.

STEP 03

Run your first local response.

Use the engine you built with the prepared spine. This example generates up to 128 tokens on GPU 0, with the drafter off and expert streaming on.

CUDA_VISIBLE_DEVICES=0 DS4_K3_QWEN=0 \
./ds4 -m "$KIMI_DATA/k3-spine.gguf" \
  --cuda --raw \
  --ssd-streaming --ssd-streaming-cold \
  --ssd-streaming-cache-experts 16 \
  --temp 0 -c 2048 -n 128 \
  -p "The three main financial statements are"

Allow time for startup and the prompt pass. A working run loads the model, prints generated text and reports timing statistics. The whole model does not have to fit in VRAM: the engine reads selected experts from NVMe.

After opening a new terminal, return to this repository and set KIMI_DATA to the same absolute path. Reactivate .venv when using the Python tools.

CUDA_VISIBLE_DEVICES=0 DS4_K3_QWEN=0 \
./ds4 -m "$KIMI_DATA/k3-spine.gguf" \
  --cuda --ssd-streaming --ssd-streaming-cold \
  --ssd-streaming-cache-experts 16 \
  --temp 0 -c 2048 -n 128 \
  --k3-serve 127.0.0.1:8080

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"kimi-k3","messages":[{"role":"user","content":"Explain cash flow in two sentences."}],"max_tokens":128,"temperature":0}'

Get it running first. Then tune it.

The basic recipe above is not the 3.11 tok/s benchmark configuration. That result uses a Qwen drafter, a specific raw-text prompt and five NVMe devices across three storage tiers. Chat gains are smaller. 44 tok/s is prefill, not output generation.

If something stops.

“nvcc not found” or an unsupported GPU architecture

Check your CUDA toolkit installation and PATH. This recipe targets the RTX PRO 6000 Blackwell; an older compiler may not recognise sm_120a.

Missing shards, inventory entries or expert files

Finish the pinned download, then rerun extraction into the same K3.1 directory. Check the extractor’s summary; do not continue with incomplete metadata.

A tensor file cannot be found during spine conversion

Check --src: it must point to resident/tensors. Check that --meta points to the metadata produced by the same extraction.

The spine looks much larger than expected

Its expert region is sparse. Compare allocated space with du; do not copy it using a method that expands its holes.

Out of memory or slower than the benchmark

Free GPU and host memory, check available disk space and compare your hardware with the tested setup. One expert directory and drafter-off decoding are not the published tuned configuration.

For help, open a GitHub issue with your OS, GPU, CUDA version, release tag and the error. Remove access tokens, personal paths and private prompts from logs before sharing them.