Can you self-host Kimi K2.5 on $5,000 of Azure credit?

Status: engineering review, not a run report. Nothing below has been provisioned or benchmarked yet. Prices are Azure list prices in USD, and versions were verified on 20 October 2026. Labels: [DOC] documented, [CALC] calculated, [EXP] needs a test, [SUB] depends on your subscription.

0. Scope, labels and placeholders

This is an engineering review and playbook only. Nothing here has been provisioned, run or measured; every command is for you to run later, after the feasibility gates pass.

Evidence labels used on every claim

Where each command runs

Placeholders (defined once, used everywhere)

# [CS] or [WS-bash] — paste once per shell session
export SUB_ID="<YOUR_SUBSCRIPTION_ID>"            # [SUB]
export LOCATION="<REGION_WITH_CAPACITY>"          # e.g. eastus2, southcentralus — [SUB] quota decides
export RG="rg-kimi-inference"
export VNET="vnet-kimi"; export SUBNET="snet-gpu"
export NSG="nsg-kimi-gpu"
export VM_NAME="vm-kimi-01"
export VM_SIZE="<SELECTED_SKU>"                   # chosen in the feasibility section
export ADMIN_USER="kimiadmin"
export DATA_DISK="disk-kimi-models"
export DATA_DISK_GB="<SIZE_FROM_MEMORY_PLAN>"
export DOMAIN="<YOUR_DOMAIN>"                     # needed for Option A or B
export API_HOST="kimi.${DOMAIN}"
export MODEL_REPO="<HF_REPO_ID>"                   # fixed in the feasibility section
export MODEL_REV="<40-char HF commit SHA>"         # resolved by command, never 'main'
export VLLM_IMAGE_TAG="<pinned vLLM tag>"
export VLLM_DIGEST="<sha256:... resolved by command>"
export SERVED_ALIAS="kimi-work"

Credit currency, credit type and expiry are unknown to me. All money figures below are in USD list price unless labelled otherwise, because Azure's retail price API returns USD by default; convert using your own invoice currency.

Feasibility decision

Verdict: feasible, with one change of hardware and no change of model. Run moonshotai/Kimi-K2.5 (native INT4) on one Standard_ND96isr_H200_v5 (8× H200 141 GB) with vLLM, TP=8. vLLM's recipe for Kimi K2.5 documents exactly this setup [DOC]. The two configurations you named are not suitable: 8× A100-80GB and 8× H100-80GB cannot hold the K2.5 weights plus working memory [CALC], and the NVFP4 variant needs Blackwell GPUs, which Azure only offers as rack-scale GB200 VMs [DOC]. The latest open-weight Kimi, K3, is not feasible on your budget: its smallest documented deployment is 32× H200 [DOC].

At list price, $5,000 USD of credit buys about 45 billed GPU-hours (roughly 30–37 hours of actual use) of the recommended VM, not 180 (Section 6). If your credit is that tight, a pay-per-token Kimi endpoint in Azure AI Foundry may serve you better. That would not be self-hosted; see the last row of the table below.

Which Kimi model is which (verified 2026-10-20)

The K2.5 repository at revision 4d01dfe0332d63057c186e0b262165819efb6611 (lastModified 2026-04-30) was observed on 2026-10-20 [DOC: HF API]. Re-resolve it with the command in Section 3 before you rely on it.

I have not swapped your model. K2.6 is a drop-in successor with the same architecture. Use it only if you choose to: set MODEL_REPO=moonshotai/Kimi-K2.6, resolve its SHA, and keep everything else.

Activated parameters are not the memory requirement. K2.5 activates 32B parameters per token, but all 1T parameters (384 experts) must stay resident in GPU memory. The 595 GB checkpoint is what has to fit.

Candidate comparison

All prices are Azure retail Linux pay-as-you-go in USD, from the Retail Prices API on 2026-10-20. A price listing shows that a SKU exists in a region. It does not prove you have quota or that capacity is available.

The H200 Spot rows in the price API show the same price as on-demand. That looks like a data anomaly, and Spot eviction would kill long sessions anyway, so it is excluded.

Region. Nothing in Australia is required. Australian H200 list prices are 30–45% higher than US prices, and a US region adds roughly 150–200 ms of network round trip from Melbourne [EXP]. That is small next to model latency. Pick a US region unless data residency matters to you. The final choice depends on your quota and capacity [SUB].

Memory plan: K2.5 on 8× H200, 32,768-token context [CALC]

Inputs: config.json gives num_hidden_layers=61, kv_lora_rank=512 and qk_rope_head_dim=64. The checkpoint is 595.2 GB = 554.3 GiB. The vision tower is MoonViT, about 400M parameters in BF16.

Chosen settings: --max-model-len 32768, --max-num-seqs 4, KV cache dtype auto (BF16). That is four times your likely single-user concurrency and avoids FP8-KV accuracy questions. The same arithmetic on 80 GB GPUs leaves negative headroom: 0.95 × 74.5 − 69.95 − 8 < 0. That is why A100 and H100 are rejected.

Storage, download and startup [CALC]

• The weights need ≥ 600 GB of storage, plus about 25 GB for the container image.

• A managed disk is capped by the VM's uncached throughput of 612 MBps (ND H200 v5 spec). Reading 595 GB from it takes at least about 16 minutes on every cold start. A P30 disk (200 MBps) would take about 50 minutes.

• The plan therefore keeps the durable copy in Blob storage (about $11/month) and restores it to the VM's local NVMe (8 disks, 28 TiB, ephemeral) at each start with azcopy. Restore time is unknown until measured [EXP]. The VM has 80 Gbps of network bandwidth, but Blob throughput is the limit.

• The first download from Hugging Face is a one-time cost. Inbound data transfer is free [DOC: price API, "Standard Data Transfer In 0.0"].

Gates before any spend

1. [SUB] Standard NDSH200v5 Family vCPUs quota of at least 96 in the chosen region. A new subscription usually has 0 and must file a quota request.

2. [SUB] The SKU is unrestricted for your subscription in that region (az vm list-skus, Section 1).

3. [SUB] Your credit type covers ND-series compute. Some sponsorship and free credits exclude GPU or Marketplace items; check Cost Management.

4. [EXP] A real allocation succeeds. Approved quota does not guarantee physical capacity.

If gate 1 or 4 fails in every US region, fall back to Standard_ND96isr_MI300X_v5, which runs the same model through vLLM ROCm per the recipe, or stop.

Missing inputs

The playbook runs with the assumptions below. Each one changes a specific setting if your real answer differs.

Target architecture

You choose one edge path: nginx on the VM (A) or Cloudflare's edge plus cloudflared (B). Both converge on vLLM, which listens only on loopback; Open WebUI talks to it on the private Docker network. Weights come from Blob storage to local NVMe at each boot, and two independent mechanisms can deallocate the VM.

1. Azure infrastructure provisioning

This section builds one GPU VM in a private-by-default subnet. It has a Standard public IP used for outbound traffic (Hugging Face, apt, Cloudflare) and no inbound internet rule except the ones a chosen access option adds. A public IP costs $0.005/h; a NAT Gateway would also charge per GB processed on the 595 GB download. Weights live in a Blob container that only the VM's managed identity can reach. Billing starts at az vm create, so run nothing past step 1.5 until gates 1–3 pass.

1.1 Subscription and providers — [CS]

az login                                   # Cloud Shell is already logged in
az account set --subscription "$SUB_ID"
az account show --query "{name:name,id:id,tenant:tenantId}" -o table
for ns in Microsoft.Compute Microsoft.Network Microsoft.Storage Microsoft.KeyVault Microsoft.Automation Microsoft.Quota Microsoft.CostManagement; do
  az provider register --namespace "$ns" --wait
  echo "$ns: $(az provider show -n "$ns" --query registrationState -o tsv)"
done

Success: each line prints Registered. Failure: AuthorizationFailed means you lack Owner or Contributor on the subscription. Ask the subscription owner; there is no workaround.

1.2 Quota (approval) — [CS]

az vm list-usage --location "$LOCATION" -o table | grep -Ei 'H200|Total Regional'

Success: the Standard NDSH200v5 Family vCPUs row (the exact family label comes from your output [SUB]) has Limit − CurrentValue ≥ 96, and so does Total Regional vCPUs. Failure: a limit of 0 is normal. Request 96 vCPUs in Portal → Quotas → Compute → filter "H200" → New quota request, and repeat for each candidate region. GPU requests can take days and may be refused.

1.3 SKU restrictions — [CS]

az vm list-skus --location "$LOCATION" --size Standard_ND96isr_H200_v5 --all \
  --query "[].{sku:name, zones:locationInfo[0].zones, restrictions:restrictions[].reasonCode}" -o json

Success: one entry with "restrictions": []. Failure: NotAvailableForSubscription means the SKU is blocked for you in this region, even if quota exists. Try another region or open a support ticket.

Quota is not capacity. Approved quota lets you ask for 96 vCPUs. Azure can still answer AllocationFailed or ZonalAllocationFailed at create or start time, and again after every deallocation. Section 6 covers what to do when that happens.

1.4 Image check — [CS]

The Ubuntu-HPC 24.04 image ships the NVIDIA datacenter driver, Fabric Manager (needed for NVSwitch on 8-GPU HGX), CUDA, NCCL and DOCA-OFED. Azure HPC docs recommend it for ND-series. The azhpc-images release ubuntu-hpc-2026092501 lists driver 580.178.04, CUDA 13.0.88, Fabric Manager 580.178.04-1ubuntu1, NCCL 2.31.2-1, DOCA OFED 3.2.3 and Docker (Moby) 29.8.1 [DOC].

export IMAGE_URN="microsoft-dsvm:ubuntu-hpc:2404:24.04.2026092501"
az vm image show --location "$LOCATION" --urn "$IMAGE_URN" \
  --query "{version:name, gen:hyperVGeneration, plan:plan, arch:architecture}" -o json

Success: gen is V2, arch is x64, plan is null. If plan is not null: run az vm image terms accept --urn "$IMAGE_URN" first. If the image is missing in the region: list versions with az vm image list --publisher microsoft-dsvm --offer ubuntu-hpc --sku 2404 --all -o table, pick the newest x86 24.04.*, and re-read its release notes for the driver version.

1.5 Network, NSG and public IP — [CS]

export MY_IP="$(curl -s https://api.ipify.org)/32"     # your workstation's IP, for SSH only
az group create -n "$RG" -l "$LOCATION"

az network nsg create -g "$RG" -n "$NSG"
# SSH: created DISABLED. Open only during admin work (see 1.9)
az network nsg rule create -g "$RG" --nsg-name "$NSG" -n allow-ssh-admin --priority 100 \
  --direction Inbound --access Deny --protocol Tcp --source-address-prefixes "$MY_IP" \
  --destination-port-ranges 22
# Explicit deny-all from the Internet (stronger than relying on defaults)
az network nsg rule create -g "$RG" --nsg-name "$NSG" -n deny-internet-all --priority 4000 \
  --direction Inbound --access Deny --protocol '*' --source-address-prefixes Internet \
  --destination-port-ranges '*'

az network vnet create -g "$RG" -n "$VNET" --address-prefixes 10.40.0.0/24 \
  --subnet-name "$SUBNET" --subnet-prefixes 10.40.0.0/26
az network vnet subnet update -g "$RG" --vnet-name "$VNET" -n "$SUBNET" \
  --network-security-group "$NSG" --default-outbound false \
  --service-endpoints Microsoft.Storage Microsoft.KeyVault

az network public-ip create -g "$RG" -n pip-kimi --sku Standard --allocation-method Static --version IPv4

Check:

az network vnet subnet show -g "$RG" --vnet-name "$VNET" -n "$SUBNET" \
  --query "{nsg:networkSecurityGroup.id, defaultOutbound:defaultOutboundAccess}" -o json
az network nsg rule list -g "$RG" --nsg-name "$NSG" -o table

Success: defaultOutbound is false, the NSG is attached, and the only custom rules are the Deny-state SSH rule and deny-internet-all. Port 8000 appears nowhere, and it never will. The default-outbound setting follows Microsoft's default outbound access guidance: new VNets are private by default for API versions after 31 March 2026, and the VM gets an explicit outbound path through its public IP.

1.6 Blob storage for the durable weight copy — [CS]

export SA="stkimi$RANDOM$RANDOM"           # must be globally unique, 3–24 lowercase alphanumerics
az storage account create -g "$RG" -n "$SA" -l "$LOCATION" --sku Standard_LRS --kind StorageV2 \
  --access-tier Hot --min-tls-version TLS1_2 --allow-blob-public-access false \
  --allow-shared-key-access false --default-action Deny --https-only true
az storage account network-rule add -g "$RG" --account-name "$SA" --vnet-name "$VNET" --subnet "$SUBNET"
az storage container-rm create -g "$RG" --storage-account "$SA" -n models   # management plane; needs no data role
echo "export SA=$SA" >> ~/.kimi.env

Success: az storage account show -n "$SA" --query "{shared:allowSharedKeyAccess,net:networkRuleSet.defaultAction}" returns false and Deny.

1.7 Key Vault for secrets — [CS]

export KV="kv-kimi-$RANDOM"
az keyvault create -g "$RG" -n "$KV" -l "$LOCATION" --enable-rbac-authorization true --retention-days 7
az role assignment create --assignee "$(az ad signed-in-user show --query id -o tsv)" \
  --role "Key Vault Secrets Officer" --scope "$(az keyvault show -n "$KV" --query id -o tsv)"
# vLLM API key: 256 random bits. Never typed into a chat app except the one client that needs it
az keyvault secret set --vault-name "$KV" -n vllm-api-key --value "$(openssl rand -hex 32)" -o none
echo "export KV=$KV" >> ~/.kimi.env

1.8 GPU VM — [CS] (billing starts here)

az vm create -g "$RG" -n "$VM_NAME" -l "$LOCATION" --size Standard_ND96isr_H200_v5 \
  --image "$IMAGE_URN" --security-type Standard \
  --admin-username "$ADMIN_USER" --ssh-key-values "<PATH_TO_YOUR_ed25519.pub>" \
  --vnet-name "$VNET" --subnet "$SUBNET" --nsg "" --public-ip-address pip-kimi \
  --os-disk-size-gb 256 --storage-sku Premium_LRS --os-disk-delete-option Detach \
  --assign-identity '[system]' --accelerated-networking true

• --security-type Standard: Secure Boot blocks unsigned DKMS kernel modules, and the driver extension doc says Secure Boot is unsupported for driver extensions. Whether Trusted Launch with Secure Boot off works on this SKU and image is [EXP].

• The OS disk is a 256 GiB Premium SSD (P15). It persists through deallocation and holds the OS, Docker images, Open WebUI data and config. It does not hold the weights.

• --os-disk-delete-option Detach keeps the disk if you delete the VM by mistake. Final cleanup is in Section 6.

Check:

az vm get-instance-view -g "$RG" -n "$VM_NAME" --query "instanceView.statuses[?starts_with(code,'PowerState')].displayStatus" -o tsv
az vm show -g "$RG" -n "$VM_NAME" --query "{size:hardwareProfile.vmSize, mi:identity.type, zone:zones}" -o json

Success: VM running, size Standard_ND96isr_H200_v5, identity SystemAssigned. Failures: AllocationFailed / ZonalAllocationFailed means no capacity. Delete the failed VM if one was half-created, then retry later or in another approved region. QuotaExceeded sends you back to 1.2. OperationNotAllowed ... security type means the SKU or image disagrees with the security type: retry with --security-type TrustedLaunch --enable-secure-boot false --enable-vtpm true [EXP].

1.9 Managed identity with narrow permissions — [CS]

The VM identity gets exactly three things: read (and, during seeding only, write) the models container; read Key Vault secrets; read and deallocate itself.

PID=$(az vm show -g "$RG" -n "$VM_NAME" --query identity.principalId -o tsv)
VM_ID=$(az vm show -g "$RG" -n "$VM_NAME" --query id -o tsv)
C_SCOPE="$(az storage account show -n "$SA" --query id -o tsv)/blobServices/default/containers/models"

# Seeding phase only. Downgraded to Reader in Section 3.3
az role assignment create --assignee-object-id "$PID" --assignee-principal-type ServicePrincipal \
  --role "Storage Blob Data Contributor" --scope "$C_SCOPE"
az role assignment create --assignee-object-id "$PID" --assignee-principal-type ServicePrincipal \
  --role "Key Vault Secrets User" --scope "$(az keyvault show -n "$KV" --query id -o tsv)"

cat > vm-self-deallocate.json <<EOF
{
  "Name": "VM Self Deallocate (kimi)",
  "Description": "Read and deallocate one VM. Nothing else.",
  "Actions": ["Microsoft.Compute/virtualMachines/read",
              "Microsoft.Compute/virtualMachines/instanceView/read",
              "Microsoft.Compute/virtualMachines/deallocate/action"],
  "AssignableScopes": ["/subscriptions/$SUB_ID/resourceGroups/$RG"]
}
EOF
az role definition create --role-definition @vm-self-deallocate.json
az role assignment create --assignee-object-id "$PID" --assignee-principal-type ServicePrincipal \
  --role "VM Self Deallocate (kimi)" --scope "$VM_ID"

az role assignment list --assignee "$PID" --all -o table

Success: exactly three assignments, each scoped to the container, the vault or the VM, and none at subscription or resource-group scope. A new custom role can take a few minutes to replicate; retry the assignment if it says the role is not found.

Admin access (SSH), on demand only — [CS]:

az network nsg rule update -g "$RG" --nsg-name "$NSG" -n allow-ssh-admin --access Allow \
  --source-address-prefixes "$(curl -s https://api.ipify.org)/32"
ssh -i <PRIVATE_KEY> "$ADMIN_USER@$(az network public-ip show -g "$RG" -n pip-kimi --query ipAddress -o tsv)"
# when finished:
az network nsg rule update -g "$RG" --nsg-name "$NSG" -n allow-ssh-admin --access Deny

If SSH is blocked or broken, az vm run-command invoke -g "$RG" -n "$VM_NAME" --command-id RunShellScript --scripts "..." runs commands through the Azure agent with no inbound port.

1.10 Discover local storage before touching it — [VM]

lsblk -d -o NAME,SIZE,MODEL,TYPE,MOUNTPOINTS
lsblk -f
findmnt -t ext4,xfs -o TARGET,SOURCE,SIZE
sudo blkid
cat /proc/mdstat

Illustrative expected output (shape only): one OS disk of about 256G with partitions mounted at /; one temp disk of about 1T, often already mounted at /mnt; and eight nvme*n1 devices of about 3.5T each, model text resembling "Microsoft NVMe Direct Disk". Device names and the image's own NVMe handling vary [EXP].

Apply these rules before any format:

1. If findmnt shows the NVMe devices already assembled and mounted by the image (for example a RAID under /mnt/nvme), use that mount and skip to 1.11.

2. Otherwise, select only devices that are type disk, report the NVMe Direct Disk model, are not the OS disk, have no partitions, no filesystem signature (sudo wipefs -n /dev/X prints nothing), and are not in /proc/mdstat.

3. Never format the OS disk or any device with existing data.

# [VM] Build the candidate list from evidence, then STOP and read it before continuing.
CAND=$(lsblk -dn -o NAME,TYPE,MODEL | awk '$2=="disk" && /NVMe Direct/ {print "/dev/"$1}')
for d in $CAND; do
  echo "$d parts=$(lsblk -n "$d" | wc -l) sig=$(sudo wipefs -n "$d" | wc -l)"
done

Proceed only if every listed device shows parts=1 (the disk itself) and sig=0. Then:

# [VM] local-nvme.sh — also reused at every boot (Section 6), because NVMe is wiped on deallocate
set -euo pipefail
N=$(echo $CAND | wc -w)
sudo mdadm --create /dev/md/kimi --level=0 --raid-devices="$N" $CAND --force --run
sudo mkfs.xfs -f -L kimi-nvme /dev/md/kimi
sudo mkdir -p /mnt/kimi && sudo mount -o noatime LABEL=kimi-nvme /mnt/kimi
sudo chown "$USER":"$USER" /mnt/kimi
df -h /mnt/kimi

Success: df shows about 28T at /mnt/kimi. Do not add it to /etc/fstab with a hard mount. After deallocation the devices come back blank and boot would hang; the boot unit in Section 6 recreates it.

1.11 Persistent vs ephemeral storage

2. System, driver and container setup

One driver method: the driver baked into the Ubuntu-HPC image. Do not also add the NVIDIA GPU Driver Extension or apt nvidia-driver-* packages. Two driver sources on one host is the most common way to break NVML. The extension's documented OS list is Ubuntu 22.04 for CUDA, which also rules it out for this 24.04 image (doc).

Pinned stack

Host driver vs container CUDA. The container ships the CUDA runtime libraries; the host provides only the kernel driver and libcuda. A container works when the host driver is at least the minimum for the container's CUDA major/minor version. NVIDIA's release notes list R580 for CUDA 13.0 and ≥ 580 as the 13.x minor-version-compatibility floor; CUDA 12.9 needs ≥ 575.51.03 [DOC]. vLLM's default images are CUDA 13.0 per the v0.32.0 release notes; that v0.31.0 is also 13.0 is [EXP], and step 2.5 verifies it. If the container reports CUDA newer than 13.0, switch to v0.31.0-x86_64-cu129 rather than upgrading the host driver.

Version choice. vLLM v0.32.0 (2026-10-18) is two days old. v0.31.0 has been out for 16 days and is well above both the K2.5 recipe's verified v0.17.0 and the K2.6 recipe's ≥ 0.25.0 minimum. The recipe's v0.17.0 is the only version Moonshot/vLLM state as verified for K2.5 [DOC]; v0.31.0 with K2.5 is [EXP] until your smoke test passes. If it fails, v0.17.0 is the documented fallback.

2.1 Freeze the kernel and driver — [VM]

An unattended kernel upgrade can leave the DKMS driver unbuilt, and the GPUs vanish after the next start.

dpkg -l | grep -E 'nvidia|fabricmanager|linux-image|linux-azure' | awk '{print $2, $3}'
sudo apt-mark hold $(dpkg -l | awk '/^ii/ && ($2 ~ /^(nvidia-|libnvidia-|cuda-drivers)/ || $2 ~ /fabricmanager/ || $2 ~ /^linux-(image|headers|modules|azure)/) {print $2}')
apt-mark showhold
sudo systemctl disable --now unattended-upgrades

Success: showhold lists the driver, Fabric Manager and kernel packages. Apply security updates deliberately: unhold, upgrade, reboot, rerun 2.2, re-hold.

2.2 GPUs, Fabric Manager and NVLink — [VM]

nvidia-smi --query-gpu=index,name,memory.total,driver_version,persistence_mode --format=csv
systemctl is-active nvidia-fabricmanager
nvidia-smi -q | grep -A2 -i 'Fabric'
nvidia-smi topo -m
nvidia-smi nvlink --status -i 0 | head -20
sudo nvidia-smi -pm 1

Illustrative expected output: 8 rows of NVIDIA H200, memory.total around 143771 MiB [EXP: the exact value replaces the 131.3 GiB conservative figure in the memory plan], driver 580.178.04. Fabric Manager active, fabric state Completed / status Success. In topo -m, every GPU pair shows NV18, and NVLink shows 18 links at about 26 GB/s each.

2.3 Docker — [VM]

docker version --format 'client={{.Client.Version}} server={{.Server.Version}}'
dpkg -l | grep -E '^ii\s+(moby-engine|docker-ce)\s' | awk '{print $2, $3}'

Use the image's Moby (from Microsoft's package repository) if it is present. Installing docker-ce on top of moby-engine conflicts. Only if neither is installed, use Docker's official repository:

# [VM] fallback only
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list
sudo apt-get update && apt-cache madison docker-ce | head -3      # choose and pin an exact version string
sudo apt-get install -y docker-ce=<VERSION> docker-ce-cli=<VERSION> containerd.io docker-buildx-plugin

2.4 NVIDIA Container Toolkit — [VM]

nvidia-ctk --version 2>/dev/null || echo "not installed"

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
apt-cache madison nvidia-container-toolkit | head -3
export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.20.1-1
sudo apt-get install -y nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
  nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
  libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}
sudo apt-mark hold nvidia-container-toolkit nvidia-container-toolkit-base libnvidia-container-tools libnvidia-container1

sudo nvidia-ctk runtime configure --runtime=docker
# Add log rotation without clobbering the runtime entry nvidia-ctk just wrote
sudo apt-get install -y jq
sudo jq '. + {"log-driver":"json-file","log-opts":{"max-size":"50m","max-file":"5"}}' /etc/docker/daemon.json | sudo tee /etc/docker/daemon.json.new >/dev/null
sudo mv /etc/docker/daemon.json.new /etc/docker/daemon.json && sudo systemctl restart docker
cat /etc/docker/daemon.json

If the image already ships a toolkit, keep its version unless it is older than 1.17, and record the version. Success: daemon.json contains both runtimes.nvidia and log-opts. Failure: docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]] means nvidia-ctk runtime configure did not run or Docker was not restarted.

2.5 Container GPU access and CUDA compatibility — [VM]

export VLLM_IMAGE_TAG=v0.31.0-x86_64
docker pull vllm/vllm-openai:${VLLM_IMAGE_TAG}
docker run --rm --gpus all --entrypoint bash vllm/vllm-openai:${VLLM_IMAGE_TAG} -c \
  'nvidia-smi -L; echo CUDA_VERSION=$CUDA_VERSION; python3 -c "import torch,vllm;print(\"torch\",torch.__version__,\"cuda\",torch.version.cuda,\"gpus\",torch.cuda.device_count(),\"vllm\",vllm.__version__); print(\"p2p0-7\",torch.cuda.can_device_access_peer(0,7))"'

Success: 8 GPUs listed; CUDA_VERSION is 13.0.x or lower; torch.version.cuda is 13.0 or lower; gpus 8; vllm 0.31.0; p2p0-7 True. Failure: CUDA driver version is insufficient for CUDA runtime version means the container CUDA is newer than the driver supports. Use v0.31.0-x86_64-cu129 and rerun.

2.6 Capacity and memory checks — [VM]

df -h / /mnt/kimi
free -g | head -2                                  # expect ~1,850 GiB host RAM
docker system df

Success: / has at least 150 GiB free after the vLLM image (about 20–25 GB [EXP]), and /mnt/kimi has at least 700 GiB free.

3. Reproducible vLLM deployment

vLLM serves moonshotai/Kimi-K2.5 at a pinned commit from local NVMe. It runs in a digest-pinned container, listens only on 127.0.0.1:8000, and is managed by systemd. Every flag below comes from the vLLM K2.5 recipe or from the memory plan.

3.1 Pin and record the model revision and container digest — [VM]

mkdir -p ~/kimi && cd ~/kimi
export MODEL_REPO=moonshotai/Kimi-K2.5
export MODEL_REV=$(curl -fsS "https://huggingface.co/api/models/${MODEL_REPO}/revision/main" | jq -r .sha)
echo "$MODEL_REV"          # observed 2026-10-20: 4d01dfe0332d63057c186e0b262165819efb6611

export VLLM_IMAGE_TAG=v0.31.0-x86_64
docker pull vllm/vllm-openai:${VLLM_IMAGE_TAG}
export VLLM_DIGEST=$(docker inspect --format '{{index .RepoDigests 0}}' vllm/vllm-openai:${VLLM_IMAGE_TAG} | cut -d@ -f2)
docker buildx imagetools inspect vllm/vllm-openai:${VLLM_IMAGE_TAG} | head -5   # cross-check against the registry
echo "$VLLM_DIGEST"        # observed on Docker Hub 2026-10-20: sha256:a4a4c0437bf7240089da5f08aa370c4aee17ae5290f7a3b468825ee26c4c3a6b

cat > ~/kimi/PINS.env <<EOF
MODEL_REPO=${MODEL_REPO}
MODEL_REV=${MODEL_REV}
VLLM_IMAGE=vllm/vllm-openai@${VLLM_DIGEST}
VLLM_IMAGE_TAG=${VLLM_IMAGE_TAG}
SERVED_ALIAS=kimi-work
RECORDED_AT=$(date -u +%FT%TZ)
EOF
sudo install -m 0644 ~/kimi/PINS.env /etc/kimi-pins.env

Success: MODEL_REV is 40 hex characters, and VLLM_DIGEST starts with sha256: and matches imagetools. The "observed" values come from the public APIs on 2026-10-20. They are not proof of what you will pull. Your recorded values are the pins. If main has moved since, decide deliberately whether to adopt the new commit, and record why.

3.2 Download the weights once — [VM]

The repository is not gated (gated: false, HF API), so no HF token is needed.

set -a; . /etc/kimi-pins.env; set +a
DEST=/mnt/kimi/models/Kimi-K2.5
mkdir -p "$DEST"
docker run --rm -v /mnt/kimi:/mnt/kimi --entrypoint hf "$VLLM_IMAGE" \
  download "$MODEL_REPO" --revision "$MODEL_REV" --local-dir "$DEST"
echo "$MODEL_REV" > "$DEST/.revision"

Verify against the hub's own sizes and LFS SHA-256s:

curl -fsS "https://huggingface.co/api/models/${MODEL_REPO}/tree/${MODEL_REV}?recursive=1" > ~/kimi/tree.json
jq -r '.[] | select(.type=="file") | "\(.path)\t\(.size)\t\(.lfs.oid // "-")"' ~/kimi/tree.json > ~/kimi/manifest.tsv
wc -l ~/kimi/manifest.tsv; ls "$DEST"/*.safetensors | wc -l          # expect 64 shards
fail=0; while IFS=$'\t' read -r p s oid; do
  a=$(stat -c %s "$DEST/$p" 2>/dev/null || echo MISSING); [ "$a" = "$s" ] || { echo "SIZE $p $a != $s"; fail=1; }
done < ~/kimi/manifest.tsv; echo "size check fail=$fail"
# SHA-256 of LFS files (several minutes; parallel)
awk -F'\t' '$3!="-"{print $3"  "$1}' ~/kimi/manifest.tsv > ~/kimi/lfs.sha256
(cd "$DEST" && sha256sum --quiet -c ~/kimi/lfs.sha256) && echo "SHA OK"

Success: size check fail=0, 64 shards, SHA OK. Total is about 595 GB. Failure: a mismatched file gets deleted and re-downloaded with the same hf download command, which resumes. If the tree API rate-limits you, wait and retry. Do not skip verification.

3.3 Seed the durable Blob copy, then drop write access

# [VM]
azcopy --version || { wget -q https://packages.microsoft.com/config/ubuntu/24.04/packages-microsoft-prod.deb && sudo dpkg -i packages-microsoft-prod.deb && sudo apt-get update && sudo apt-get install -y azcopy; }
export AZCOPY_AUTO_LOGIN_TYPE=MSI
azcopy copy "/mnt/kimi/models/Kimi-K2.5/*" "https://${SA}.blob.core.windows.net/models/Kimi-K2.5/${MODEL_REV}/" --recursive
azcopy list "https://${SA}.blob.core.windows.net/models/Kimi-K2.5/${MODEL_REV}/" --machine-readable | wc -l

SA is the storage account name from 1.6. Copy it onto the VM: echo SA=<name> | sudo tee -a /etc/kimi-pins.env.

# [CS] downgrade the VM identity to read-only on the container
az role assignment create --assignee-object-id "$PID" --assignee-principal-type ServicePrincipal --role "Storage Blob Data Reader" --scope "$C_SCOPE"
az role assignment delete --assignee "$PID" --role "Storage Blob Data Contributor" --scope "$C_SCOPE"

Success: the object count roughly matches manifest.tsv plus .revision. A later azcopy copy to the container fails with AuthorizationPermissionMismatch; that is the intended outcome.

3.4 Secrets at service start, never on disk — [VM]

sudo tee /usr/local/sbin/kimi-secrets.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
# Fetches the vLLM API key from Key Vault with the VM's managed identity into tmpfs.
set -euo pipefail
. /etc/kimi-pins.env
TOKEN=$(curl -fsS -H Metadata:true "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fvault.azure.net" | jq -r .access_token)
KEY=$(curl -fsS -H "Authorization: Bearer $TOKEN" "https://${KV}.vault.azure.net/secrets/vllm-api-key?api-version=7.4" | jq -r .value)
install -d -m 0700 /run/kimi
printf 'VLLM_API_KEY=%s\n' "$KEY" > /run/kimi/vllm.env
chmod 0600 /run/kimi/vllm.env
EOF
sudo chmod 0750 /usr/local/sbin/kimi-secrets.sh
echo "KV=<your-vault-name>" | sudo tee -a /etc/kimi-pins.env
sudo /usr/local/sbin/kimi-secrets.sh && sudo stat -c '%a %U %n' /run/kimi/vllm.env

Success: 600 root /run/kimi/vllm.env. /run is tmpfs, so the key disappears at every deallocation and is re-fetched at the next start. Rotation: az keyvault secret set --vault-name $KV -n vllm-api-key --value "$(openssl rand -hex 32)" [CS], then sudo systemctl restart kimi-vllm [VM], then update the one client that holds it.

3.5 The service — [VM]

sudo tee /etc/systemd/system/kimi-vllm.service >/dev/null <<'EOF'
[Unit]
Description=vLLM serving Kimi-K2.5 (alias kimi-work) on 127.0.0.1:8000
Requires=docker.service kimi-restore.service
After=docker.service kimi-restore.service nvidia-fabricmanager.service network-online.target
StartLimitIntervalSec=3600
StartLimitBurst=3

[Service]
Type=simple
EnvironmentFile=/etc/kimi-pins.env
ExecStartPre=/usr/local/sbin/kimi-secrets.sh
ExecStartPre=-/usr/bin/docker rm -f kimi-vllm
ExecStartPre=-/usr/bin/docker network create --internal=false kimi-net
ExecStart=/usr/bin/docker run --name kimi-vllm --rm \
  --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \
  --network kimi-net -p 127.0.0.1:8000:8000 \
  --env-file /run/kimi/vllm.env \
  -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 \
  -v /mnt/kimi/models/Kimi-K2.5:/models/Kimi-K2.5:ro \
  -v /var/lib/kimi/vllm-cache:/root/.cache/vllm \
  --health-cmd "python3 -c \"import urllib.request;urllib.request.urlopen('http://127.0.0.1:8000/health',timeout=5)\"" \
  --health-interval 30s --health-start-period 3600s --health-retries 3 \
  ${VLLM_IMAGE} \
  --model /models/Kimi-K2.5 \
  --served-model-name ${SERVED_ALIAS} \
  --host 0.0.0.0 --port 8000 \
  --tensor-parallel-size 8 \
  --max-model-len 32768 \
  --max-num-seqs 4 \
  --gpu-memory-utilization 0.90 \
  --kv-cache-dtype auto \
  --mm-encoder-tp-mode data \
  --tool-call-parser kimi_k2 --enable-auto-tool-choice \
  --reasoning-parser kimi_k2 \
  --trust-remote-code
ExecStop=/usr/bin/docker stop -t 120 kimi-vllm
Restart=on-failure
RestartSec=60
TimeoutStartSec=infinity
TimeoutStopSec=180

[Install]
WantedBy=multi-user.target
EOF
sudo mkdir -p /var/lib/kimi/vllm-cache
sudo sed -i 's/^#\?SystemMaxUse=.*/SystemMaxUse=2G/' /etc/systemd/journald.conf && sudo systemctl restart systemd-journald
sudo systemctl daemon-reload

kimi-restore.service is defined in Section 6.2. Until it exists, comment out its two references and start manually. Flag rationale:

--trust-remote-code justification. The checkpoint declares custom classes (auto_map → modeling_kimi_k25, tokenization_kimi.py, kimi_k25_processor.py), and both the recipe and the NVIDIA card pass this flag. The risk is that repository Python executes on your VM. It is contained by (a) the pinned commit, (b) offline mode, (c) a read-only mount, and (d) your review before first start: ls /mnt/kimi/models/Kimi-K2.5/*.py should list exactly 9 files totalling about 205 KB, matching the hub tree. Skim them for network or subprocess calls: grep -nE 'subprocess|socket|requests|urllib|os\.system' /mnt/kimi/models/Kimi-K2.5/*.py.

Licence. The weights and code are under a Modified MIT licence (LICENSE). It is MIT, plus a requirement to "prominently display 'Kimi K2.5'" in the UI of commercial products with more than 100 million MAU or more than USD 20 million monthly revenue. Personal daily work does not trigger that clause, but keep the copyright notice with any copy. If you choose the NVFP4 checkpoint, the NVIDIA Open Model License applies as well. Neither licence requires accepting a gate on Hugging Face for K2.5.

3.6 Start and observe — [VM]

date -u +%T; sudo systemctl start kimi-vllm
journalctl -u kimi-vllm -f -o cat | grep -E --line-buffered 'Loading|weights|KV cache|CUDA graph|Started server|ERROR|Traceback'
# in a second shell:
until curl -fsS http://127.0.0.1:8000/health >/dev/null; do sleep 10; done; date -u +%T

Illustrative expected output (not from a real run): weight-loading progress across 8 workers; a line reporting available KV-cache memory and maximum concurrency for 32,768 tokens; CUDA-graph capture; then Application startup complete.

After the first successful start: sudo systemctl enable kimi-vllm.

3.7 Smoke tests — [VM]

set -a; . /run/kimi/vllm.env; set +a
B=http://127.0.0.1:8000
curl -s -o /dev/null -w '%{http_code}\n' $B/v1/models                                   # expect 401
curl -s -H "Authorization: Bearer wrong" -o /dev/null -w '%{http_code}\n' $B/v1/models   # expect 401
curl -fsS -H "Authorization: Bearer $VLLM_API_KEY" $B/v1/models | jq -r '.data[].id'     # expect kimi-work
curl -fsS -H "Authorization: Bearer $VLLM_API_KEY" -H 'Content-Type: application/json' $B/v1/chat/completions -d '{
  "model":"kimi-work","max_tokens":64,"temperature":0.6,
  "messages":[{"role":"user","content":"Reply with the single word: ready"}],
  "chat_template_kwargs":{"thinking":false}}' | jq '{content:.choices[0].message.content, usage}'
ss -ltnp | grep 8000                                                                    # expect 127.0.0.1:8000 only

Success: 401, 401, kimi-work, content containing "ready", and the listener on 127.0.0.1:8000 only. Instant mode uses chat_template_kwargs: {thinking: false} and temperature 0.6. Thinking mode uses temperature 1.0 and top_p 0.95, per the model card.

3.8 Workload benchmark — [VM]

No official published H200 throughput or latency figures for K2.5 were found. Every number in this table must come from your own run. Leave the cells empty until then.

# Terminal 1: sample VRAM every second during the benchmark
nvidia-smi --query-gpu=timestamp,index,memory.used,utilization.gpu --format=csv,noheader -lms 1000 > ~/kimi/vram.csv &
# Terminal 2: chat-shaped load, concurrency 1 then 2
for c in 1 2; do
 docker exec kimi-vllm vllm bench serve --backend openai-chat --endpoint /v1/chat/completions \
   --base-url http://127.0.0.1:8000 --model kimi-work --tokenizer /models/Kimi-K2.5 --trust-remote-code \
   --dataset-name random --random-input-len 4096 --random-output-len 512 \
   --num-prompts 20 --max-concurrency $c --header "Authorization=Bearer $VLLM_API_KEY" \
   --save-result --result-dir /root/.cache/vllm/bench --result-filename c${c}.json
done
kill %1; awk -F', ' '{gsub(/ MiB/,"",$3); if($3>m[$2])m[$2]=$3} END{for(g in m)print "GPU",g,"peak MiB",m[g]}' ~/kimi/vram.csv

If your vLLM build rejects --header, pass the key through OPENAI_API_KEY=$VLLM_API_KEY in the docker exec -e instead [EXP].

4. Secure remote access

Pick one option. A puts nginx with a Let's Encrypt certificate on the VM and opens only TCP 443 (and 80 for ACME). B opens no inbound port and reaches the VM through a named Cloudflare Tunnel guarded by Cloudflare Access. In both, port 8000 is bound to loopback and never appears in the NSG, only two API routes are reachable, and /health, /metrics, /invocations, /tokenize, /pause and the other routes vLLM leaves unauthenticated (vLLM security doc) return 404.

Recommendation: Option B if you will use the browser front end (Open WebUI) on your phone. Option A if you want a native OpenAI-compatible app, because few apps can send Cloudflare's extra headers (Section 5).

Credentials and where they live

Option A — HTTPS reverse proxy on the VM

A.1 NSG and DNS — [CS] and your DNS host

az network nsg rule create -g "$RG" --nsg-name "$NSG" -n allow-https --priority 200 --direction Inbound \
  --access Allow --protocol Tcp --source-address-prefixes Internet --destination-port-ranges 443
az network nsg rule create -g "$RG" --nsg-name "$NSG" -n allow-acme-http --priority 210 --direction Inbound \
  --access Allow --protocol Tcp --source-address-prefixes Internet --destination-port-ranges 80
az network public-ip show -g "$RG" -n pip-kimi --query ipAddress -o tsv     # create A records api.<domain> and chat.<domain>

If the domain is on Cloudflare DNS, set both records to DNS only (grey cloud). Otherwise Cloudflare would terminate TLS, which is Option B's trust model without its controls.

A.2 Install nginx and certbot — [VM]

sudo apt-get install -y nginx
sudo snap install --classic certbot && sudo ln -sf /snap/bin/certbot /usr/bin/certbot
sudo mkdir -p /var/www/certbot
az keyvault secret set --vault-name "$KV" -n gateway-api-key --value "$(openssl rand -hex 32)" -o none   # [CS]

A.3 Key material generated at boot — [VM]

sudo tee /usr/local/sbin/kimi-nginx-keys.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
. /etc/kimi-pins.env
tok(){ curl -fsS -H Metadata:true "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fvault.azure.net" | jq -r .access_token; }
get(){ curl -fsS -H "Authorization: Bearer $(tok)" "https://${KV}.vault.azure.net/secrets/$1?api-version=7.4" | jq -r .value; }
GW=$(get gateway-api-key); UP=$(get vllm-api-key)
install -d -m 0750 -g www-data /run/kimi
umask 027
cat > /run/kimi/nginx-keys.conf <<CONF
map \$http_authorization \$kimi_authorized { default 0; "Bearer ${GW}" 1; }
map \$host \$vllm_upstream_auth { default "Bearer ${UP}"; }
CONF
chgrp www-data /run/kimi/nginx-keys.conf
EOF
sudo chmod 0750 /usr/local/sbin/kimi-nginx-keys.sh
sudo mkdir -p /etc/systemd/system/nginx.service.d
printf '[Service]\nExecStartPre=/usr/local/sbin/kimi-nginx-keys.sh\nExecReload=\nExecReload=/usr/local/sbin/kimi-nginx-keys.sh\nExecReload=/usr/sbin/nginx -s reload\n' | sudo tee /etc/systemd/system/nginx.service.d/kimi.conf
sudo systemctl daemon-reload

The phone holds only the gateway key. nginx swaps it for the vLLM key on the way upstream, so the vLLM key never leaves the VM.

A.4 Certificate — [VM]

export API_HOST=api.<domain>; export CHAT_HOST=chat.<domain>
sudo tee /etc/nginx/conf.d/00-acme.conf >/dev/null <<EOF
server { listen 80; server_name ${API_HOST} ${CHAT_HOST};
  location /.well-known/acme-challenge/ { root /var/www/certbot; }
  location / { return 301 https://\$host\$request_uri; } }
EOF
sudo rm -f /etc/nginx/sites-enabled/default && sudo systemctl restart nginx
sudo certbot certonly --webroot -w /var/www/certbot -d "$API_HOST" -d "$CHAT_HOST" \
  --agree-tos -m <YOUR_EMAIL> --no-eff-email --deploy-hook "systemctl reload nginx"
sudo certbot renew --dry-run

Success: certificate files under /etc/letsencrypt/live/$API_HOST/, and the dry run reports success. The VM is usually off, but snap's certbot timer renews within its 30-day window whenever the VM runs. If a certificate does expire, the next boot renews it before clients notice anything beyond one failed request.

A.5 Route allowlist, auth, limits and streaming — [VM]

sudo tee /etc/nginx/conf.d/10-kimi.conf >/dev/null <<EOF
include /run/kimi/nginx-keys.conf;
limit_req_zone \$binary_remote_addr zone=kimi_rl:10m rate=30r/m;
limit_req_zone \$binary_remote_addr zone=login_rl:10m rate=10r/m;
limit_conn_zone \$binary_remote_addr zone=kimi_conn:10m;
limit_req_status 429; limit_conn_status 429;
log_format kimi_meta '\$time_iso8601 \$remote_addr "\$request_method \$uri" \$status \$body_bytes_sent \$request_time';
upstream vllm { server 127.0.0.1:8000; keepalive 8; }

server {
  listen 443 ssl; http2 on; server_name ${API_HOST};
  ssl_certificate /etc/letsencrypt/live/${API_HOST}/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/${API_HOST}/privkey.pem;
  ssl_protocols TLSv1.2 TLSv1.3;
  add_header Strict-Transport-Security "max-age=31536000" always;
  access_log /var/log/nginx/kimi_api.log kimi_meta;      # metadata only, never bodies
  client_max_body_size 20m;                               # room for base64 images
  proxy_http_version 1.1; proxy_set_header Connection "";
  proxy_set_header Authorization \$vllm_upstream_auth;

  location = /v1/models {
    limit_except GET { deny all; }
    if (\$kimi_authorized = 0) { return 401; }
    proxy_pass http://vllm;
  }
  location = /v1/chat/completions {
    limit_except POST { deny all; }
    if (\$kimi_authorized = 0) { return 401; }
    limit_req zone=kimi_rl burst=10 nodelay; limit_conn kimi_conn 4;
    proxy_buffering off; proxy_cache off; gzip off;      # SSE must not be buffered
    proxy_read_timeout 900s; proxy_send_timeout 900s;
    proxy_pass http://vllm;
  }
  location / { return 404; }
}

server {
  listen 443 ssl; http2 on; server_name ${CHAT_HOST};
  ssl_certificate /etc/letsencrypt/live/${API_HOST}/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/${API_HOST}/privkey.pem;
  ssl_protocols TLSv1.2 TLSv1.3;
  add_header Strict-Transport-Security "max-age=31536000" always;
  access_log /var/log/nginx/kimi_chat.log kimi_meta;
  client_max_body_size 50m;
  location /api/v1/auths/signin { limit_req zone=login_rl burst=5 nodelay; proxy_pass http://127.0.0.1:3000; }
  location / {
    proxy_pass http://127.0.0.1:3000;
    proxy_http_version 1.1; proxy_set_header Upgrade \$http_upgrade; proxy_set_header Connection "upgrade";
    proxy_set_header Host \$host; proxy_set_header X-Forwarded-Proto https;
    proxy_buffering off; proxy_read_timeout 900s;
  }
}
EOF
sudo nginx -t && sudo systemctl restart nginx

A.6 Tests — [WS] (any machine off the VM; the phone tests are in Section 5)

export API=https://api.<domain>; export GW=<gateway-api-key>
curl -s -o /dev/null -w 'noauth %{http_code}\n' $API/v1/models                                    # 401
curl -s -o /dev/null -w 'badauth %{http_code}\n' -H 'Authorization: Bearer nope' $API/v1/models    # 401
curl -s -H "Authorization: Bearer $GW" $API/v1/models | jq -r '.data[].id'                         # kimi-work
for p in /health /metrics /invocations /tokenize /pause /v1/completions /docs /openapi.json; do
  printf '%-18s %s\n' $p "$(curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $GW" $API$p)"; done   # all 404
curl -s -o /dev/null -w 'GET-on-POST %{http_code}\n' -H "Authorization: Bearer $GW" $API/v1/chat/completions  # 403
IP=$(dig +short api.<domain>); timeout 5 bash -c "</dev/tcp/$IP/8000" && echo OPEN-FAIL || echo "8000 closed: PASS"
# Streaming: lines must arrive spread over time, not all at once at the end
curl -sN $API/v1/chat/completions -H "Authorization: Bearer $GW" -H 'Content-Type: application/json' \
  -d '{"model":"kimi-work","stream":true,"max_tokens":200,"messages":[{"role":"user","content":"Count to 40 slowly."}],"chat_template_kwargs":{"thinking":false}}' \
  | while IFS= read -r l; do printf '%s %s\n' "$(date +%T.%3N)" "${l:0:60}"; done | head -8
# Rate limit: expect some 429 after the burst
for i in $(seq 1 25); do curl -s -o /dev/null -w '%{http_code} ' -X POST -H "Authorization: Bearer $GW" $API/v1/chat/completions -d '{}'; done; echo

Pass: 401, 401, kimi-work, eight 404s, 403, 8000 closed: PASS, streaming timestamps spread across at least 1 second and ending in data: [DONE], and at least one 429. The streaming read timeout is 900 s at nginx. vLLM has no proxy-imposed limit beyond that.

Who can read plaintext in A: only the VM (nginx and vLLM). TLS runs from the phone to nginx.

Option B — named Cloudflare Tunnel with Access

Tunnel is transport; Access is authorisation. cloudflared makes outbound-only connections from the VM to Cloudflare, so no inbound port is needed. That alone authenticates nobody. Cloudflare Access policies decide who may reach each hostname, and cloudflared checks Access's signed JWT (Cf-Access-Jwt-Assertion) before forwarding, through originRequest.access (origin parameters). vLLM's own bearer key is a third, independent layer.

Prerequisites: a Cloudflare account; a domain whose DNS zone is active on Cloudflare (nameservers moved) so cloudflared tunnel route dns can create the hostnames; and a Zero Trust organisation, which gives you the <TEAM> name in <TEAM>.cloudflareaccess.com.

B.1 NSG — [CS]: no change. No 80 or 443. Confirm with az network nsg rule list -g "$RG" --nsg-name "$NSG" -o table.

B.2 Install cloudflared from Cloudflare's repository — [VM] (pkg.cloudflare.com)

sudo mkdir -p --mode=0755 /usr/share/keyrings
curl -fsSL https://pkg.cloudflare.com/cloudflare-main.gpg | sudo tee /usr/share/keyrings/cloudflare-main.gpg >/dev/null
echo 'deb [signed-by=/usr/share/keyrings/cloudflare-main.gpg] https://pkg.cloudflare.com/cloudflared noble main' | sudo tee /etc/apt/sources.list.d/cloudflared.list
sudo apt-get update && apt-cache madison cloudflared | head -3
sudo apt-get install -y cloudflared=<PINNED_VERSION> && sudo apt-mark hold cloudflared
cloudflared --version

B.3 Create the named tunnel and hostnames — [VM]

cloudflared tunnel login                      # prints a URL: open it on your workstation, pick the zone
cloudflared tunnel create kimi                # prints the tunnel UUID and writes ~/.cloudflared/<UUID>.json
export TUNNEL_ID=<UUID>
cloudflared tunnel route dns kimi api.<domain>
cloudflared tunnel route dns kimi chat.<domain>
sudo install -d -m 0755 /etc/cloudflared
sudo install -m 0600 ~/.cloudflared/${TUNNEL_ID}.json /etc/cloudflared/
shred -u ~/.cloudflared/cert.pem              # account-level cert: not needed to run the tunnel

Back up the credentials JSON to Key Vault from your workstation with az keyvault secret set --vault-name $KV -n cf-tunnel-cred --file <UUID>.json.

B.4 Access applications — Cloudflare dashboard (workstation)

In Zero Trust → Access → Applications → Add → Self-hosted:

Create the service token under Access → Service credentials → Service Tokens with a 1-year duration. Copy the Client ID and Secret once; the secret is not shown again. Without the Service Auth action, Access would show a login page to the API client instead (service tokens doc). Copy each app's Application Audience (AUD) tag for the config below.

B.5 Ingress with route allowlist and origin JWT check — [VM]

sudo tee /etc/cloudflared/config.yml >/dev/null <<EOF
tunnel: ${TUNNEL_ID}
credentials-file: /etc/cloudflared/${TUNNEL_ID}.json
originRequest:
  connectTimeout: 10s
ingress:
  - hostname: api.<domain>
    path: ^/v1/(models|chat/completions)$
    service: http://127.0.0.1:8000
    originRequest:
      access: { required: true, teamName: <TEAM>, audTag: [ <AUD_API> ] }
  - hostname: chat.<domain>
    service: http://127.0.0.1:3000
    originRequest:
      access: { required: true, teamName: <TEAM>, audTag: [ <AUD_CHAT> ] }
  - service: http_status:404
EOF
cloudflared tunnel --config /etc/cloudflared/config.yml ingress validate
cloudflared tunnel --config /etc/cloudflared/config.yml ingress rule https://api.<domain>/v1/models    # matches rule 0
cloudflared tunnel --config /etc/cloudflared/config.yml ingress rule https://api.<domain>/metrics      # matches catch-all 404
sudo cloudflared --config /etc/cloudflared/config.yml service install
sudo systemctl enable --now cloudflared && systemctl is-active cloudflared
cloudflared tunnel info kimi                  # shows active connections to Cloudflare edge

The path value is a Go regular expression (config file doc). The tunnel cannot restrict methods; vLLM rejects a GET on /chat/completions with 405.

B.6 Two separate authentication headers. An API client must send both:

• CF-Access-Client-Id and CF-Access-Client-Secret, checked by Cloudflare Access.

• Authorization: Bearer <vllm-api-key>, checked by vLLM.

That Access forwards Authorization untouched is [EXP]; the test below proves it. Do not use Access's single-header mode (read_service_tokens_from_header): it uses the Authorization header and would collide with the bearer key.

B.7 Tests — [WS]

export API=https://api.<domain>; export CID=<client-id>; export CSEC=<client-secret>; export K=<vllm-api-key>
curl -s -o /dev/null -w 'no-cf %{http_code} -> %{redirect_url}\n' $API/v1/models                         # 302 to <TEAM>.cloudflareaccess.com, or 403
curl -s -o /dev/null -w 'bad-cf %{http_code}\n' -H "CF-Access-Client-Id: $CID" -H 'CF-Access-Client-Secret: wrong' $API/v1/models   # non-2xx
curl -s -o /dev/null -w 'cf-no-bearer %{http_code}\n' -H "CF-Access-Client-Id: $CID" -H "CF-Access-Client-Secret: $CSEC" $API/v1/models   # 401 from vLLM
curl -s -H "CF-Access-Client-Id: $CID" -H "CF-Access-Client-Secret: $CSEC" -H "Authorization: Bearer $K" $API/v1/models | jq -r '.data[].id'   # kimi-work
for p in /health /metrics /invocations /tokenize /pause /v1/completions; do
  printf '%-16s %s\n' $p "$(curl -s -o /dev/null -w '%{http_code}' -H "CF-Access-Client-Id: $CID" -H "CF-Access-Client-Secret: $CSEC" -H "Authorization: Bearer $K" $API$p)"; done   # all 404
curl -sN $API/v1/chat/completions -H "CF-Access-Client-Id: $CID" -H "CF-Access-Client-Secret: $CSEC" -H "Authorization: Bearer $K" \
  -H 'Content-Type: application/json' -d '{"model":"kimi-work","stream":true,"max_tokens":200,"messages":[{"role":"user","content":"Count to 40 slowly."}],"chat_template_kwargs":{"thinking":false}}' \
  | while IFS= read -r l; do printf '%s %s\n' "$(date +%T.%3N)" "${l:0:60}"; done | head -8
# Port 8000 and 443 on the VM's public IP must both be closed
IP=$(az network public-ip show -g "$RG" -n pip-kimi --query ipAddress -o tsv)
for port in 8000 443 80; do timeout 5 bash -c "</dev/tcp/$IP/$port" && echo "$port OPEN-FAIL" || echo "$port closed: PASS"; done
# Origin JWT check: the VM's own journal must show the rejected attempts never reached vLLM

Pass: the first two are non-2xx and absent from journalctl -u kimi-vllm; the third is 401; the fourth returns kimi-work; six 404s; streaming timestamps spread out; ports 8000, 443 and 80 closed.

Timeouts. Cloudflare returns HTTP 524 if the origin sends nothing for the Proxy Read Timeout, 125 s by default; only Enterprise plans can raise it (524 doc). Always use "stream": true, so tokens, including reasoning tokens, keep bytes flowing. Whether a 32K-token prefill stays under 125 s before the first byte is [EXP]; the 3.8 TTFT measurement answers it.

Who can read plaintext in B: Cloudflare's edge terminates TLS, so Cloudflare can process prompts and responses in plaintext before re-encrypting them to cloudflared. Then cloudflared and vLLM on the VM see them. Choose A if that is unacceptable for your material.

Private-network alternative (any client, no public hostname)

If your chosen phone client cannot send the Access headers and you do not want Option A's open port 443, install Tailscale on the VM and the phone (downloads). Keep the NSG unchanged and publish the Option A nginx api server block, re-pointed to listen on 127.0.0.1:8081 without TLS, through tailscale serve, which provisions a *.ts.net HTTPS certificate. Check the exact syntax with tailscale serve --help [EXP]. Only devices in your tailnet can connect, and the gateway key still applies.

5. Mobile setup and daily work

Recommended client: Open WebUI, run on the VM and used from the phone's browser as an installed web app (iOS Safari or Android Chrome). It works with both access options, and with Option B's interactive Access login. It also supplies what vLLM does not: stored chat history, file upload and retrieval, image attachments and tool execution (Open WebUI features). A native OpenAI-compatible app is the secondary path. It needs Option A, or Option B with custom-header support.

Client support honesty. I verified Open WebUI's features and its v0.12.3 release (16 Oct 2026, releases) from its own pages. I did not find official documentation for any native iOS or Android chat app that both accepts a custom base URL and sends arbitrary headers. Chatbox's custom-provider fields are described only by third-party guides (example). Treat any native app as [EXP] until it passes the tests below.

5.1 Run Open WebUI on the VM — [VM]

export OWUI_TAG=v0.12.3                     # or the newest release that is at least 14 days old
docker pull ghcr.io/open-webui/open-webui:${OWUI_TAG}
export OWUI_DIGEST=$(docker inspect --format '{{index .RepoDigests 0}}' ghcr.io/open-webui/open-webui:${OWUI_TAG} | cut -d@ -f2)
echo "OWUI_IMAGE=ghcr.io/open-webui/open-webui@${OWUI_DIGEST}" | sudo tee -a /etc/kimi-pins.env
az keyvault secret set --vault-name "$KV" -n owui-secret-key --value "$(openssl rand -hex 32)" -o none   # [CS]

sudo tee /etc/systemd/system/kimi-webui.service >/dev/null <<'EOF'
[Unit]
Description=Open WebUI on 127.0.0.1:3000
Requires=docker.service
After=docker.service kimi-vllm.service
[Service]
EnvironmentFile=/etc/kimi-pins.env
ExecStartPre=-/usr/bin/docker rm -f open-webui
ExecStartPre=-/usr/bin/docker network create kimi-net
ExecStartPre=/bin/sh -c '. /etc/kimi-pins.env; T=$(curl -fsS -H Metadata:true "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%%3A%%2F%%2Fvault.azure.net" | jq -r .access_token); install -d -m 0700 /run/kimi; printf "WEBUI_SECRET_KEY=%%s\\n" "$(curl -fsS -H "Authorization: Bearer $T" "https://$KV.vault.azure.net/secrets/owui-secret-key?api-version=7.4" | jq -r .value)" > /run/kimi/owui.env; chmod 600 /run/kimi/owui.env'
ExecStart=/usr/bin/docker run --name open-webui --rm --network kimi-net -p 127.0.0.1:3000:8080 \
  --env-file /run/kimi/owui.env -e WEBUI_URL=https://chat.<domain> \
  -v /var/lib/kimi/open-webui:/app/backend/data ${OWUI_IMAGE}
ExecStop=/usr/bin/docker stop -t 30 open-webui
Restart=on-failure
RestartSec=15
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload && sudo systemctl enable --now kimi-webui
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3000/       # 200

Then in Open WebUI, from the phone or workstation:

1. Open https://chat.<domain>. Option B passes the Access email PIN first. Create the first account, which becomes admin, with a strong password from your password manager.

2. Admin Panel → Settings → General: disable new sign-ups.

3. Admin Panel → Settings → Connections → OpenAI API → +: URL http://kimi-vllm:8000/v1, key = vllm-api-key. This runs server-side on the VM, so the vLLM key never reaches the phone. Disable the Ollama connection.

4. Verify: the model picker lists kimi-work.

5. Model settings for kimi-work: temperature 0.6 with chat_template_kwargs {"thinking": false} for quick work, or 1.0 with thinking on for hard problems (model card). Turn on native function calling if you will use tools.

Install on the phone: iOS Safari → Share → Add to Home Screen; Android Chrome → ⋮ → Install app / Add to Home screen.

5.2 Native API client fields (Option A, or B with header support) — [PH]

Does the app append /v1? Apps differ, and guides for the same app disagree. Settle it empirically: tap Fetch models, then on the VM run sudo tail -n 5 /var/log/nginx/kimi_api.log (Option A). The logged path must be exactly /v1/models. If you see /v1/v1/models, drop /v1 from the base URL. If you see /models, add it.

5.3 Mobile-compatible test requests — [PH]

Run them from any phone HTTP tool: iOS Shortcuts Get Contents of URL, Termux curl on Android, or a REST client app. Set the method to POST, add headers Content-Type: application/json plus your option's auth headers, and paste the body. URL means https://api.<domain>/v1.

1. Model discovery: GET URL/models. Pass: JSON containing "id": "kimi-work".

2. Ordinary chat — POST URL/chat/completions

{"model":"kimi-work","max_tokens":128,"temperature":0.6,
 "chat_template_kwargs":{"thinking":false},
 "messages":[{"role":"user","content":"In one sentence, what is a mixture-of-experts model?"}]}

Pass: choices[0].message.content is one sentence, and finish_reason is stop.

3. Streaming: the same body plus "stream": true. Pass: the app shows text arriving progressively; with curl, data: lines arrive spread out and end in data: [DONE]. With thinking on, reasoning arrives in a separate reasoning_content field (from --reasoning-parser kimi_k2). Apps that ignore that field show nothing until the answer starts.

4. System message and history — vLLM is stateless; history works only because the client resends it.

{"model":"kimi-work","max_tokens":64,"chat_template_kwargs":{"thinking":false},
 "messages":[
  {"role":"system","content":"You are terse. Answer in at most five words."},
  {"role":"user","content":"My project codename is BLUEGUM."},
  {"role":"assistant","content":"Noted: BLUEGUM."},
  {"role":"user","content":"What is my project codename?"}]}

Pass: the answer contains BLUEGUM in five words or fewer.

5a. Tool-call generation

{"model":"kimi-work","max_tokens":256,"chat_template_kwargs":{"thinking":false},
 "tools":[{"type":"function","function":{"name":"get_exchange_rate",
   "description":"Get the AUD exchange rate for a currency code",
   "parameters":{"type":"object","properties":{"code":{"type":"string"}},"required":["code"]}}}],
 "tool_choice":"auto",
 "messages":[{"role":"user","content":"What is the AUD rate for USD right now?"}]}

Pass: finish_reason is tool_calls, and message.tool_calls[0].function.name is get_exchange_rate, with arguments parsing as JSON containing "code":"USD". Copy the returned tool_calls[0].id.

5b. Return a tool result — resend history plus your result:

{"model":"kimi-work","max_tokens":128,"chat_template_kwargs":{"thinking":false},
 "tools":[ "…same tools array as 5a…" ],
 "messages":[
  {"role":"user","content":"What is the AUD rate for USD right now?"},
  {"role":"assistant","content":"","tool_calls":[ "…the tool_calls array returned in 5a, verbatim…" ]},
  {"role":"tool","tool_call_id":"<id from 5a>","content":"{\"code\":\"USD\",\"aud_per_unit\":1.52,\"source\":\"TEST FIXTURE\"}"}]}

Pass: the answer states 1.52 and does not invent a source. (1.52 is a test value, not a real rate.)

6. Image input — K2.5 accepts images through MoonViT:

{"model":"kimi-work","max_tokens":128,"chat_template_kwargs":{"thinking":false},
 "messages":[{"role":"user","content":[
   {"type":"text","text":"What object is in this photo? One word."},
   {"type":"image_url","image_url":{"url":"data:image/jpeg;base64,<BASE64_OF_A_SMALL_PHOTO>"}}]}]}

Pass: a plausible one-word answer. A body over 20 MB is rejected by nginx (413) in Option A; shrink the image. In Open WebUI, just attach the photo.

Documents. vLLM's chat API takes text and images, not PDF files. In Open WebUI, attach the PDF or DOCX: Open WebUI extracts the text, chunks and embeds it, and retrieves relevant passages into the prompt. The embedding runs inside the Open WebUI container, on CPU by default [EXP: confirm under Admin → Documents]. Pass: ask about a fact on page 3 of a test PDF, and the answer cites it.

5.4 Who does what

Tools run with the Open WebUI container's network access on your VM. Install only tools you have read, because a tool is code execution triggered by model output.

5.5 Start and deallocate the VM from the phone, without subscription-wide credentials

Create a dedicated operator identity that can only start, deallocate and read this one VM. Use it in the official Azure mobile app, never in a chat app. [CS]:

cat > vm-operator.json <<EOF
{"Name":"VM Operator (kimi)","Description":"Start, deallocate and read one VM",
 "Actions":["Microsoft.Compute/virtualMachines/read","Microsoft.Compute/virtualMachines/instanceView/read",
            "Microsoft.Compute/virtualMachines/start/action","Microsoft.Compute/virtualMachines/deallocate/action"],
 "AssignableScopes":["/subscriptions/$SUB_ID/resourceGroups/$RG"]}
EOF
az role definition create --role-definition @vm-operator.json
# Create a separate Entra user (or use your own account if you accept its wider rights on the phone)
az ad user create --display-name "Kimi Operator" --user-principal-name kimi-operator@<your-tenant-domain> \
  --password "<strong-temporary-password>" --force-change-password-next-sign-in true
az role assignment create --assignee kimi-operator@<your-tenant-domain> --role "VM Operator (kimi)" \
  --scope "$(az vm show -g $RG -n $VM_NAME --query id -o tsv)"

Then enable MFA for that user under Entra → Users → Authentication methods, or through Security defaults.

[PH]: install the Azure app from Microsoft and sign in as kimi-operator. Open Virtual machines → vm-kimi-01 → Start. Stop from the app with Stop, then confirm the status reads Stopped (deallocated); plain Stopped still bills compute. If the app ever shows only Stopped, run az vm deallocate from Cloud Shell, which the Azure app also opens. Stops through this role are always deallocations, because the role grants no powerOff action.

Pass: as kimi-operator, the app can start and deallocate vm-kimi-01, and it shows no other resources, and no delete or resize option succeeds.

6. Cost control, shutdown and recovery

About 45 billed GPU-hours, or roughly 30–37 hours of actual use, after setup, a 10% reserve and three months of fixed costs [CALC]. All figures are Azure retail Linux pay-as-you-go list prices in USD for eastus2, from the Retail Prices API on 2026-10-20. They are not your negotiated or credit-offer prices.

6.1 Prices and assumptions

Fixed cost ≈ $49.66/month whether you use the model or not.

Credit checks [SUB]:

1. Portal → Cost Management → Credits (or the sponsorship portal for sponsored offers): balance, currency and expiry.

2. The offer's terms: whether GPU VMs and Marketplace images are eligible. Ubuntu-HPC has no plan charge (plan: null, step 1.4), so no Marketplace fee applies.

3. If your billing currency is not USD, re-query in it, for example: curl -s "https://prices.azure.com/api/retail/prices?currencyCode='AUD'&\$filter=armSkuName eq 'Standard_ND96isr_H200_v5' and armRegionName eq '$LOCATION' and priceType eq 'Consumption'" | jq '.Items[] | select(.skuName|test("Spot|Low")|not) | {retailPrice,currencyCode}'

6.2 Usable hours [CALC]

Assumptions: USD 5,000 credit, 3 months to expiry, 10% reserve, a one-time setup of 6 VM-hours (download, verify, seed Blob, tests and benchmarks), and per-session overhead of 0.5 h cold start [EXP] plus a 20-minute idle tail before auto-deallocation.

The same budget on Standard_ND96isr_MI300X_v5 at $48.00/h gives about 85 billed hours. Re-run the table once you have measured the real cold start (acceptance criterion 6).

6.3 Deallocate and verify — [CS] or [PH]

az vm deallocate -g "$RG" -n "$VM_NAME"
az vm get-instance-view -g "$RG" -n "$VM_NAME" \
  --query "instanceView.statuses[?starts_with(code,'PowerState')].code" -o tsv

Pass: PowerState/deallocated. Fail: PowerState/stopped means the OS was shut down (for example sudo poweroff inside the VM). Compute is still billed in that state; run the deallocate command.

After deallocation the following continue to bill: the OS disk, the static public IP, the Blob copy, Key Vault and Automation. The local NVMe and temp disk are erased. Compute stops.

6.4 Budget alerts — [CS]

Budgets notify you; they do not stop anything. Cost data can lag by hours, which is why 6.5 and 6.6 exist.

az rest --method put \
  --url "https://management.azure.com/subscriptions/$SUB_ID/providers/Microsoft.Consumption/budgets/kimi-credit?api-version=2023-05-01" \
  --body @- <<EOF
{"properties":{"category":"Cost","amount":4500,"timeGrain":"Annually",
  "timePeriod":{"startDate":"$(date -u +%Y-%m-01)T00:00:00Z","endDate":"<CREDIT_EXPIRY_DATE>T00:00:00Z"},
  "notifications":{
   "a25":{"enabled":true,"operator":"GreaterThan","threshold":25,"contactEmails":["<YOUR_EMAIL>"],"thresholdType":"Actual"},
   "a50":{"enabled":true,"operator":"GreaterThan","threshold":50,"contactEmails":["<YOUR_EMAIL>"],"thresholdType":"Actual"},
   "a80":{"enabled":true,"operator":"GreaterThan","threshold":80,"contactEmails":["<YOUR_EMAIL>"],"thresholdType":"Actual"},
   "f100":{"enabled":true,"operator":"GreaterThan","threshold":100,"contactEmails":["<YOUR_EMAIL>"],"thresholdType":"Forecasted"}}}}
EOF
az rest --method get --url "https://management.azure.com/subscriptions/$SUB_ID/providers/Microsoft.Consumption/budgets/kimi-credit?api-version=2023-05-01" --query properties.amount

Some credit-based offers do not support Consumption budgets. If the PUT returns an offer-type error, track spend in Cost Management → Credits instead [SUB].

6.5 Independent maximum-runtime cap (off the VM)

Two mechanisms that do not depend on the VM's own software.

(a) Daily auto-shutdown, which deallocates — [CS]:

az vm auto-shutdown -g "$RG" -n "$VM_NAME" --time 1200 --email "<YOUR_EMAIL>"   # 12:00 UTC = 23:00 AEDT / 22:00 AEST

(b) Hourly runbook cap of MAX_RUNTIME_HOURS=4. Create an Automation account with a system-assigned identity and give that identity the VM Operator (kimi) role (5.5) on the VM:

az extension add -n automation
az automation account create -g "$RG" -n aa-kimi -l "$LOCATION" --sku Basic
az resource update -g "$RG" -n aa-kimi --resource-type Microsoft.Automation/automationAccounts --set identity.type=SystemAssigned
AA_PID=$(az resource show -g "$RG" -n aa-kimi --resource-type Microsoft.Automation/automationAccounts --query identity.principalId -o tsv)
az role assignment create --assignee-object-id "$AA_PID" --assignee-principal-type ServicePrincipal \
  --role "VM Operator (kimi)" --scope "$(az vm show -g $RG -n $VM_NAME --query id -o tsv)"

In the portal, under Automation account aa-kimi:

1. Variables: create a string variable KimiFirstSeenRunning with an empty value.

2. Runbooks: create a PowerShell 7.2 runbook with the code below, then publish it.

3. Schedules: create an hourly recurrence (Automation's minimum interval), then link it to the runbook.

param([string]$Rg = 'rg-kimi-inference', [string]$Name = 'vm-kimi-01', [double]$MaxHours = 4)
Connect-AzAccount -Identity | Out-Null
$s = (Get-AzVM -ResourceGroupName $Rg -Name $Name -Status).Statuses | Where-Object Code -like 'PowerState/*'
$first = Get-AutomationVariable -Name 'KimiFirstSeenRunning'
$now = (Get-Date).ToUniversalTime()
if ($s.Code -in 'PowerState/running','PowerState/starting','PowerState/stopped') {
  if ([string]::IsNullOrEmpty($first)) { Set-AutomationVariable -Name 'KimiFirstSeenRunning' -Value $now.ToString('o'); return }
  $age = $now - [datetime]::Parse($first).ToUniversalTime()
  if ($age.TotalHours -ge $MaxHours) {
    Write-Output "Deallocating after $([math]::Round($age.TotalHours,2)) h"
    Stop-AzVM -ResourceGroupName $Rg -Name $Name -Force | Out-Null     # Stop-AzVM deallocates by default
    Set-AutomationVariable -Name 'KimiFirstSeenRunning' -Value ''
  }
} elseif (-not [string]::IsNullOrEmpty($first)) { Set-AutomationVariable -Name 'KimiFirstSeenRunning' -Value '' }

Because the check runs hourly, the worst-case overrun is about 1 h beyond the cap (about $85). PowerState/stopped counts as running here, because it still bills. Test: set MaxHours to 0.01, start the VM, and wait for two runs. Pass: the job output shows "Deallocating", and 6.3 reads PowerState/deallocated.

6.6 On-VM idle shutdown with draining — [VM]

GPU utilisation is not a valid idle signal: it is about 0% between a user's messages, and non-zero during CUDA-graph capture at startup. The watcher uses vLLM's own request counters instead.

sudo tee /usr/local/sbin/kimi-idle.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
# Deallocates this VM after IDLE_MIN of no inference activity, with grace and draining.
set -uo pipefail
IDLE_MIN=20; BOOT_GRACE_MIN=60; STATE=/var/lib/kimi/idle.state
m(){ curl -fsS --max-time 5 http://127.0.0.1:8000/metrics 2>/dev/null | awk -v k="$1" '$1 ~ "^"k"({|$)" {s+=$2} END{printf "%d", s+0}'; }
up_min=$(( $(cut -d. -f1 /proc/uptime) / 60 ))
[ "$up_min" -lt "$BOOT_GRACE_MIN" ] && exit 0                                  # boot, restore and load grace
[ "$(systemctl show -p ActiveState --value kimi-restore)" = activating ] && exit 0  # weights still restoring
[ -n "$(who)" ] && exit 0                                                       # an admin is logged in
[ -f /var/lib/kimi/keep-awake ] && exit 0                                       # manual override
run=$(m vllm:num_requests_running); wait=$(m vllm:num_requests_waiting); tok=$(m vllm:prompt_tokens_total)
now=$(date +%s); read -r last_tok last_t < "$STATE" 2>/dev/null || { last_tok=-1; last_t=$now; }
if [ "$run" -gt 0 ] || [ "$wait" -gt 0 ] || [ "$tok" != "$last_tok" ]; then echo "$tok $now" > "$STATE"; exit 0; fi
[ $(( (now - last_t) / 60 )) -lt "$IDLE_MIN" ] && exit 0
logger -t kimi-idle "idle ${IDLE_MIN}m: draining"
systemctl stop nginx cloudflared 2>/dev/null                                   # stop new requests at the edge
for i in $(seq 1 24); do [ "$(m vllm:num_requests_running)" -eq 0 ] && [ "$(m vllm:num_requests_waiting)" -eq 0 ] && break; sleep 5; done
systemctl stop kimi-webui kimi-vllm
RID=$(curl -fsS -H Metadata:true "http://169.254.169.254/metadata/instance/compute/resourceId?api-version=2021-02-01&format=text")
TOK=$(curl -fsS -H Metadata:true "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F" | jq -r .access_token)
logger -t kimi-idle "deallocating $RID"
curl -fsS -X POST -H "Authorization: Bearer $TOK" -H 'Content-Length: 0' "https://management.azure.com${RID}/deallocate?api-version=2024-07-01"
EOF
sudo chmod 0750 /usr/local/sbin/kimi-idle.sh
printf '[Unit]\nDescription=Kimi idle check\n[Service]\nType=oneshot\nExecStart=/usr/local/sbin/kimi-idle.sh\n' | sudo tee /etc/systemd/system/kimi-idle.service
printf '[Unit]\nDescription=Kimi idle check every 5 min\n[Timer]\nOnBootSec=10min\nOnUnitActiveSec=5min\n[Install]\nWantedBy=timers.target\n' | sudo tee /etc/systemd/system/kimi-idle.timer
sudo systemctl daemon-reload && sudo systemctl enable --now kimi-idle.timer
curl -s http://127.0.0.1:8000/metrics | grep -E '^vllm:(num_requests_running|num_requests_waiting|prompt_tokens_total)' | head

The last command verifies the metric names exist in your vLLM build [EXP]. If they differ, fix them in the script, or the watcher will treat a busy server as idle. Test: set IDLE_MIN=1 and BOOT_GRACE_MIN=0, leave the VM idle, and watch journalctl -t kimi-idle. Pass: "draining" then "deallocating", and 6.3 then reads PowerState/deallocated. Repeat while streaming a long answer; pass if no deallocation occurs until the answer completes and 20 minutes pass.

6.7 Restart and cache restoration — [VM]

After every deallocation, the NVMe RAID and the weights are gone. This unit rebuilds them before kimi-vllm starts.

sudo cp ~/kimi/manifest.tsv /etc/kimi-manifest.tsv
sudo tee /usr/local/sbin/kimi-restore.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
. /etc/kimi-pins.env
log(){ logger -t kimi-restore "$*"; echo "$(date -u +%T) $*"; }
log start
if ! findmnt /mnt/kimi >/dev/null; then
  mdadm --assemble --scan || true
  if [ -e /dev/md/kimi ]; then mount -o noatime /dev/md/kimi /mnt/kimi
  else
    CAND=$(lsblk -dn -o NAME,TYPE,MODEL | awk '$2=="disk" && /NVMe Direct/ {print "/dev/"$1}')
    for d in $CAND; do [ "$(lsblk -n "$d" | wc -l)" -eq 1 ] && [ -z "$(wipefs -n "$d")" ] || { log "refusing: $d not blank"; exit 1; }; done
    mdadm --create /dev/md/kimi --level=0 --raid-devices=$(echo $CAND | wc -w) $CAND --force --run
    mkfs.xfs -f -L kimi-nvme /dev/md/kimi; mkdir -p /mnt/kimi; mount -o noatime /dev/md/kimi /mnt/kimi
  fi
fi
D=/mnt/kimi/models/Kimi-K2.5
if [ "$(cat $D/.revision 2>/dev/null)" != "$MODEL_REV" ]; then
  log "azcopy restore $MODEL_REV"; mkdir -p "$D"
  AZCOPY_AUTO_LOGIN_TYPE=MSI azcopy copy "https://${SA}.blob.core.windows.net/models/Kimi-K2.5/${MODEL_REV}/*" "$D/" --recursive --log-level ERROR
fi
bad=0; while IFS=$'\t' read -r p s _; do [ "$(stat -c %s "$D/$p" 2>/dev/null)" = "$s" ] || { log "size mismatch $p"; bad=1; }; done < /etc/kimi-manifest.tsv
[ $bad -eq 0 ] || exit 1
log done
EOF
sudo chmod 0750 /usr/local/sbin/kimi-restore.sh
printf '[Unit]\nDescription=Restore NVMe RAID and Kimi weights\nAfter=network-online.target\nWants=network-online.target\n[Service]\nType=oneshot\nRemainAfterExit=yes\nTimeoutStartSec=3h\nExecStart=/usr/local/sbin/kimi-restore.sh\n[Install]\nWantedBy=multi-user.target\n' | sudo tee /etc/systemd/system/kimi-restore.service
sudo systemctl daemon-reload && sudo systemctl enable kimi-restore kimi-vllm kimi-webui

Now re-enable the kimi-restore references in kimi-vllm.service, if you commented them out in 3.5.

Cold-start test (acceptance criterion 6): [PH] or [CS] start the VM and note the time. Then [VM] run journalctl -t kimi-restore -o short-iso and journalctl -u kimi-vllm -o short-iso | grep -m1 'startup complete'. Pass: /health returns 200 and Section 5.3 test 2 passes from the phone, with no manual steps.

6.8 Credential recovery

6.9 Complete cleanup — [CS] and Cloudflare dashboard

az group delete -n "$RG" --yes                        # VM, disks, IP, VNet, NSG, storage, Key Vault, Automation
az keyvault list-deleted --query "[?name=='$KV'].name" -o tsv && az keyvault purge -n "$KV"
az role definition delete --name "VM Self Deallocate (kimi)"
az role definition delete --name "VM Operator (kimi)"
az ad user delete --id kimi-operator@<your-tenant-domain>
az rest --method delete --url "https://management.azure.com/subscriptions/$SUB_ID/providers/Microsoft.Consumption/budgets/kimi-credit?api-version=2023-05-01"
az group exists -n "$RG"                               # expect: false
az resource list --query "[?contains(id,'kimi')].id" -o tsv  # expect: empty

In Cloudflare, delete the tunnel kimi, the api. and chat. DNS records, the two Access applications and the service token. Pass: Cost Management shows no new rg-kimi-inference charges 48 hours later.

Acceptance criteria

The deployment is successful only when every box below is ticked with saved evidence: command output, a screenshot or a log excerpt with timestamps. Nothing in this document counts as evidence; it is all plan or illustration.

Pins and configuration

• /etc/kimi-pins.env holds a 40-character MODEL_REV, a sha256: vLLM digest and an Open WebUI digest, each with a recording date (3.1, 5.1)

• All 64 shards pass the size and SHA-256 checks against the hub manifest (3.2)

• az role assignment list --assignee <VM identity> shows only container Reader, Key Vault Secrets User and VM Self Deallocate, scoped as designed (1.9, 3.3)

1. Working mobile inference

• From the phone on mobile data (Wi-Fi off), tests 1–4 of 5.3 pass in the chosen client

• Test 5a returns finish_reason: tool_calls with valid JSON arguments; 5b produces an answer using the returned value

• Test 6 (image) passes; a PDF question answered through Open WebUI cites the right page

• Streaming visibly renders token by token on the phone

2. Rejected unauthorised access

• No credentials: 401 (A) or an Access redirect/403 (B), and the request is absent from journalctl -u kimi-vllm in case B

• Wrong key: 401. Wrong Access secret: non-2xx

• /health, /metrics, /invocations, /tokenize, /pause, /v1/completions all return 404 externally even with valid credentials

• Rate limit returns 429 under burst (A)

• Open WebUI sign-up is disabled: a fresh browser cannot create an account

• kimi-operator can start and deallocate the VM and nothing else

3. Closed public inference port

• ss -ltnp on the VM shows 8000 bound to 127.0.0.1 only

• An external TCP connect to <public IP>:8000 fails; in B, 80 and 443 fail too

• az network nsg rule list shows no rule mentioning 8000 and SSH set to Deny

4. Measured memory and performance (3.8 table filled from your run)

• Peak VRAM per GPU recorded and ≤ 90% of memory.total

• vLLM's reported KV capacity gives at least 4 concurrent 32,768-token sequences

• Median TTFT (4K prompt, c=1), median TPOT and output tok/s at c=1 and c=2 recorded, with 0 failed requests

• Warm restart time and cold start time recorded

5. Successful restart

• After a full deallocate and start, kimi-restore rebuilds the NVMe RAID, restores the weights, and kimi-vllm reaches /health 200 with no manual steps

• The same phone client works afterwards without reconfiguration, and Open WebUI chat history is still present

• A deliberate docker kill kimi-vllm is followed by automatic restart (Restart=on-failure)

6. Verified deallocation

• Manual deallocate from the phone ends in PowerState/deallocated (6.3)

• The on-VM idle watcher deallocates after the idle window, and does not during a long streaming answer (6.6)

• The runbook cap deallocates when its test threshold is exceeded (6.5b), and daily auto-shutdown is configured (6.5a)

• A budget exists with actual and forecast alerts, or the reason it cannot exist is recorded (6.4)

• Section 6.2 recomputed with your measured cold-start time and your real credit currency and expiry

Sources

All sources were opened and verified on 2026-10-20. Prices came from the live Retail Prices API on the same day.

Moonshot / Hugging Face

• moonshotai organisation: model list (Kimi-K3 newest)

• Kimi-K3 model card

• Kimi-K2.6 model card

• Kimi-K2.5 model card, config.json, LICENSE, API metadata (sha, gated), file tree

• Repository sizes: K2.5, K2.6, K3, K2.5-NVFP4

NVIDIA

• nvidia/Kimi-K2.5-NVFP4 model card

• Dynamo Kimi-K3 recipe

• CUDA Toolkit release notes (driver table)

• CUDA compatibility

• Container Toolkit install guide

vLLM

• Kimi-K2.5 recipe

• Kimi-K2.6 recipe

• Moonshot recipe index

• Security: what --api-key protects

• GitHub releases

• Docker Hub tags and digests

Microsoft Azure

• VM sizes: ND H200 v5, ND H100 v5, NDm A100 v4, ND MI300X v5

• HPC/ND VM images and drivers, azhpc-images releases

• N-series Linux driver setup, NVIDIA GPU Driver Extension

• Default outbound access

• Retail Prices API

• Foundry catalog: Kimi-K2.5

Cloudflare

• Tunnel configuration file

• Origin parameters (access)

• Service tokens

• Error 524 / Proxy Read Timeout

• Package repository

Clients

• Open WebUI features, quick start, releases

• Chatbox setup (third-party guide)

• Tailscale downloads

Conflicting or outdated documentation noted in this review

• The vLLM recipe index lists K2.6 "INT4 714G", while the K2.6 recipe says "equivalent aggregate VRAM (~640 GB)". This review treats 640 GB GPUs as unsupported.

• The K2.5 recipe was verified on vLLM v0.17.0. This plan pins v0.31.0 and keeps v0.17.0 as the documented fallback.

• The NVIDIA NVFP4 card says to use vllm/vllm-openai:latest. This review does not use unpinned tags.

• The NDm A100 v4 page's accelerator table shows "80" GB for 8 GPUs, while its host table says 8× 80 GB. The per-GPU reading is used.

• The H200 Spot price rows equal the on-demand prices. They were treated as unreliable and excluded.

• Cloudflare's 524 timeout is 125 s on the current page, not the 100 s often quoted elsewhere.