Local models
Run AI models locally, securely, private.
Pull a model from Hugging Face and serve it on your own machine. Your prompts and your code never leave the computer, there is nothing to pay per token, and it answers on both the OpenAI and the Claude API — so whatever you already wrote works by changing one address.
then kosa8 model pull — the whole session is below
Verified
Eight things it does, and the output that proves each one.
This whole page was written after running the commands on it: a 468MiB model pulled, served, answered on both dialects, pushed to a stock registry and pulled back byte-identical.
Run models from the terminal
One command pulls, one serves. No Python environment, no notebook, no account.
kosa8 model pull … && kosa8 model serve …
Straight from Hugging Face
Give it a repo and a quantisation; it resolves the GGUF and fetches it.
pulled … (468MiB) from hf.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF
Speaks OpenAI and Claude
Two dialects on one port. Point an existing app at it by changing the base URL.
POST /v1/chat/completions · POST /v1/messages
Uses the GPU
llamay serves on the GPU where the machine has one; without it, kosa8's built-in engine serves on the CPU.
running on llamay · 175 tok/s
Models as OCI artifacts
A model becomes an image: content-addressed, versioned, signable like any other.
artifactType application/vnd.kosa8.model.v1+json
Any registry you already run
Standard manifests, so your existing registry stores models with no plugin.
pushed to 127.0.0.1:5000 · pulled back sha256 identical
Nothing billed, nothing sent
No key, no meter, no request leaving the laptop. Works on a plane.
$0.00 · 0 bytes egress
A model list, like an image list
The things you have pulled are inventory, with where each came from.
kosa8 model ls
How it works
One session, start to finish.
The same run the landing page shows, in full: find it, serve it, talk to it, ship it, get it back.
# 1 · install it curl -fsSL https://kosa8.com/install.sh | sh checksum verified · installed kosa8 0.17.0 kosa8 doctor 13 checks · all clear # 2 · find a model on Hugging Face kosa8 model pull Qwen/Qwen2.5-0.5B-Instruct-GGUF:q4_k_m pulling Qwen/Qwen2.5-0.5B-Instruct-GGUF:q4_k_m: 38% (178MiB/468MiB) pulling Qwen/Qwen2.5-0.5B-Instruct-GGUF:q4_k_m: 91% (427MiB/468MiB) pulled qwen2.5-0.5b-instruct-q4_k_m kosa8 model ls NAME SIZE SOURCE PULLED qwen2.5-0.5b-instruct-q4_k_m 468MiB hf.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF 2026-07-29 # 3 · serve it on your own GPU kosa8 model serve qwen2.5-0.5b-instruct-q4_k_m kosa8 model gateway on http://127.0.0.1:7434 OpenAI: POST /v1/chat/completions Anthropic: POST /v1/messages running on llamay # 4 · talk to it, in the dialect you already use curl -s :7434/v1/chat/completions -d '{…}' | jq -c .choices[0].message {"role":"assistant","content":"Kosa8 works."} 40 prompt + 6 completion · 175 tok/s · $0.00 # the same server, the same port, speaking Claude curl -s :7434/v1/messages -d '{…}' | jq -c . {"type":"message","role":"assistant","content":[{"type":"text","text":"Hello!"}], "stop_reason":"end_turn","usage":{"input_tokens":35,"output_tokens":3}} # 5 · ship it to a registry you already run kosa8 run -p 5000:5000 registry:2 kosa8: published 5000/tcp -> 0.0.0.0:5000 kosa8 model push qwen2.5-0.5b-instruct-q4_k_m 127.0.0.1:5000/models/qwen2.5:0.5b-q4 checksum 100% (468MiB/468MiB) · manifest 100% (751B/751B) pushed to 127.0.0.1:5000/models/qwen2.5:0.5b-q4 weights sha256:74a4da8c9fdbcd15bd1f6d01d621410d31c6fc0098… (468MiB) artifactType application/vnd.kosa8.model.v1+json ↑ a stock registry:2 stored it. No plugin, nothing kosa8-specific. # 6 · get it back, byte for byte kosa8 model pull-oci 127.0.0.1:5000/models/qwen2.5:0.5b-q4 --name from-registry weights 100% (468MiB/468MiB) serve it with `kosa8 model serve from-registry` shasum -a 256 ~/.kosa8/models/*/*.gguf 74a4da8c9fdbcd15… qwen2.5-0.5b-instruct-q4_k_m.gguf 74a4da8c9fdbcd15… from-registry.gguf ↑ same bytes · nothing left the machine · $0.00
Honest about it
What this is not.
The fast path is llamay
kosa8 model serve uses llamay for
GPU serving, which kosa8 does not install for you. Without it the built-in engine serves on
the CPU, and says so. kosa8 doctor checks for it. On macOS and Linux:
curl -fsSL https://llamay.com/install.sh | sh.
The runner is not sandboxed
It is a host process, not a sealed machine. The isolation story on the rest of this site is about sandboxes; the model gateway is bound to 127.0.0.1 and that is the extent of it.
A small model is a small model
Local inference is bounded by the machine in front of you. A 0.5B model at 175 tok/s is genuinely useful for classification, extraction and tool-calling — and it is not going to out-argue a frontier model.
Your laptop already has a GPU.
One line to install, one command to pull a model, one to serve it. No key, no meter.