Local models

Run AI models locally, securely, private.

Pull a model from Hugging Face and serve it on your own machine. Your prompts and your code never leave the computer, there is nothing to pay per token, and it answers on both the OpenAI and the Claude API — so whatever you already wrote works by changing one address.

Download for Mac Read the guide

Nothing leaves your machineNo token costGPU acceleratedOffline

then kosa8 model pull — the whole session is below

Verified

Eight things it does, and the output that proves each one.

This whole page was written after running the commands on it: a 468MiB model pulled, served, answered on both dialects, pushed to a stock registry and pulled back byte-identical.

Run models from the terminal

One command pulls, one serves. No Python environment, no notebook, no account.

kosa8 model pull … && kosa8 model serve …

Straight from Hugging Face

Give it a repo and a quantisation; it resolves the GGUF and fetches it.

pulled … (468MiB) from hf.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF

Speaks OpenAI and Claude

Two dialects on one port. Point an existing app at it by changing the base URL.

POST /v1/chat/completions · POST /v1/messages

Uses the GPU

llama.cpp offloads to Metal and picks the layer count for the machine it lands on.

MTL0: Apple M4 (12124 MiB free) · 175 tok/s

Models as OCI artifacts

A model becomes an image: content-addressed, versioned, signable like any other.

artifactType application/vnd.kosa8.model.v1+json

Any registry you already run

Standard manifests, so your existing registry stores models with no plugin.

pushed to 127.0.0.1:5000 · pulled back sha256 identical

Nothing billed, nothing sent

No key, no meter, no request leaving the laptop. Works on a plane.

$0.00 · 0 bytes egress

A model list, like an image list

The things you have pulled are inventory, with where each came from.

kosa8 model ls

How it works

One session, start to finish.

The same run the landing page shows, in full: find it, serve it, talk to it, ship it, get it back.

the whole session · install → answer → ship → back

# 1 · install it

checksum verified · installed kosa8 0.1.5

  13 checks · all clear
 
# 2 · find a model on Hugging Face

pulling Qwen/Qwen2.5-0.5B-Instruct-GGUF:q4_k_m: 38% (178MiB/468MiB)
pulling Qwen/Qwen2.5-0.5B-Instruct-GGUF:q4_k_m: 91% (427MiB/468MiB)
pulled qwen2.5-0.5b-instruct-q4_k_m

NAME                           SIZE     SOURCE                                  PULLED
qwen2.5-0.5b-instruct-q4_k_m   468MiB   hf.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF   2026-07-29
 
# 3 · serve it on your own GPU

starting llama-server for qwen2.5-0.5b-instruct-q4_k_m…
kosa8 model gateway on http://127.0.0.1:7434
  OpenAI:    POST /v1/chat/completions
  Anthropic: POST /v1/messages
  MTL0: Apple M4 (12124 MiB free) · layers: auto
 
# 4 · talk to it, in the dialect you already use

{"role":"assistant","content":"Kosa8 works."}
  40 prompt + 6 completion · 175 tok/s · $0.00
 
# the same server, the same port, speaking Claude

{"type":"message","role":"assistant","content":[{"type":"text","text":"Hello!"}],
 "stop_reason":"end_turn","usage":{"input_tokens":35,"output_tokens":3}}
 
# 5 · ship it to a registry you already run

kosa8: published 5000/tcp -> 0.0.0.0:5000

checksum 100% (468MiB/468MiB) · manifest 100% (751B/751B)
pushed to 127.0.0.1:5000/models/qwen2.5:0.5b-q4
  weights sha256:74a4da8c9fdbcd15bd1f6d01d621410d31c6fc0098… (468MiB)
artifactType  application/vnd.kosa8.model.v1+json
  ↑ a stock registry:2 stored it. No plugin, nothing kosa8-specific.
 
# 6 · get it back, byte for byte

weights 100% (468MiB/468MiB)
  serve it with `kosa8 model serve from-registry`

74a4da8c9fdbcd15…  qwen2.5-0.5b-instruct-q4_k_m.gguf
74a4da8c9fdbcd15…  from-registry.gguf
  ↑ same bytes · nothing left the machine · $0.00

Honest about it

What this is not.

It needs llama.cpp

kosa8 model serve runs llama-server, which kosa8 does not install for you. kosa8 doctor checks for it before you spend a download on finding out. On a Mac: brew install llama.cpp.

The runner is not sandboxed

It is a host process, not a sealed machine. The isolation story on the rest of this site is about sandboxes; the model gateway is bound to 127.0.0.1 and that is the extent of it.

A small model is a small model

Local inference is bounded by the machine in front of you. A 0.5B model at 175 tok/s is genuinely useful for classification, extraction and tool-calling — and it is not going to out-argue a frontier model.