How do I point a coding agent at a local model?

01Steps

1. Install kosa8 and check which engine will serve. Install with the one-line installer, then run kosa8 doctor. It reports whether llamay, the faster engine that uses the GPU, is installed. Without it, kosa8's built-in engine serves on the CPU, which works but is slow for a model the size a coding agent needs.

   curl -fsSL https://kosa8.com/install.sh | shCOPY kosa8 doctorCOPY curl -fsSL https://llamay.com/install.sh | shCOPY

2. Pull a coding model from Hugging Face. Models come from Hugging Face in GGUF form: name the repository, and the quantisation after a colon. --name gives it a short local name. The 7B Qwen2.5-Coder file at q4_k_m is about 4.4 GiB; the 1.5B and 3B repositories are smaller if your machine has less memory.

   kosa8 model pull Qwen/Qwen2.5-Coder-7B-Instruct-GGUF:q4_k_m --name coderCOPY kosa8 model lsCOPY

3. Serve it on the local gateway. Leave this running in its own terminal. It serves one model, on one fixed port, until you press Ctrl-C. If your organisation's signed policy keeps an approved model list, kosa8 model check coder tells you first whether it may be served.

   terminal

   kosa8 model serve coder
   kosa8 model gateway on http://127.0.0.1:7434
     OpenAI:    POST /v1/chat/completions
     Anthropic: POST /v1/messages

4. Check that it answers. The gateway serves exactly one model, so it answers whatever name the client sends in model.

   curl -s http://127.0.0.1:7434/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"coder","messages":[{"role":"user","content":"Reply with the single word: ready"}],"max_tokens":8}'COPY

5. Point the agent at it. Set the base URL the agent's own docs name. For Aider, that is OPENAI_API_BASE plus a model name prefixed with openai/. The gateway has no login, so any placeholder key will do. kosa8 has not separately documented or tested Aider; these are Aider's documented settings for an OpenAI-compatible endpoint.

   export OPENAI_API_BASE=http://127.0.0.1:7434/v1COPY export OPENAI_API_KEY=localCOPY aider --model openai/coderCOPY

02Which coding agents accept a custom base URL?

Most do, but accepting a base URL is not the same as working with any server behind it. The question is whether the agent needs native tool calling, where it sends a list of tools and expects structured tool calls back. Since kosa8 v0.11.0 the gateway does both, on its OpenAI and Anthropic endpoints: it offers the tools to the model and returns the calls the model makes. How well that works depends on the model; a model trained for tool calling, such as Qwen2.5-Coder, calls far more reliably than a small general one.

An agent like Aider asks the model to reply with whole files or search-and-replace blocks in plain text, and applies them itself, so an OpenAI-compatible chat endpoint is enough. Agents like Claude Code, Codex CLI and OpenCode run their loop through tool calls: read this file, run this command, edit these lines. That needs the gateway's tool calling, which arrived in v0.11.0, and a model that is good at it.

03Does Claude Code work with a local model?

Not through kosa8's gateway. Claude Code reads ANTHROPIC_BASE_URL to reach an LLM gateway, but Anthropic's docs say it does not support routing Claude Code to non-Claude models through any gateway, and Claude Code acts through tool use, which the gateway does not provide.

The gateway's /v1/messages endpoint is for apps that use an Anthropic library to generate text: set ANTHROPIC_BASE_URL=http://127.0.0.1:7434 and they work unchanged. To keep Claude Code's own commands contained, see the guide on running Claude Code safely on a Mac.

04Can Codex CLI use Ollama or another local model?

Yes, through its own local path: codex --oss runs against a local provider such as Ollama or LM Studio, and config.toml can define a custom provider with a base_url. OpenAI's config examples use wire_api = "responses", the Responses API, which kosa8's gateway does not serve.

For an OpenAI library in your own code, OPENAI_BASE_URL=http://127.0.0.1:7434/v1 is the one change. The gateway also answers /v1/models, and /v1/completions like /v1/chat/completions.

05Does a local model keep my code private?

The prompts do: the gateway listens on 127.0.0.1, answers only requests addressed to 127.0.0.1 or localhost, and sends nothing off the machine, at $0 per token. It has no login of its own, so any program running as you can use it while it is served, and the model runner is a host process, not a sealed sandbox.

The agent is a separate matter. Whatever model it talks to, the commands it runs still run on your machine unless you put them somewhere else, such as a kosa8 sandbox created with --egress deny for the code it writes. Reaching the gateway from inside a sandbox is not documented. And a small model is a small model: private and free to run, not a drop-in replacement for the largest hosted ones on hard tasks.

06Other options

Several local servers expose an OpenAI-compatible API, and some agents have their own local mode. Which fits depends on whether your agent needs tool calling.

• Ollama serves local models; Codex CLI's --oss mode targets it or LM Studio, and Codex's docs show a provider entry for Ollama at http://localhost:11434/v1.
• LM Studio and llama.cpp's server are configured as OpenAI-compatible local servers in OpenCode's provider docs, which also give tool-calling tips for Ollama.
• A hosted model behind a key broker keeps the agent's full tool use while the API key stays outside the sandbox: kosa8's built-in kits for Claude Code, Codex, Gemini CLI and OpenCode do this.
• A full VM in UTM or Lima can hold both the model server and the agent when you want one box with everything inside it.

07Sources

• kosa8 docs: Local models
• kosa8: Local models
• kosa8 docs: Credentials (brokered kits)
• Aider docs: OpenAI compatible APIs
• Aider docs: Edit formats
• Claude Code docs: Other LLM gateways
• Claude Code docs: Gateway compatibility guide
• OpenAI Codex docs: Advanced configuration
• OpenCode docs: Providers
• Hugging Face: Qwen2.5-Coder-7B-Instruct-GGUF

08Related guides

• How do I run a coding agent fully air-gapped?
• How do I build a self-hosted code interpreter for my LLM?
• Is Aider safe? Running Aider in a sandbox
• How do I run Codex CLI safely?
• How do I run Claude Code safely on a Mac?
• How do I run an AI agent sandbox on my own machine instead of the cloud?
• How do I give an AI agent API keys without exposing them?