Use a local model.

Puggie can talk to a model server on your own computer, a GPU box or a spare machine on your network. No Puggie account, no token bill and no source checkout: start an OpenAI-compatible server, add its address in the app, and give it to a bot.

What Puggie supports

Any server that speaks the OpenAI chat API works. These four get a little extra help: Puggie asks them for their model list and reads the context and vision support they report.

The base URL must be the API root that has /models and /chat/completions under it; for most servers that means it ends in /v1. Puggie does not download, load or manage models. It only sends requests to the server while a bot works.

Before you start

A bot is not a single chat turn. It reads a long prompt with tool definitions, calls tools, reads the results and goes on for several turns. A model that is pleasant in a chat window can still be a poor fit for a bot: it needs to follow instructions, call tools reliably and keep enough context to remember the job.

• Pick a model built for tools and instructions. Models such as Qwen3, Llama 3.3 or 4, Mistral Small, Devstral and GPT-OSS are good starting points. Tiny 1–3B models are fast, but usually lose track of tools and long jobs.
• Give it context. 32,768 tokens is a practical minimum; 65,536 or more is better for long chats and code. Every token of context costs memory. The prompt and tool list grow with the conversation.
• Give it room. A 7–8B model at 4-bit quantization needs roughly 8 GB of memory, a 14B model roughly 12–16 GB, and a 32B model roughly 24 GB, plus context. CPU-only works, but expect conversation speed rather than chat speed. llama.cpp and Ollama use Apple Silicon's GPU; vLLM is happiest on an NVIDIA or ROCm GPU.
• Keep one model loaded. Bots send one request at a time. A single loaded model with one parallel slot is faster and uses less memory than loading several.

Run the server

The commands below listen on 127.0.0.1 only. That is what you want when the bot runs on the same computer as the model. To reach it from another machine, put HTTPS in front of it (below) rather than opening the raw port.

llama.cpp

Download or build llama-server and point it at a GGUF file. The alias is the model ID Puggie will show.

llama-server \
  -m /models/Qwen3-14B-Q4_K_M.gguf \
  --alias qwen3-14b \
  --host 127.0.0.1 --port 8080 \
  --ctx-size 32768 \
  --n-predict 8192 \
  --jinja \
  --parallel 1

--jinja uses the model's own chat template, which is what makes tool calls work. --ctx-size is the context Puggie reads back from /props; do not leave it at the server's small default if you want agent work. If the model has a vision projector, llama.cpp reports it too.

curl -s http://127.0.0.1:8080/v1/models

Ollama

Start the server with the context length you want, then pull a model. Setting the context at the server keeps it from changing when the model reloads.

OLLAMA_HOST=127.0.0.1:11434 \
OLLAMA_CONTEXT_LENGTH=32768 \
OLLAMA_NUM_PARALLEL=1 \
OLLAMA_MAX_LOADED_MODELS=1 \
ollama serve

In another terminal:

ollama pull qwen3:14b
ollama run qwen3:14b "Say hello"

The model ID is exactly what ollama list prints, including the tag, for example qwen3:14b. If your Ollama version ignores OLLAMA_CONTEXT_LENGTH, make a model with the setting in its Modelfile:

ollama show --modelfile qwen3:14b > Modelfile
printf '\nPARAMETER num_ctx 32768\n' >> Modelfile
ollama create qwen3-14b-32k -f Modelfile

curl -s http://127.0.0.1:11434/v1/models

vLLM

vLLM serves an OpenAI-compatible endpoint and reports its context as max_model_len. Tool calling needs both a model that supports it and the right parser.

vllm serve Qwen/Qwen3-14B \
  --served-model-name qwen3-14b \
  --host 127.0.0.1 --port 8000 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes

hermes is the common parser for Qwen models; check vLLM's tool-calling table for the parser your model needs. --served-model-name sets the ID Puggie sees. Lower --gpu-memory-utilization if the GPU is shared with something else.

curl -s http://127.0.0.1:8000/v1/models

LM Studio

1. Install LM Studio and download a model that supports tool use.
2. Load it and set the context length to 32,768 (or as much as your memory allows).
3. Open the Developer tab, start the local server on port 1234, and turn on tool use if your version offers it.
4. Use http://127.0.0.1:1234/v1 and the model key from its model list.

LM Studio reports the loaded context and marks vision models, so the app usually fills in the right values. If you turn on "serve on local network", remember that the address is still plain HTTP and Puggie will refuse it from another machine: use the HTTPS proxy below.

Another OpenAI-compatible server

It needs at least GET /v1/models (optional if you will type the model IDs) and POST /v1/chat/completions with streaming and tool calls. A server that only offers /v1/completions, or that ignores the tools field, can chat but cannot run a bot's tools.

curl -s https://models.example.com/v1/models

Networking: where the server and the bot are

Puggie accepts https:// anywhere. It accepts plain http:// only for 127.0.0.1, localhost and [::1]. That is a fixed rule, not a warning you can dismiss. It is also the part people get wrong, because "localhost" means the computer running the bot, not the computer you are holding.

To give a server on another machine an HTTPS name, keep it bound to 127.0.0.1 and put a reverse proxy in front of it. Caddy is one small option; the block below gets a trusted certificate for a name that resolves to the machine:

models.example.com {
    reverse_proxy 127.0.0.1:8080
}

Then the base URL is https://models.example.com/v1. Tailscale Serve does the same job with a *.ts.net HTTPS name if the machines are on your tailnet. A self-signed certificate is not trusted when the bot connects, so it fails with a connection error; use a certificate from a public CA or Tailscale. A compose service name such as http://model:8080/v1 is also rejected: it is not loopback, even though it is on the same Docker network.

Add it in Puggie

Models belong to the machine that runs the bot. Add the endpoint on that machine, not just on the phone.

1. Open Setup → Models. If you have more than one computer, a row of chips at the top chooses the machine. "This phone" is one of them.
2. Scroll to the bottom of the provider list and tap Custom OpenAI-compatible endpoint.
3. Give it a Name you will recognise, such as Home model. The name becomes the provider's ID; using the same name again updates that endpoint, and a different name adds another one.
4. Paste the Base URL, including /v1, for example http://127.0.0.1:8080/v1 or https://models.example.com/v1.
5. Leave API key empty unless your server asks for one. If it does, paste it here; it is stored in the encrypted store on that machine, never in the endpoint's config file.
6. For Model IDs, leave the field empty and tap Find to use every model the server lists, or type one or more IDs separated by commas. If the server has no model list, type the IDs and save without Find.
7. Turn on The models think (reasoning) only if the model reports a separate reasoning field. Turn on The models read images for a vision model; the switch applies to every model at this endpoint, so keep text and vision models at separate endpoints if you mix them.
8. Tap Save and choose model. The model list opens; pick one to try it in a chat.

To point an existing bot at it, open the bot and use Model → Change model, or tap the model chip above the message box. A new bot asks for a model under Model; if the list is empty, Add a model account or server opens the same form.

The endpoint's sheet has Remove; it deletes the endpoint and its key. Sessions using it stop working until you add it again. A model added on the phone is not visible to a bot on a computer, and the other way round; add it once on each machine that should use it.

Settings that matter for bots

• Context. Puggie reads the context from llama.cpp's /props, vLLM's max_model_len, LM Studio's loaded context, or Ollama's num_ctx / loaded context. If the server reports none, Puggie assumes 131,072 tokens, which is more than most local models can take. Set the context in the server, not in a hidden app setting, so Puggie knows the truth.
• Output. Puggie asks for at most 8,192 output tokens, or a quarter of the context if that is smaller.
• Tools. The model must support OpenAI tool calling and the server must pass the tools through. A model that only chats will answer in words but never run a command, read a file or use a connector.
• Thinking. The switch marks a model that returns reasoning; it does not make a model smarter. Leave it off when you are unsure, and turn it on if the model's answers arrive with a separate reasoning part.
• What it costs. A local model has no token price. Puggie counts a bot's use as time on a local model rather than money, so its daily limit is minutes of work.
• Quick calls. On the phone, Settings → Decision model can use a llama.cpp model for small yes/no questions, or a Jev classifier from OpenRouter. Ollama, vLLM and LM Studio endpoints are not offered there.

On-phone models

Puggie does not run a bot's model on the phone. A phone-sized model that is useful for real work is slow, hot and battery-hungry, and Android can stop a background server when the app leaves the screen. Bots that need to keep working belong on a computer.

What does run on the phone are small helpers, mostly on Pixels with Gemini Nano: dictation, Tidy up, reading a photo, Scan, catch-up, tappable dates, Translate, memory by meaning and making a pet from a photo. They help you use the app; they do not answer a bot's messages.

You can add an OpenAI-compatible server running on the phone itself at a loopback address such as http://127.0.0.1:8080/v1, because the plain-HTTP rule allows loopback. That is an experiment, not a supported path: only bots on the phone can reach it, and the server has to stay running while they work. For anything unattended, put the model on a computer and give it an HTTPS name.

If something goes wrong

• "Use https:// (plain http only for this device)." The base URL is plain HTTP to something other than loopback. Use https://, or move the bot to the computer that runs the server and use 127.0.0.1.
• "The server did not list any models." /v1/models is missing, the URL is wrong, the server is not running, or the key is missing. Test it from the machine that runs the bot with curl -sS https://models.example.com/v1/models. If the server simply has no model list, type the model IDs and save without Find.
• Find takes a moment and then fails. It waits five seconds for a reply. If the server is still starting, wait and try again, or type the IDs and save without Find.
• "Could not load models" after saving. The endpoint saved but the chat request failed. Check the key, the model ID and the server log. A model ID with the wrong tag is the usual cause with Ollama.
• The answer is words but the bot never uses a tool. The model or server does not support tool calls. For llama.cpp use --jinja; for vLLM set --enable-auto-tool-choice and the right parser; in LM Studio turn on tool use; with Ollama pick a model that supports tools.
• The context is cut off or the server returns a context-length error. Raise the context on the server, lower the number of tools or start a new chat. Make sure the server reports the context Puggie should use; the 131,072 default is not a promise.
• It is very slow, swaps or runs out of memory. Use a smaller quantization, a smaller model, a shorter context, or offload fewer layers. Close other models; one loaded model and one parallel slot is the right shape for a bot.
• A certificate or connection error. Puggie does not trust a self-signed certificate. Use a public CA or Tailscale HTTPS, check that the name resolves from the machine running the bot, and test with curl there, not from your laptop.
• It works on the phone but not on a computer, or the other way round. The endpoint was added on the wrong machine. Models are per machine; add it again with that machine chosen in Setup → Models.
• The container cannot reach 127.0.0.1. That address is the container. Use an HTTPS name, or change the machine to run "On the computer" on its page if you want loopback to be the host.
• "No API key saved" for an endpoint that used to work. The endpoint had a key when it was saved, but the encrypted store no longer has it. Save the endpoint again and paste the key.

The published images carry curl, so you can test from the same place the bot runs. For a machine added by worker.sh, use docker exec puggie-worker curl -sS https://models.example.com/v1/models (or podman exec if it chose Podman); for the hub, docker compose exec hub curl -sS https://models.example.com/v1/models. If that works but the app does not, the endpoint was added on another machine.

For the server-specific details, see the llama.cpp server README, Ollama's OpenAI compatibility notes, vLLM's OpenAI server docs and LM Studio's endpoint docs.

New to self-hosting? Start with the self-hosting guide, then come back here for the model. Join the beta to get the app.