Nodes
Queues
Recent requests
Nodes
Add or update a node
API keys
Add or rotate a key
Secrets are written to the .env file; config.yaml only ever
stores a ${VAR} reference. Adding or rotating a key takes effect
immediately — no restart needed.
Model placement policy
Pin or restrict a model
This is the real placement control: pin_nodes restricts a model to those
nodes, exclude_nodes keeps it off them. Distinct from the
pinned badge on a node card, which only means Ollama is holding the model
resident because of the keep-alive below.
Context window is what the router asks the node for when a request does
not specify its own options.num_ctx. Ollama otherwise applies a conservative
built-in default, and for an embedding model longer input is rejected rather than
truncated — the input length exceeds the context length error. Set the model's
real capability here (nomic-embed-text supports 8192) and every client gets it
without changing any application.
Raise it together with KV accounting. A bigger context multiplies the
KV-cache VRAM every concurrent request needs. If the model has no
kv_cache_bytes_per_token (and no cluster-wide
scheduler.kv_cache_bytes_per_token), admission costs each request at zero and
only max_concurrency applies — which counts requests, not bytes. On an 8 GB
card that combination does not queue under load, it runs out of memory. Measure the real
cost against /api/ps size_vram and set it in
config.yaml, or lower concurrency. The router logs
router.context_without_kv_accounting at startup naming any model in this state.
Runtime settings
-1m (any negative duration) keeps every warmed model resident forever —
which is why models show as pinned. A real duration like 10m
lets idle models unload on their own and frees VRAM without intervention.
Config file
Every save validates the whole config first and leaves a timestamped
.bak beside it, so a bad edit is rejected rather than applied.
Add a model
Pulls to every node with enough VRAM. Downloads run in the background and resume if interrupted.