Running it

Brain Cores

A Core is a whole inference runtime — engine, quantisation, KV format, serving strategy — presented as one OpenAI-compatible endpoint. Cerveau picks a URL and needs to know nothing else, which is exactly what lets llama.cpp and vLLM coexist without the loop knowing either exists.

Two Cores, two jobs

Neither can do the other's work, which is why there are two rather than a winner.

	llama.cpp	vLLM
Runs	Large mixture-of-experts models	Dense models that fit the card
Strategy	Offloads expert weights to system RAM	Keeps weights resident, batches continuously
Wins when	The model does not fit in VRAM	The model fits, and there is real concurrency
Shares the GPU	Yes	No — it takes the whole card

Offloading experts is nearly free on an MoE model because only a fraction of the parameters fire per token — roughly 3B of 35B on the reference model. The rest can sit in system memory without the whole thing falling apart. That is what runs a 35B model on a 24 GB card.

The registry

Cores are declared, not hardcoded. The registry holds each one's endpoint, systemd unit, health path, chat template, capabilities and startup timeout, so adding a Core is a config edit rather than a code change.

The line a profile system could not hold A profile is a set of flags for one engine. The moment vLLM entered, the launch script would have needed Python venvs, a different process model and its own health semantics. Cores sidestep that: each owns its launch logic, and the harness only ever picks a URL.

Start commands are shown, never executed A harness that can start engines can also stop the one it is talking to, and the failure mode is a machine with no model at all. The Start field is there so the panel can tell you how to bring the other Core up. You run it.

Switching

Deliberately coarse and manual: a Core switch is a task-boundary act, never automatic per-turn. The panel lists Cores from the registry, a confirmation names the real cost — roughly 90 seconds with no model — and the panel shows stopping → loading → ready.

Three requirements were learned the hard way:

A real "switching" state, or the health guards fire spuriously and the panel screams core unreachable during a deliberate action.
Refuse to switch mid-turn, or the model is swapped out from under a running autopilot session.
The engine unit is a user systemd unit, so the core restarts it with systemctl --user — no sudo, no root daemon.

Why a swap survives at all

Session context lives in Typesense and the embedder, not in the model's KV cache. Destroying the Core does not destroy the session; the new Core recalls what it needs.

The honest limit What carries across is a briefing, not a memory. The previous Core's exact reasoning chain — the tool output it saw, the half-formed hypothesis at turn seven — dies with its KV cache. Perfect for "start a fresh subtask with a stronger model"; lossy for "continue this exact debugging thread".

llama.cpp

llama-server -m Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \
  --host 127.0.0.1 --port 8080 --jinja

--jinja is not optional: without it the server does not apply the model's chat template, and tool calls come back as prose that nothing can parse.

Expert-split profiles

--n-cpu-moe N decides how many expert layers live in system RAM. Measured on one RTX 3090, 16-core CPU, 128 GB DDR5, Qwen3.6-35B-A3B at Q4_K_M:

Profile	--n-cpu-moe	Generation	VRAM left	Trade
shared	34	73 tok/s	~12 GB	Half the card stays free for a speech model, a vision model, whatever else you run.
fast	16	98 tok/s	~4 GB	About a third quicker. Room for one small companion model, not several.
max	10	107 tok/s	~1.4 GB	Fastest, and it claims the entire GPU to get there.

Your numbers will differ with quant, RAM speed and core count. The point of the table is the shape of the trade, not the digits: what you spend to go faster is VRAM you could have spent on something else, and the context window is unaffected either way.

vLLM

The second Core, for dense models. The build in use is patched vLLM 0.27.1, not a stock install — pip install vllm does not reproduce it.

vllm serve models/Qwen3.8-27B-W4A16-AutoRound-fast \
  --served-model-name qwen3.8-27b --host 0.0.0.0 --port 18020 \
  --gpu-memory-utilization 0.88 --max-model-len 98304 \
  --max-num-seqs 8 --api-server-count 1 --language-model-only \
  --kv-cache-dtype fp8 --mamba-ssm-cache-dtype float16 \
  --async-scheduling --max-num-batched-tokens 2048 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml

What the flags are doing

--kv-cache-dtype fp8

An fp8 KV cache is what took the window to 96K. Quality was measured on fp8, not bf16.

--max-model-len 98304

96K. Higher starts preempting: the KV pool divided by the window has to leave more than one sequence.

--language-model-only

The weights carry a vision tower. Loading it costs VRAM, and VRAM is the KV pool — vision is paid for in context, directly.

--tool-call-parser qwen3_xml

vLLM validates this. The wrong parser fails silently, as prose.

no speculative decoding

MTP is exact-by-construction speed with no quality gain. Off returns its working memory to the KV cache.

gpu-memory-utilization is a moving target vLLM claims its share of free memory at startup, and your desktop's share varies — a browser with GPU compositing is enough to move it. The same command at 0.90 started one day and was refused the next. 0.88 is the safer default, and on a lighter day it produced a larger KV pool than 0.90 did on a busy one. If startup fails with that ValueError, lower it or close what is holding VRAM. Never force it.

vLLM will not share the GPU ComfyUI, llama-server and anything else CUDA must be stopped first. llama.cpp coexists with other things; vLLM does not. This is also why the embedder moved to the CPU — see Memory.

Cerveau-side settings that must match

model_ctx

Set to 98304 in ~/.config/cerveau/config.json. Mismatch and the packer plans against the wrong budget.

CRV_MODEL_NAME

qwen3.8-27b. vLLM validates the name; llama.cpp ignores it.

CRV_MODEL_KEY

Required. vLLM returns 401 without it.

What the benchmark actually showed

Two things worth knowing before you tune anything.

The MoE wins despite having more parameters

Decode is memory-bandwidth-bound, and agent work is decode-dominated. Only about 3B of 35B activate per token, so parking the inactive experts in RAM is nearly free — measured 70.3 tok/s at 3.44 J/token, the cheapest and fastest configuration tested, with 4/4 benchmark projects delivered and 11.1 GB of VRAM left over for other models.

Speculative decoding did not pay off

Nineteen configurations, none beat baseline Speculative decoding was benchmarked nineteen ways on this model on a 3090. Not one improved on the baseline: A3B decode is already too cheap for drafting to earn back its overhead. Do not add MTP to the MoE Core.

On a dense model it is a different story — a FastMTP sidecar at depth 3 measured 0.93 acceptance and 48.2 tok/s. Depth 6 collapsed to 0.23 and 38 tok/s, so the 0.93 is acceptance at depth 3, not headroom to extend.

Dense models pay for every offloaded layer

Every layer is touched every token, so --n-cpu-moe is meaningless and any CPU-resident layer costs on every token. A dense Core has to be fully GPU-resident. vLLM cannot offload at all — --cpu-offload-gb measured a 34× collapse, from 56.87 to 1.65 tok/s — which is why it can never serve the MoE, and why the two Cores are complements rather than competitors.

The metric that matters The vLLM dense Core is slower per token and better per task. What is being measured is tokens spent to reach a working answer, not tokens per second.

A third Core

A Core is an OpenAI-compatible URL and a description. Anything that speaks that API can be one, and adding it needs no code in the loop, the window manager or the guard. That is the whole reason the abstraction exists.

PreviousThe three modes NextConfiguration