Before touching any cleanup script, internalize these rules:
If Redis data can grow forever, it eventually will.
Kubernetes adds unique failure modes:
1
redis-cli INFO memory
Key fields:
used_memory_humanused_memory_rss_humanmem_fragmentation_ratiomaxmemorymaxmemory_policy1
redis-cli INFO keyspace
This tells you where keys live, not how large they are.
Redis is fast — until you store huge values.
Common offenders:
stat:* (Sidekiq statistics)1
2
3
4
5
6
7
8
9
10
11
DB=0
TOP=20
redis-cli -n "$DB" --scan \
| while read -r key; do
bytes=$(redis-cli -n "$DB" MEMORY USAGE "$key" 2>/dev/null)
[ -z "$bytes" ] && bytes=0
printf "%12s %s\n" "$bytes" "$key"
done \
| sort -nr \
| head -n "$TOP"
Never use KEYS *.
1
redis-cli BGSAVE
Locate and copy:
1
2
3
redis-cli CONFIG GET dir
redis-cli CONFIG GET dbfilename
cp /var/lib/redis/dump.rdb /backup/redis/pre-cleanup-$(date +%F).rdb
Why this works:
DEL blocks Redis while freeing memory.
UNLINK frees memory asynchronously.
1
2
3
4
5
6
DB=0
redis-cli -n "$DB" --scan MATCH 'Course#linked_course_uuids_and_self*' \
| while read -r key; do
redis-cli -n "$DB" UNLINK "$key"
done
1
2
3
4
5
6
7
DB=0
redis-cli -n "$DB" --scan MATCH 'stat:*' \
| while read -r key; do
redis-cli -n "$DB" UNLINK "$key"
sleep 0.01
done
By default:
stat:* keys never expireThis is expected behavior — and dangerous without tuning.
config/initializers/sidekiq.rb
1
2
3
4
5
6
7
8
9
10
11
12
Sidekiq.configure_server do |config|
config.on(:startup) do
Sidekiq.redis do |conn|
retention_days = 30
ttl = retention_days * 24 * 60 * 60
conn.scan_each(match: 'stat:*') do |key|
conn.expire(key, ttl)
end
end
end
end
1
2
3
4
class MyWorker
include Sidekiq::Worker
sidekiq_options retry: 5
end
Disable retries for non-critical jobs:
1
sidekiq_options retry: false
1
2
3
4
Sidekiq.configure_server do |config|
config.options[:dead_timeout] = 30 * 24 * 60 * 60
config.options[:dead_max_jobs] = 2000
end
Unbounded Redis is dangerous in containers.
1
2
redis-cli CONFIG SET maxmemory 512mb
redis-cli CONFIG SET maxmemory-policy allkeys-lru
Choose a value below your pod memory limit.
Run:
1
redis-cli MEMORY DOCTOR
If caused by historical peak:
Try:
1
redis-cli MEMORY PURGE
Guaranteed fix:
1
redis-cli INFO stats | egrep 'evicted_keys|expired_keys|keyspace_hits|keyspace_misses'
1
2
redis-cli ZCARD retry
redis-cli ZCARD dead
1
redis-cli --scan | head -n 20
If you fix retention, Redis becomes boring again — and boring is good.
If you want, this guide can be adapted into:
Just say the word.
]]>One of the smallest tests I use is also one of the dumbest:
hellohello againI am not doing that because I think it is a meaningful intelligence test.
I am doing it because I want to see whether the stack behaves differently on the second request. If I expect prompt caching or KV reuse, I want some sign of it in latency, logs, or runtime warnings. If nothing changes, or LM Studio logs complain about cache support, that already tells me something useful.
That tiny test changed how I think about debugging agents.
One thing I keep coming back to with local agents is this: if you only test them on real work, debugging takes forever.
A big task fails and you do not know why.
Was function calling broken? Did the model ignore the skill instructions? Did the prompt get too long? Did the agent runtime format messages wrong? Did LM Studio drop some warning you only notice after the fact?
When the task is large, every failure mode gets mixed together. That makes the whole setup feel mystical when it usually is not.
What helped me was building tiny puzzles.
By “puzzle,” I do not mean benchmark games or synthetic IQ tests. I mean small, repeatable probe tasks that isolate one capability at a time. A good probe gives you a yes-or-no answer about one layer of the stack.
That is much more useful than asking a vague question and hoping the agent “feels smart.”
This distinction matters.
A benchmark tries to rank systems.
A probe tries to isolate a failure.
A benchmark asks:
Which model is better?
A probe asks:
What broke?
For debugging, I care much more about the second question.
A good agent-debugging probe should be:
If the task is too broad, you learn almost nothing.
That is why I like tests like:
Those are not glamorous. They are useful.
I would break agent testing into five buckets:
You can think of this as a cheap local eval harness for yourself.
Here is the compact version:
| Probe | What it tests | Common failure |
|---|---|---|
| single-tool sanity check | basic function calling | model answers without using the tool |
| tool-loop continuation | post-tool reasoning | runtime stops after the tool result |
| skill trigger check | skill awareness | skill never enters context or gets ignored |
| buried instruction retention | prompt pressure | instruction gets lost in long context |
hello / hello again |
cache or runtime behavior | no reuse signal, warning in logs |
This is the first thing I test because if it is broken, everything above it becomes noise.
The goal here is not “can the model describe tool use?” The goal is “can the model produce the exact structured action the runtime needs, and can the runtime round-trip the result correctly?”
Give the agent one tool and one obvious task.
For example:
You have one tool:
read_file(path). ReadREADME.mdand tell me the first sentence.
Why this works:
Possible outcomes:
Once the trivial case passes, I like a short ladder:
That ladder tells you where the system starts to break.
At this stage, I want to see:
If any one of those is missing, debugging gets much harder.
This one is easy to confuse with general intelligence.
If you use skills, rulesets, or task-specific system instructions, you should test whether the agent notices and follows them before you test complicated work.
The puzzle here is not “is the output good?” The puzzle is “did the agent notice the right behavior cue?”
Create two nearly identical tasks where only one should trigger the skill.
For example:
If the humanizer skill is available and Task B does not activate the expected behavior, that is not a generic writing failure. That is a skill-selection failure.
Make the skill instruction produce a visible artifact.
Examples:
That way you can verify whether the skill was active without guessing.
That last one is common. The agent acts like it vaguely remembers the rule but not enough to follow it cleanly.
This is where local setups get weird fast.
A model can look fine on a short prompt and then fall apart once you add:
That is not a minor edge case. That is the normal shape of agent work.
So I like to test prompt pressure deliberately instead of waiting for a real task to reveal it.
Run the same task at different prompt sizes:
The task itself should stay almost the same. Only the context load changes.
That helps answer a very useful question:
Is the model bad at the task, or is the system bad under context pressure?
One of my favorite tiny tests is embarrassingly simple:
hellohello againI am not using that to test intelligence. I am using it to test cache behavior and runtime traces.
If the stack supports prompt caching or KV reuse, I want to see evidence in latency, logs, or runtime counters. If I expect cache reuse and nothing changes, that is already a clue.
This is where I often check LM Studio logs. I want to see:
That tiny test is not sufficient, but it is cheap and surprisingly revealing.
Ask the model to follow one simple instruction buried at the very end of a long prompt.
For example:
After reading everything above, answer with exactly:
CACHE_OK
If it misses that reliably only when the prompt gets large, you have learned something real about instruction retention under load.
This sounds obvious, but I have wasted plenty of time ignoring it.
If you change the model, the prompt, the agent settings, and the runtime configuration all at once, a bad result tells you almost nothing.
So when I run these probes, I try to be strict:
Half of debugging is refusing to create your own confusion.
This is the distinction that saves the most time.
When an agent fails, I try to ask:
If you do not separate those layers, you end up rewriting prompts when the real problem is message formatting, or blaming the model when the real problem is unsupported KV cache in the backend.
I usually debug in this order:
This is Pi, OpenCode, OpenHands, or whatever harness you are using.
Questions:
This layer is responsible for a lot more weirdness than people expect.
Questions:
This layer is boring until it is broken. Then everything looks cursed.
For a local stack, this is where LM Studio, Ollama, MLX, llama.cpp, or another runtime starts to matter.
Questions:
This is why I check LM Studio logs so often. If the runtime is complaining, I want to know before I waste an hour blaming the prompt.
Only after the first three layers look healthy do I spend much time blaming the model itself.
Questions:
Sometimes the answer really is “the model is weak at this.” But that should be the last conclusion, not the first.
If you are serious about debugging agents, text output alone is not enough.
You want a trace.
At minimum, I would want:
That is the minimum useful set.
Even a fake trace like this is better than nothing:
1
2
3
4
5
6
7
8
9
10
11
12
13
request_id=42
agent=opencode
model=gemma-4
prompt_tokens=4187
completion_tokens=96
ttft_ms=842
total_latency_ms=2310
tool_calls=1
tool_parse_ok=true
tool_roundtrip_ok=true
cache_reuse=false
stop_reason=tool_call
warning="RotatingKVCache Quantization NYI"
If I can capture something shaped like that for each probe, I can usually stop guessing and start comparing.
Once you have the basics, the metrics I care about most are:
That last one matters because random-seeming instability often is not random at all. It clusters around one part of the stack.
If I were setting this up from scratch, I would keep a small set of named probes.
Goal: verify basic tool calling.
Prompt:
Use the available file-read tool to read
README.mdand return only its first sentence.
Goal: verify that the agent continues after a tool result instead of narrating.
Prompt:
Read
README.md, then tell me how many top-level sections it has.
This forces a read, then a second reasoning step based on the returned content.
Goal: verify that the right skill or ruleset activates.
Prompt:
Humanize the following draft.
Expected result: visible behavior that clearly matches the skill instructions.
Goal: check prompt-pressure behavior.
Prompt:
[long filler context] Final instruction: reply with exactly
CACHE_OK
Goal: see whether repeated prompts behave differently.
Prompt sequence:
hellohello againThis is not deep, but it is cheap. For local debugging, cheap matters.
Goal: catch backend-specific failures early.
Procedure:
I want to catch things like:
Goal: test the full coding loop in miniature.
Prompt:
Read
foo.txt, replacecatwithdog, then verify the file changed.
This sounds silly, but it exercises the real sequence:
That makes it more valuable than a pure text-generation test.
This is the other trap.
A probe can pass while the real workflow is still broken.
Examples:
So I do not use probes to declare victory. I use them to narrow the search space.
I would also keep a tiny log for myself.
Nothing fancy. Just enough to compare runs without relying on memory:
This matters because agent debugging gets fuzzy very quickly. A failure diary turns “I think it got worse after that config change” into something you can actually verify.
The best puzzles are not the most clever ones. They are the ones that make failure obvious.
Bad puzzle:
“Review this medium-sized repository and suggest improvements.”
If it fails, you learn almost nothing.
Better puzzle:
“You have one tool. Read one file. Return one sentence.”
If it fails, you know where to look.
That is the mindset shift.
Do not start by asking whether the agent is smart. Start by asking whether one layer of the system is behaving.
This is worth saying because people drift into benchmark brain very easily.
The goal of these puzzles is not to prove that one model is superior in the abstract. The goal is to make the stack debuggable.
That means:
I still care about real tasks in the end. Of course I do. But I do not trust a real-task result very much if I do not have a clean probe suite underneath it.
That is especially true for local agent setups, where a lot of failures come from runtime behavior, prompt overhead, API compatibility, or cache paths rather than from some deep model limitation.
If I were debugging Pi or OpenCode against LM Studio from scratch, I would do this in order:
hello / hello again cache smoke test.That order is boring. It also saves time.
When an agent fails, I do not want a mystery. I want a small broken puzzle.
That is what lets me debug one layer at a time:
Real work is too expensive to be your only test harness.
Tiny puzzles are cheaper. And for debugging, cheaper usually wins.
]]>This is the setup guide I wanted while I was trying to make LM Studio work as a local engine for coding agents.
This post is not about theory. It is about getting a model running locally, exposing it through LM Studio’s OpenAI-compatible endpoint, wiring it into Pi or OpenCode, and checking that the stack is alive before you waste time debugging the wrong thing.
If you want the broader context on why I was doing this and where the setup still falls short, read the companion article:
Using LM Studio and Gemma as a Local Engine for Coding Agents
The target architecture is simple:
http://127.0.0.1:1234/v1.That does not give you a perfect local agent. It gives you a local model backend your agent can talk to.
Before you start, make sure you have:
One warning up front: model selection is constrained by memory much faster than people expect.
For example, I hit memory trouble when pushing into Qwen 35B-class MLX setups. Bigger is not automatically better if the runtime becomes unstable or dies during actual agent work.
Open LM Studio and load the model you want to serve.
My actual workflow is:
http://127.0.0.1:1234/v1.The important detail is that you are not just chatting in the LM Studio UI. You are turning it into a local API server.
Before configuring any agent client, make sure the LM Studio endpoint responds.
Run:
1
curl http://127.0.0.1:1234/v1/models
If that works, you should get a model list back.
If it does not work, stop there and fix the server first. Do not start editing Pi or OpenCode config until this responds.
These are the first settings I would touch before trying agent workloads.
This helped the most when I was seeing prompt stalls or weird runtime behavior.
If you see large prompts stuck at 0%, this is one of the first things I would change.
Do not leave context length on auto if you want more predictable behavior.
Choose an explicit number based on your machine and the kind of tasks you are running.
Gemma can leak reasoning markers into visible output. That is annoying in normal chat and actively bad in agent loops.
If you are seeing leaked thought markers, add stop tokens early instead of pretending this will sort itself out later.
~/.pi/agents/models.jsonIf you are using Pi-style agents, this is the general shape:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
"providers": {
"lm-studio": {
"id": "local-lm-studio",
"name": "LM Studio",
"api": "openai-completions",
"compatibility": "legacy-system-role",
"apiKey": "ollama",
"baseUrl": "http://127.0.0.1:1234/v1",
"models": [
{
"id": "qwen3.5-9b-sushi-coder-rl-mlx",
"_launch": true,
"name": "qwen 3.5 sushi coder",
"contextWindow": 84000,
"input": ["text"],
"reasoning": true
},
{
"id": "gemma-4-e4b-it-mlx",
"_launch": true,
"name": "Gemma 4 E4B",
"contextWindow": 84000,
"input": ["text"],
"reasoning": true
}
]
}
}
}
What matters here:
baseUrl should point at the LM Studio endpointapiKey is usually just a placeholder for local usecompatibility can matter if the client is picky about role handlingreasoning: true is only useful if your harness can handle reasoning output cleanlyThat last point is not academic. If the harness does not know what to do with reasoning output, turning it on can make the whole setup noisier instead of smarter.
For OpenCode, the provider block is usually simpler:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
{
"lm-studio": {
"models": {
"gemma-4-26b-a4b-it-mlx": {
"_launch": true,
"name": "Gemma 4 26B A4B IT MLX"
}
},
"name": "LM Studio",
"npm": "@ai-sdk/openai-compatible",
"options": {
"baseURL": "http://127.0.0.1:1234/v1"
}
}
}
The important part is not the shape of the JSON. The important part is that OpenCode is treating LM Studio like an OpenAI-compatible backend.
That is what makes this setup useful for experimentation. You can swap the model backend without rewriting the rest of the agent harness.
Do not jump straight into a huge repository review.
Do these first:
You want to prove the stack works in the smallest possible way before turning the task size up.
This matters more for local models than for stronger cloud models.
If Pi is dragging around a huge system prompt, a giant skill catalog, MCP server descriptions, tool docs, and every meta-instruction imaginable, you are burning context before the real work even starts.
If you are struggling with context pressure, simplify aggressively:
You are not trying to make the agent less capable. You are trying to stop wasting the context window on scaffolding.
This is the part people usually skip. They should not.
This is a machine constraint problem, not a prompting problem.
If the model is too large for the actual workflow, move down to something that stays alive under repeated turns.
This showed up for me as runtime weirdness and failed expectations around caching.
I would not assume KV-related optimizations are fully reliable just because they exist in a settings panel.
If the prompt sits at 0%, treat runtime cache state as suspicious immediately.
My default response is:
This is real and it matters.
In an agent loop, think-tag leakage can break parsing, pollute tool output, and waste context.
For the LM Studio runtime I was testing, shared KV for Gemma was not something I could treat as usable.
If that is part of your mental model for why the setup should be fast or stable, remove that assumption first.
One runtime error I hit looked like this:
1
2
Error: Error in iterating prediction stream:
NotImplementedError: RotatingKVCache Quantization NYI
That is useful because it tells you this is not a “maybe I should rewrite my prompt” situation.
It points to a runtime capability gap.
If the whole setup is failing, I would check things in this order:
127.0.0.1:1234?curl http://127.0.0.1:1234/v1/models work?That order has saved me time because it forces me to debug the runtime before I start rewriting prompts for no reason.
If I had to set this up again from zero, I would keep it boring:
curl http://127.0.0.1:1234/v1/models works.That sounds conservative. It is also much faster than trying to brute-force your way through three problems at once.
The biggest trap in local agent setup is mixing all the variables together.
Do not debug:
all at the same time.
Make the stack boring first. Then make it ambitious.
]]>The first time I tried to wire a local coding agent around Gemma, I thought the hard part would be the model.
It wasn’t.
The model looked flaky because my agent loop was flaky.
One of the first tasks I gave it was boring on purpose: find a file, make a small code change, then run the relevant test. The model did the first part correctly. It asked for the file. It asked for the test. Then it drifted. Instead of taking the next tool step, it started explaining what should happen next like a consultant with a checklist.
At first glance that looked like a model failure. It wasn’t. I was parsing the response stream too early and mishandling the turn after the tool result. The model never really got a clean chance to continue.
That changed how I think about coding agents.
A coding agent is not just a model with a bigger prompt. It is a small software system wrapped around a model. The model does the reasoning. The runtime does the state management, tool execution, file edits, and validation.
That distinction matters because people blame or praise “the model” for behavior the surrounding harness is actually causing.
After spending time with Gemma and OpenCode-style local workflows, I keep coming back to the same conclusion: the model is only one part of the system. The loop around it is what turns text generation into software work.
If I had to reduce the whole thing to one line, it would be this:
Most of what feels magical in a coding agent is just a model sitting inside a well-built loop.
At a high level, a coding agent has five moving parts:
A normal chatbot answers once. An agent reads, acts, checks what happened, then goes again.
Here is the simplest version of the flow:
1
2
3
4
5
6
7
8
9
User request
-> context builder
-> model
-> tool call
-> runtime executes tool
-> tool result goes back to model
-> patch / command / follow-up tool call
-> tests or lint
-> final answer
Not glamorous, but more useful than model marketing.
Before the model can do useful work, the agent has to decide what context to provide.
That usually includes:
This part gets hand-waved a lot. The agent is not dumping an entire repository into the model. It is assembling a useful working set.
In practice, good agents do some combination of:
The important point is simple: context is assembled. It is not magically remembered.
Once the context is ready, the model gets a turn.
For a coding task such as “fix the failing login flow,” the model is not supposed to immediately output code. A good model will first decide what information it still needs:
Reasoning helps, but reasoning on its own is not enough. If the runtime does not support tools and iteration, the model can only describe a plan. It cannot carry it out.
That is the gap between:
auth.rb and run the tests”auth.rb, running the tests, seeing the failure, and proposing a patchIf you have built one of these loops badly, you can feel the difference immediately. The model sounds smart. Nothing gets done.
The model does not directly touch your filesystem. It emits structured intent.
Depending on the runtime, that may look like a function call, JSON object, or tool invocation block. The meaning is the same:
“Run this shell command.”
“Read this file.”
“Apply this patch.”
This is one of the most useful mental models in the whole setup:
The model does not execute tools. The runtime executes tools.
That separation is what makes the system manageable. The runtime can validate arguments, reject unsafe actions, log what happened, and feed the results back into the conversation.
It also means a lot of agent bugs are not really model bugs. They are runtime bugs:
I ran into exactly this in my own local setup. The model looked flaky until I realized the harness was the flaky part.
One subtle point here: the runtime is not just a dumb pipe. It defines the contract. It decides which tools exist, what arguments are allowed, how results are formatted, and what the model gets back when something fails, times out, or succeeds. The model can only operate inside that contract.
This sounds boring until it breaks.
Different models and runtimes expect different message shapes, role names, and tool-call formats. Some want explicit tool messages with IDs. Some expect tool results folded back into a user turn. Some tolerate loose formatting. Some absolutely do not.
If you get this wrong, the failure mode is annoying because it does not always look like a protocol error. It just looks like the model got weird. It ignores a tool result. It repeats itself. It forgets what just happened. It starts narrating instead of acting.
That is another reason I hesitate when people talk about agent quality as if it were just a model ranking problem. A surprising amount of the real work is message plumbing.
This is the part many first-time agent builders miss.
The basic loop looks like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
messages = initial_context
while True:
response = llm(messages, tools=tools)
if response.tool_calls:
messages.append(response)
for call in response.tool_calls:
result = execute_tool(call)
messages.append(result)
continue
return response.final_text
The crucial line is continue.
After each tool result, the model needs another turn. That is how it moves from:
Without that loop, you do not have much of an agent. You have a one-shot assistant that knows how to talk about tool syntax.
The runtime also needs clear stopping conditions. A good agent should stop when the checks pass, when it is genuinely blocked and needs user input, or when another retry is just burning tokens without improving anything. Otherwise you get the other classic failure mode: the agent that keeps “working” long after it should have stopped.
This is what a healthy loop looks like in practice.
Imagine the user asks:
“Fix the failing login test.”
What happens next is usually something like this:
auth.rb and the matching test file.In rough pseudo-transcript form:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
user: Fix the failing login test.
assistant -> tool: run_test("bundle exec rspec spec/requests/login_spec.rb")
tool -> assistant: failure in "returns 401 for expired token"
assistant -> tool: read_file("app/services/auth.rb")
assistant -> tool: read_file("spec/requests/login_spec.rb")
tool -> assistant: [file contents]
assistant -> tool: apply_patch(...)
tool -> assistant: patch applied
assistant -> tool: run_test("bundle exec rspec spec/requests/login_spec.rb")
tool -> assistant: 1 example, 0 failures
assistant: Fixed. The token expiry check was comparing strings instead of timestamps.
That is the job. Not one huge leap of intelligence. A sequence of small moves grounded in feedback.
A naive agent does everything in sequence. Better agents can overlap independent work.
Reading three files at once is usually fine. Searching two directories in parallel is usually fine. Running a linter and a type check at the same time is often fine.
But the runtime has to know where parallelism stops being safe. Reading a file and patching it at the same time is a bug. Running a test against code that another step is still modifying is a bug. Parallelism is useful, but only when the operations are actually independent.
When the agent decides to change code, the safest path is usually not “rewrite the whole file.”
Better runtimes prefer targeted edits such as:
That approach helps for two reasons:
This is one reason patch-based workflows feel noticeably more reliable than naive full-file rewrites.
An agent is only useful if it can compare its changes against reality.
For software tasks, “reality” usually means one or more of:
The model proposes a change. The runtime runs the relevant checks. The model then sees the result and decides whether the job is actually done.
That is the difference between a flashy demo and a tool you might actually trust. The demo stops when the code looks plausible. The useful tool stops when the environment says the change holds up.
This is also where models run into a hard limit. They are good at predicting plausible next steps. They are much worse at knowing, from their own internal confidence alone, whether those steps actually worked. That is why verification is not optional. The model’s guess is not the ground truth.
When people say a coding agent “just kind of fell apart,” the failure is often boring:
This is why I am suspicious of sweeping claims about model quality without any discussion of runtime quality. A fragile harness can make a good model look bad. A disciplined harness can make a merely decent model feel much better than expected.
As the session gets longer, the agent’s job gets harder.
Every tool result, file read, and patch explanation consumes context window space. If you keep everything, the model eventually drowns in stale logs and low-value history.
So real agents need compaction strategies:
This is not the glamorous part of agent design, but it matters more than people think. A lot of agent failures are really context failures wearing a fake mustache.
There is also a tradeoff here that people skip past too quickly: compaction is lossy. Summaries are useful, but sometimes the exact detail you threw away is the detail you needed three turns later. Long-running agents are always balancing recall against context budget.
Different models are better or worse at planning, tool use, structured output, and code generation. That absolutely affects the experience.
But once you start building agents, you realize something uncomfortable:
A strong model in a weak harness is frustrating. A decent model in a strong harness is often more useful than it has any right to be.
The harness determines whether the model can:
That is why two products using similarly capable models can feel wildly different in practice.
This point matters even more for local agents.
Cloud coding products usually have polished runtimes, mature prompt formatting, and enough infrastructure around the model to hide a lot of rough edges. Local setups are less forgiving.
You run into issues like:
That does not make local agents pointless. I still like them. It just means the boring systems work matters even more. If your local agent feels unstable, it may not need a smarter model first. It may need a better loop, cleaner context, and stricter verification.
Another missing piece in a lot of simplified agent diagrams is the operating envelope.
Real coding agents usually do not have unlimited power. Some tools are read-only. Some filesystem paths are writable and others are not. Some commands require explicit approval. Network access may be blocked. Destructive operations may be denied or wrapped in extra checks.
That is not an annoying implementation detail. It is part of how the system works. The runtime is not just giving the model hands. It is also deciding what the hands are allowed to touch.
The same goes for observability. If the agent cannot show you what tool it called, what came back, what got truncated, and why it stopped, debugging turns into superstition.
The mental model I keep settling on is this:
The runtime gives the model senses, memory, and hands:
Once you see the system that way, a lot of confusing behavior stops being confusing. You stop asking, “Why didn’t the model just do it?” and start asking the more useful question:
“What part of the agent loop failed?”
There are more advanced pieces beyond the basic loop:
Those matter, but they come later.
The first-order problem is still the same boring one: can the model ask for a tool, can the runtime execute it, can the result get fed back correctly, and can the system verify the change before it stops?
An LLM coding agent does not build software by generating one brilliant answer.
It builds software by repeatedly doing four things well:
If you want to build a better local agent, spend less time imagining a magical autonomous coder and more time improving those four steps.
What looks like intelligence is often just good plumbing.
]]>I did not start this experiment because I wanted a nicer chatbot on my laptop.
What I wanted was much more specific: a local model endpoint I could plug into agent-style workflows for code review, repo questions, bounded refactors, and private documentation-heavy tasks. Something that felt close enough to the OpenAI-compatible APIs many tools already expect, but without sending every prompt, diff, and internal doc set to the cloud.
That turned out to be possible. It was also a lot less smooth than the demo videos make it look.
The core setup that worked best for me was LM Studio plus a GGUF build of Gemma 4 26B. Once I got the settings under control, it became a usable local engine for agent clients. Not perfect. Not something I would trust blindly. But good enough that I would actually use it.
This post is the version I wish I had found before I started.
There were a few reasons I kept pushing on local setup instead of giving up and using cloud models for everything.
First, privacy. If I am experimenting with agent workflows against private repositories, internal notes, or ugly half-finished local docs, I want the option to keep all of that on my machine.
Second, cost. Agent loops are noisy. They read files, retry, summarize, call tools, and sometimes spiral. That is manageable when you are testing locally. It gets annoying fast when every bad prompt is burning money.
Third, iteration speed. I wanted to try weird things: change prompts, swap harnesses, feed in local notes, break the loop, fix the loop, try again. Local models are slower in raw quality terms, but they are very forgiving for this kind of experimentation.
And fourth, I wanted to understand the boundary between “local model” and “local agent.” Those are not the same thing, and a lot of the confusion here comes from treating them as if they are.
After trying a few variations, this is the shape that felt the most practical:
That last bullet matters.
LM Studio gives you model hosting, local inference, and a familiar API surface. That is useful. But it is not the agent. It is the model backend.
The agent still has to do the hard part:
If that loop is weak, a good local model still feels unreliable. If the loop is solid, even an imperfect local model becomes usable much faster than you would expect.
The setup details ended up being long enough that I split them into a separate post:
LM Studio local agent runbook: Pi and OpenCode step by step
That runbook covers:
~/.pi/agents/models.jsonFor this specific setup, GGUF was the path of least resistance.
I am not making a universal claim that GGUF is always better than every other format. I am saying that in my own tests, it was the easiest way to get Gemma into a stable LM Studio workflow for agent-style tasks.
The key word there is stable.
I care less about benchmark bragging rights than about whether the model can survive long prompts, diffs, and repeated turns without drifting into nonsense or getting stuck in some half-broken internal state. GGUF plus LM Studio gave me a workflow I could keep using. That was enough for me.
The first wave of problems had very little to do with “intelligence” and a lot to do with runtime behavior.
This was the most annoying one.
I would send a large diff or a code-heavy prompt and LM Studio would sit there looking busy while doing nothing useful. In practice, the model was not making forward progress. It just looked stuck.
The fix that helped most was turning off Unified KV Cache and setting the context length manually instead of leaving it on auto.
That was my first clue that local agent work is often about runtime stability, not just model selection.
Gemma can leak internal reasoning markers or thought-channel fragments into the visible output. If you are just chatting in a UI, that is ugly. If you are piping the response into an agent workflow, it is worse than ugly. It can break downstream parsing or contaminate structured output.
I saw this most often when the prompt mixed code review instructions with tool-like formatting expectations.
Adding stop tokens helped. Tightening the system prompt helped. But the deeper lesson was that local agent clients need output sanitation anyway. You should not assume the model will always hand you clean final text.
A local coding model can look fine on a short task and then fall apart the moment you give it a real repository diff, a stack trace, and a few tool results in the same conversation.
This is where local setups stop feeling like “cheap cloud replacements” and start feeling like engineering systems with real constraints. Context is not just a number on the model card. It is a budget, and agent loops spend that budget aggressively.
This one took me a while to admit.
On a single prompt, Gemma could often produce a respectable answer about code. That made me optimistic. But agent workloads are harsher than single prompts. They require consistency across turns, clean tool-use behavior, and enough formatting discipline that the harness can keep going.
A model that gives one good answer is not automatically a good agent backend.
These are the problems I actually ran into while trying to make this usable for agents.
One thing I wish more local-LLM writeups did: include the ugly error text.
When you are debugging this stack, vague advice is not that helpful. Concrete failure signatures are.
This was one of the first practical limits I hit.
On paper, a larger local model always sounds attractive. In practice, once I pushed into Qwen 35B-class MLX setups, memory pressure became the story. The model might load partway, fail during inference, or behave badly once context and KV usage started climbing.
That pushed me toward setups that were slightly smaller but much easier to keep alive for repeated agent turns.
This is one reason I think local agent work should be judged as a system, not just a model preference. A model that looks stronger in theory is not useful if it keeps falling over in the workflow you actually want.
This was another source of confusion.
I expected KV cache behavior to be a straightforward performance win. In this setup, it was not. My experience was that KV caching on the MLX path inside LM Studio was not something I could treat as fully reliable for agent-style workloads.
That matters because agent sessions are exactly the kind of workload where you want caching to help. Long turns, repeated context, retries, and follow-up prompts should benefit from it. But if that layer is unstable, it stops being an optimization and turns into a source of weird failures.
One example I hit looked like this:
1
2
Error: Error in iterating prediction stream:
NotImplementedError: RotatingKVCache Quantization NYI
If you run into that kind of error, I would treat it as a runtime capability gap first, not something you are supposed to fix with prompting.
In plain English: the stack is telling you that the KV/cache path you are trying to use is not fully implemented for that configuration yet.
This deserves to be called out separately because it wasted a lot of time for me.
Gemma plus shared KV sounded like exactly the kind of optimization that should help agent loops. In practice, in LM Studio runtime, I had to treat shared KV as not supported for the setup I was using.
That matters because if you assume shared KV is working, you can end up chasing fake explanations for behavior that is really coming from unsupported runtime features.
My rule here became simple: if Gemma is behaving strangely and shared KV is part of the configuration, remove that variable first.
This was the most visible symptom.
When I saw LM Studio sit at 0% on a prompt that should have started normally, the issue often traced back to invalid or broken KV caching state. Turning off Unified KV Cache and resetting the session usually helped more than trying to tweak the prompt itself.
That is why I would treat 0% prompt stalls as a runtime problem first, not a prompting problem first.
If I hit this again, my default response would be:
That sequence saved me time because it kept me from debugging the wrong layer.
Another important lesson here: the prompt is often innocent.
If the runtime state is broken, rewriting the prompt five times is just a way to waste an afternoon.
This one showed up often enough that I think it belongs in the troubleshooting section, not just the general setup notes.
Gemma can leak internal reasoning or think-tag markers into visible output. Depending on the client, this may show up as raw reasoning blocks, partial channel markers, or output that looks like it forgot to separate internal thought from the final answer.
That is annoying in a chat window. In an agent loop, it is worse because it can:
What helped me:
I would treat this as a normal engineering concern when using Gemma for agents, not as a weird edge case.
Another lesson from the agent side: the prompt budget disappears fast.
If Pi is carrying a huge system prompt, a long skill catalog, MCP server descriptions, tool instructions, and model-specific behavior rules, you can burn a surprising amount of context before the real task even starts.
For local models, that overhead hurts more.
The practical fix was to simplify the Pi prompt aggressively:
The goal is not to make the agent dumb. The goal is to stop spending half the context window on scaffolding.
For local coding models in particular, I think this matters a lot. Smaller prompt overhead usually means:
If the model is struggling, one of the highest-leverage fixes is often not “find a smarter model.” It is “stop stuffing the prompt with framework overhead the model does not need.”
If I were setting this up from scratch and it started failing, this is the order I would check things:
That sequence is not elegant, but it is the one I trust now.
These were the settings and habits that moved the setup from “interesting demo” to “something I would actually plug into a tool.”
This was the biggest quality-of-life improvement in my setup for long prompts and review-style workloads.
If you are seeing prompt hangs, weird stalls, or long code-heavy requests that never quite start, this is the first thing I would change.
I had better results when I chose an explicit context size instead of trusting auto mode.
That gave me more predictable behavior, especially when I was switching between smaller code questions and much larger review tasks.
If your model output is spraying thought tags into normal text, stop tokens are worth trying immediately.
This is not a complete fix. It is more like putting guardrails around a messy edge. Still worth doing.
This is less a setting and more a survival rule: do not assume the local model should ingest your entire diff, your entire style guide, and a huge pile of tool output in one go.
Chunk work where you can. Summarize aggressively. Keep the agent loop disciplined.
I would not hard-sell one magical sampling profile for every coding task. In practice, I found that code review, repo Q&A, and action-oriented tool use want slightly different behavior.
For me, the more important point was not the exact number. It was avoiding the temptation to keep turning the model into a “creative” assistant when what I actually needed was predictable structured behavior.
This is the part that is usually missing from local LLM articles.
Running Gemma in LM Studio does not give you a coding agent. It gives you a model server.
To get agent behavior, something above that server has to implement a loop like this:
That sounds obvious when written out. It is also where a lot of local setups quietly fail.
If the harness does not execute the tool correctly, the model looks dumb. If the harness appends tool results in a format the model does not handle well, the model looks confused. If the harness keeps dumping huge command output back into context, the model looks forgetful.
The local endpoint is only one piece of the system.
What I like about LM Studio in this role is that it can behave like a local OpenAI-compatible backend. That makes experimentation easier because a lot of agent tooling already assumes that style of interface.
So the setup I kept coming back to was:
http://localhost:1234/v1That works well for:
What it does not solve is model capability mismatch. If your agent framework expects extremely clean function calling, long-context reliability, or excellent multi-step planning under pressure, local Gemma may still feel rough compared to stronger cloud models.
That is not a failure of LM Studio. It is just the reality of the stack.
Once I stopped expecting a local model to be a universal drop-in replacement for the best cloud systems, the setup got much more useful.
Here is where I think it makes sense.
This is one of the better use cases because the task is bounded and the output format is easy to evaluate.
The model can read a diff, point out obvious risk, summarize a change, and flag suspicious sections. You still need judgment. You still need tests. But it is useful enough to be worth keeping around.
If you want to ask questions about a codebase, architecture notes, or internal docs that you do not want to upload elsewhere, local is attractive.
This gets even better when the agent harness is disciplined about retrieval and does not just dump huge blobs of text into the prompt.
I would trust this more for “rename this pattern in these files” than for “redesign the authentication system.”
That distinction is important. Local models can be very handy for narrow, repetitive engineering work. They are much less trustworthy when the task becomes open-ended or architectural.
If the task depends on local notes, internal guides, or a private personal knowledge base, a local endpoint starts to feel compelling even when the model is not state of the art.
Sometimes privacy and convenience matter more than squeezing out the last 10% of model quality.
This setup is useful, but I would not oversell it.
Single-turn demos flatter local models. Multi-step agent loops expose every weakness.
You notice formatting issues, small reasoning slips, and context handling problems much more when the model has to survive several turns in a row.
Even when the model technically accepts a large context, that does not mean it uses it well.
If your workflow depends on feeding in very large diffs, long logs, and many tool results without careful pruning, you will probably have a bad time.
Some failures are obvious. The model outputs malformed JSON. Or it leaks thought tags into a tool call. Or it starts narrating instead of acting.
Some failures are subtler. The tool call shape is technically valid but not useful. The model calls the wrong tool. The model forgets why it called the tool in the first place.
Again, this is why the harness matters so much.
If I need maximum reliability on a complex, high-context, multi-step coding task, I would still reach for a stronger cloud model first.
The local setup wins when privacy, cost control, experimentation, or offline access matter enough to justify the trade.
If your goal is to build a local engine for agent usage, I think LM Studio plus Gemma is a reasonable place to start.
Not because it is perfect. Because it is accessible.
You get a local server, an OpenAI-compatible endpoint, and a model that is capable enough to make the experiment real. That is a good combination for people who want to learn how local agents actually work instead of just reading about them.
I would recommend it to:
I would not recommend it as your only serious option if:
My own conclusion is pretty simple.
LM Studio plus Gemma can absolutely work as a local engine for agent workflows. It is good enough for real experiments and some real tasks. But the useful mental model is not “I installed a local genius on my laptop.”
It is “I built a constrained local backend, then did the engineering work required to make an agent around it behave.”
That framing is less glamorous. It is also much closer to the truth.
]]>Last week I spent an evening trying to get Gemma 4 (26B) to run a simple coding task through my own agent: “find all the Ruby files in this directory and replace before_filter with before_action.” The first tool call worked perfectly. The model correctly asked to run find . -name '*.rb'. But when I fed the file list back to it, instead of calling sed or a file editor, it started explaining what I should do next, as if I were asking for advice rather than expecting it to act.
I bumped the temperature down. I rewrote the prompt three times. I tried adding explicit instructions like “you must call a tool.” Nothing helped.
The problem wasn’t Gemma 4. The problem was my agent. I was treating it like a chatbot with tool access, but what I actually needed was a state machine.
Ollama gives you model execution, streaming, and a tool call output format. It even supports a thinking channel now. But it doesn’t give you an agent loop. It doesn’t execute tools, manage state, or decide whether to call the model again. It just returns whatever the model outputs and leaves the rest to you.
llama.cpp is even more bare. You get raw model inference and nothing else. That’s fine if you want to build everything from scratch. But it means you can’t just drop in a tool schema and expect multi-step behavior to work.
This gap is where things fall apart. The model knows how to request a tool call. But if your runtime doesn’t execute that call, append the result to the message history, and feed it back into the model, the conversation stops dead. Or worse, the model improvises. That usually means it starts describing what would happen if someone ran the tool, instead of actually asking for it to run.
Here’s the core loop, stripped down:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
messages = initial_messages
while True:
response = llm(messages, tools=tools)
if response.tool_calls:
messages.append(response)
for call in response.tool_calls:
result = execute_tool(call)
messages.append({
"role": "tool",
"tool_call_id": call.id,
"content": json.dumps(result)
})
# loop back
else:
return response.content
That’s it. The continue back into the model after each tool execution is the part most people miss, or implement badly. Without it, you get one tool call and then nothing.
My original code collected the streaming response and checked for tool calls, but I was only looking at the first response chunk. With Gemma 4’s thinking channel enabled, the first chunk contained reasoning text, and the actual tool call came several chunks later. I was executing on incomplete output.
The fix was simple: accumulate the full stream, then parse. Don’t act on partial data.
When you use something like Claude Code or Codex, the smooth experience comes from a tight integration between what the model exposes and what the agent runtime does with it. Here’s a mapping of the main LLM features to what your agent needs to support.
What the model does: Outputs structured tool call objects with a name and arguments. It doesn’t call the tool. It just signals intent. In the raw token stream, this often looks like a special marker followed by JSON:
1
2
3
</think>
{"name": "run_shell", "arguments": {"command": "ls -la"}}
What the agent must build: Parse the tool call from the stream, validate the JSON arguments, dispatch to the right function, capture stdout/stderr/exit code, and return the result as a structured tool role message. The model expects the result to come back with the same tool_call_id it used. If you lose that ID, the model has no way to know which tool call this result corresponds to. That matters even more when the model fires multiple tool calls in parallel.
What the model does: Emits reasoning tokens before the actual answer or tool call. These tokens represent the model’s “chain of thought,” the step-by-step reasoning that leads to a decision. In some models, these tokens are hidden from the final output but still influence the model’s behavior.
What the agent must build: Separate the thinking tokens from the actual output. You have two choices: show them to the user (for transparency, like Claude Code does) or hide them (for a cleaner UI). Critically, thinking tokens must not be treated as tool calls or as the final answer. They’re an internal monologue. If your agent confuses them with real output, the user sees garbled text and the tool loop breaks.
I found that enabling the thinking channel for simple tasks actually made things worse. The model would reason out loud about what tool to call, then somehow convince itself the task was already done. Now I only enable reasoning for the planning phase, figuring out what steps to take, and disable it for the actual tool execution loop.
What the model does: Emits tokens one at a time (or in small batches). Tool call JSON often arrives across multiple chunks. Thinking tokens and answer tokens can interleave.
What the agent must build: A streaming buffer that accumulates all chunks until the model signals completion. Only then should you parse the full response to decide what happened. If you act on partial data, you’ll try to execute a tool call with truncated JSON, or display an incomplete answer to the user.
This is the most common bug I see in custom agents. The stream arrives in chunks, the first chunk looks like a tool call, the agent fires off the tool, and the rest of the tool call data arrives too late. Then the model gets confused because it never got a result for the full tool call it made.
What the model does: Expects messages in a specific format. Different models have different expectations. OpenAI models expect system, user, assistant, and tool roles. Gemma 4 works better with fewer role types — mainly user and model — with system instructions flattened into the conversation context.
What the agent must build: A message format adapter that translates between your internal representation and what the model expects. If you’re using Ollama’s OpenAI-compatible endpoint, the adapter is built in. If you’re talking to the model directly (llama.cpp, or Ollama’s native API), you need to handle this yourself.
The thing that tripped me up: I was appending tool results with a tool role, but my Ollama setup expected them as part of a user turn. The model received messages in a format it didn’t recognize, and the tool loop silently broke — no error, just the model ignoring the result and generating something else.
What the model does: Has a fixed context window (e.g., 8K, 16K tokens). Everything in the message history — user prompts, model responses, tool results — consumes tokens. When you exceed the window, older tokens get truncated.
What the agent must build: A strategy for managing context growth. Tool results can be large — a grep -r across a codebase might return thousands of lines. If you dump every tool result into the message history without thinking, you’ll burn through your context window in three turns.
Common approaches: summarize tool results before appending, truncate output that exceeds a threshold, or maintain a separate “memory” that the model can query instead of keeping everything in the active context. For coding tasks, I’ve had good luck truncating file contents to relevant sections and summarizing long shell output.
What the model does: Can emit multiple tool calls in a single response when it determines they’re independent — for example, reading five files at once. The model expects these to be executed and the results returned in the same turn.
What the agent must build: Logic to detect multiple tool calls, decide whether to run them in parallel or sequentially, and collect all results before feeding them back to the model. Running independent file reads in parallel is a nice optimization, but running dependent tool calls in parallel (read a file, then write to it) is a bug.
What the model does: Signals that it’s done generating by hitting a stop sequence or a special end-of-turn token. The runtime (Ollama, llama.cpp) tells you when generation is complete.
What the agent must build: Use the completion signal to decide the next step. If the response contains tool calls → execute and loop. If it’s plain text → return it to the user. Without reliable completion detection, you can’t reliably distinguish between “the model is still thinking” and “the model is done.”
Here’s how these pieces interact in a working agent:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
User sends request
↓
Agent builds message history (with context management)
↓
Agent calls LLM with tools schema
↓
LLM streams tokens (thinking + answer + possibly tool calls)
↓
Agent buffers the full stream
↓
Agent parses: thinking tokens → tool calls OR final answer
↓
If tool calls:
→ Agent executes them (parallel if independent)
→ Agent captures results
→ Agent appends results to message history
→ Agent loops back to LLM call
If final answer:
→ Agent returns to user
Every arrow in that diagram is a place where something can go wrong. The streaming buffer is where partial-data bugs live. The message history is where format mismatches cause silent failures. The tool executor is where you need to handle errors, timeouts, and large outputs.
The message type:
1
2
3
4
type Message =
| { role: "user"; content: string }
| { role: "assistant"; content?: string; tool_calls?: ToolCall[] }
| { role: "tool"; tool_call_id: string; content: string }
A tool schema — nothing fancy:
1
2
3
4
5
6
7
8
9
10
11
{
"name": "get_weather",
"description": "Get weather by city",
"parameters": {
"type": "object",
"properties": {
"city": { "type": "string" }
},
"required": ["city"]
}
}
The core loop:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
async function runAgent(messages: Message[]): Promise<string> {
while (true) {
const response = await llm(messages)
if (response.tool_calls) {
messages.push(response)
for (const call of response.tool_calls) {
const result = await executeTool(call)
messages.push({
role: "tool",
tool_call_id: call.id,
content: JSON.stringify(result)
})
}
continue
}
return response.content
}
}
One thing I picked up from watching how Claude Code behaves: it doesn’t use a single model call for everything. There’s a planning phase (which may involve tool calls) and an execution phase (which produces the final answer). You can approximate this by using one pass of the model to decide on steps and call tools, then a second pass — sometimes even with temperature set to 0 — to synthesize the result.
It’s not strictly necessary for simple tasks, but once you’re doing multi-step refactoring work, the split makes the agent more predictable. The planner can wander through tool calls without worrying about producing a clean final answer. The executor gets a complete set of tool results and just needs to summarize.
Just use Ollama to start. You’ll save yourself days. The tool support works out of the box, the thinking channel is already wired in, and you can focus on building your agent loop instead of wrestling with low-level inference.
Switch to llama.cpp only when you need control over something that Ollama hides from you. I haven’t needed to yet, but when I do, it’ll be because I want to optimize the prompt format or handle the thinking channel more precisely.
You don’t need a complex session system. You don’t need the full Ollama API. You don’t need abstractions around abstractions. You need:
gemma4:26b26B fits in unified memory alongside everything else I’m running, and the tool calling is reliable enough that I actually use it for real tasks now — not just experiments.
The thing I keep coming back to is that model quality matters less than people think once your loop is correct. A good agent loop makes a mediocre model feel capable. A broken agent loop makes a great model feel useless.
]]>We’ve all been there. You want to build a Retrieval-Augmented Generation (RAG) system, so you do the “standard” thing: you chunk your entire database, throw it into a vector database, and wire it up to a massive cloud LLM.
Then the bill arrives.
Worse, you realize that while the cloud model is brilliant at reasoning, it suffers from “lost in the middle” syndrome when fed thousands of tokens of irrelevant context. You are paying a premium to confuse the smartest models on the market.
The solution isn’t to stop using cloud models—it’s to stop using them for the dirty work. By shifting the indexing, embedding, and initial retrieval phases to a local environment using tools like Ollama and LightRAG, you can build a highly precise, multi-tenant knowledge graph that only sends the most critical context to the expensive models.
Here is how we designed a local, Hub-and-Spoke Graph RAG system using PostgreSQL, and the deep architectural lessons we learned along the way.
Traditional RAG relies heavily on Vector Databases (Dense Retrieval). While great for finding semantically similar sentences, vectors are terrible at understanding relationships. If a customer asks, “How does feature X affect my billing?”, a pure vector search might pull up a document about Feature X, and a separate document about Billing, but completely miss the bridge between them.
This is where LightRAG steps in. It builds a Knowledge Graph (GraphRAG), extracting entities (nodes) and their relationships (edges).
Our architecture uses a central Python Gateway to orchestrate this data flow from our existing PostgreSQL database into isolated LightRAG workspaces.
llama3.1 (for local reasoning/extraction) and nomic-embed-text (for embeddings).By running a daily ETL (Extract, Transform, Load) script, we pull rows from Postgres, format them into highly structured markdown blocks, and push them to specific LightRAG workspaces (e.g., customer_facing, internal_dev, product_codex).
One of the biggest mistakes in enterprise AI is creating a single, monolithic vector database. If you dump your API documentation, product roadmaps, and English learning resources into the same index, the LLM’s latent space gets muddy.
By utilizing Multi-Workspace Isolation, we apply a principle as old as database design: separation of concerns. The Gateway ensures that a customer asking a grammar question only queries the customer_facing graph, while a developer debugging an endpoint queries the internal_dev graph. It drastically reduces hallucination and limits unauthorized data access.
Code and product documentation are inherently relational. A function calls another function; a product feature requires a specific database schema.
When LightRAG processes a document, it doesn’t just store the text. It uses the local Ollama model to actively read the text and extract a graph. The initial ingestion is slow—local LLMs churn hard to build these relationships—but the querying is lightning fast. When you query the system, you perform a Hybrid Retrieval: BM25 (exact keyword) + Vector (semantic) + Graph (relational).
We leverage a cheap, local LLM to do the heavy lifting of reading and mapping the data (the Graph extraction phase). We only use expensive cloud models (like Gemini) when the user asks a highly complex, reasoning-heavy question. The local system retrieves the perfect 800-token context block from the graph, and we forward only that precise block to the cloud model. We turned a $100/month prompt habit into pennies.
While this local Docker Compose stack is incredibly powerful, preparing it for massive scale requires a few architectural evolutions.
Right now, if 100 users ask the chatbot, “How do I reset my password?”, the system performs the full graph-retrieval and generation process 100 times. The Fix: Implement a Semantic Cache layer in front of the API Gateway. If the embedded intent of a new query has a 95% similarity match to a recently answered query, return the cached response instantly. This drops latency from seconds to milliseconds.
Currently, our routing is deterministic (e.g., if user_type == 'dev'). As the system grows, we can deploy a micro-model (like an 8B parameter model) at the Gateway level to act purely as a “Traffic Cop.”
The Fix: The Traffic Cop reads the prompt and decides which workspace to query. If a Project Manager asks, “Did the new API endpoint cause customer complaints?”, the routing agent can intelligently query both the internal_dev workspace and the customer_facing workspace, synthesizing a cross-functional answer.
Batch syncing via cron jobs leaves the knowledge base out of date for up to 24 hours. The Fix: Move from an ETL script to an Event-Driven architecture using PostgreSQL triggers or a message broker (like RabbitMQ/Kafka). When a developer merges a pull request or updates a PRD, a webhook instantly fires a payload to LightRAG, keeping the Knowledge Graph updated in near real-time.
Building AI systems isn’t just about calling the smartest API; it’s about systems engineering. By treating LLMs as modular components within a traditional software architecture—using local models for data processing and graph generation, and strictly controlling data flow via workspaces—you build systems that are not only cheaper to run, but exponentially more accurate.
The future of RAG isn’t just bigger context windows; it’s smarter, structured, local retrieval.
]]>In 2026, the bottleneck for AI coding isn’t model intelligence—it’s the Context Tax. As agents like Claude Code (CC) and Codex become more autonomous, they tend to “over-read” your codebase, leading to massive input bills and hit-rate limits.
Here is the definitive breakdown of how to architect your workflow for maximum token thrift.
The Idea: Move from “sending files” to “sending blueprints.” Use high-density index files to guide the AI.
ai-codex or RepoMix.ls -R or cat on 50 different files just to find a single variable. Huge savings on “discovery” tokens.The Idea: Silence the AI’s “inner polite assistant.” Every word of “Sure, I’d be happy to help!” is a token you paid for.
/caveman full in CC or add Output: Telegraphic, fragments only, no preamble to your rules.The Idea: Stop letting 2,000 lines of “Test Passed” logs flood your chat history.
npm install log into a 3-line summary. Keeps your conversation “clean” for much longer.rtk --raw <command>.The Idea: Don’t read the haystack to find the needle. Use AST (Abstract Syntax Tree) indexing.
The Idea: Intercept and optimize the communication between your IDE and the LLM.
config.json.The Idea: Use high-density languages or “Garbage Collection” to keep the window small.
/compact command or Session Layering.tmp folder—delete it often.For the ultimate 2026 setup, combine these:
ai-codex (The Map)CocoIndex (The AST Surgeon)Lean-ctx (The Token Shredder)Caveman mode (The Assistant Muzzle)Deeper Thinking: Token saving isn’t just about money; it’s about Model IQ. The more “junk” tokens (logs, politeness, redundant imports) you feed an AI, the lower its effective reasoning becomes. A lean context is a smart context.
]]>In the world of containerization, Docker has long been the household name. However, for developers who need more than just a place to run a single process, Incus has emerged as the premier community-driven alternative.
While Docker focuses on Application Containers (packaging a single app), Incus focuses on System Containers (packaging a full Linux OS). Think of Incus as a way to create “instant Virtual Machines” that run at the speed of a container.
| Feature | Docker | Incus |
|---|---|---|
| Philosophy | “One process per container” | “One full OS per container” |
| Primary Use | Microservices, CI/CD, Deployment | Development Labs, AI Sandboxing, VPS replacement |
| Init System | No (Usually just entrypoint) |
Yes (systemd, OpenRC work natively) |
| Security | Process-level isolation | Unprivileged containers by default + VM support |
| Persistence | Volatile (requires Volumes/Bind mounts) | Persistent (acts like a physical disk) |
| Hardware | Hard to pass through GPUs/USB | Native, low-latency device passthrough |
If you already know Docker, learning Incus is a matter of mapping your existing knowledge to a new set of verbs.
| Action | Docker Command | Incus Command |
|---|---|---|
| Start a container | docker run -d --name web ubuntu |
incus launch images:ubuntu/24.04 web |
| List containers | docker ps |
incus list |
| Access shell | docker exec -it web bash |
incus shell web |
| Stop container | docker stop web |
incus stop web |
| Remove container | docker rm -f web |
incus delete -f web |
| Create Image | docker commit web my-image |
incus publish web --alias my-image |
| View Logs | docker logs web |
incus info --show-log web |
| Copy Files | docker cp file web:/path |
incus file push file web/path |
Incus is now officially supported in the latest Ubuntu repositories, making installation a breeze.
1
2
3
4
5
6
7
8
9
# Install the core packages
sudo apt update && sudo apt install -y incus
# Add your user to the management group
sudo usermod -aG incus-admin $USER
newgrp incus-admin
# Initialize the system (Interactive Wizard)
incus admin init
Tip: During init, choosing ZFS or Btrfs for storage allows for near-instant snapshots.
Unlike Docker Hub, Incus uses multiple “remotes.” The most common is the community-maintained images: server.
1
2
3
4
5
# Launch a persistent Ubuntu 24.04 container
incus launch images:ubuntu/24.04 dev-box
# Launch a MicroVM (for AI sandboxing or extra security)
incus launch images:ubuntu/24.04 ai-box --vm
Instead of manual configuration, you can use Profiles to apply settings (like GPU access or mounted folders) to many containers at once.
1
2
3
4
5
6
7
8
9
10
# Create a profile for Rails development
incus profile create rails-dev
# Add a device to map your code folder from the host
incus profile device add rails-dev my-code disk \
source=/home/user/projects/app \
path=/root/app
# Apply this profile to your container
incus profile add dev-box rails-dev
This is where Incus shines over Docker for development. Before making a big change:
1
2
3
4
5
# Create a snapshot
incus snapshot create dev-box pre-upgrade
# Messed up? Restore instantly
incus restore dev-box pre-upgrade
Yes, you can have the best of both worlds. To run Docker inside an Incus container (nesting):
1
2
3
incus config set dev-box security.nesting=true
incus restart dev-box
# Now install docker inside the dev-box as usual!
Use Docker when you have a finished app that you want to ship to the cloud. Use Incus when you are building that app. It provides a stable, persistent, and high-performance environment that handles system services and hardware with ease—all while keeping your host machine clean and organized.
]]>In a modern microservices or multi-app architecture, Redis is often the “glue” that holds everything together. It manages your user sessions, speeds up your app via caching, and handles background job orchestration.
However, many teams fall into the trap of the “Single Point of Failure”—using one Redis instance for everything. If that instance blips during a cloud provider node upgrade, your entire platform goes dark. Here is the blueprint for a “Bulletproof” Web Service.
The most important best practice is Decoupling. You should split your Redis usage into three distinct functional groups (StatefulSets in Kubernetes).
| Group | Data Type | Priority | Failure Impact |
|---|---|---|---|
| Session | User IDs, CSRF tokens | Critical | Users are logged out (Global outage) |
| Cache | HTML fragments, API results | Medium | Site slows down (Degraded performance) |
| Sidekiq | Background Job Metadata | High | Emails/Uploads stop (Data delay) |
When sharing sessions across multiple apps (e.g., dashboard.example.com and learn.example.com), you must use a centralized Redis store so a user remains logged in as they move between subdomains.
For most SaaS or Educational platforms, 2 to 4 hours is the “sweet spot.”
To ensure GCP node upgrades don’t kill all your Redis replicas at once, use Pod Anti-Affinity. This forces Kubernetes to place your Redis pods on different physical nodes.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
# Partial StatefulSet Spec
spec:
template:
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- redis-session
topologyKey: "kubernetes.io/hostname"
Don’t let a Redis connection error trigger a 500 page. Use error handlers to “fail soft.”
config/environments/production.rb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# 1. Resilient Cache
config.cache_store = :redis_cache_store, {
url: ENV['REDIS_CACHE_URL'],
connect_timeout: 1,
read_timeout: 0.2,
error_handler: -> (method:, returning:, exception:) {
Rails.logger.error "Redis Cache Down: #{exception.message}"
returning # Returns nil, forcing a DB fetch instead of a crash
}
}
# 2. Shared Session Store
Rails.application.config.session_store :redis_store,
servers: [ENV['REDIS_SESSION_URL']],
key: '_shared_org_session',
domain: :all, # Allows subdomains to share the cookie
expire_after: 4.hours
If the Sidekiq Redis is down, we want to avoid crashing the web request when enqueuing a job.
app/jobs/application_job.rb
1
2
3
4
5
6
7
8
9
10
class ApplicationJob < ActiveJob::Base
# Detect if Redis is alive; if not, run the job immediately (Inline)
self.queue_adapter = begin
Sidekiq.redis(&:ping)
:sidekiq
rescue StandardError => e
Rails.logger.warn "Sidekiq Redis Unreachable: Falling back to :inline. #{e.message}"
:inline
end
end
As your app grows, you need to know if your Redis cost is justified. Use this “Internal Audit” script to see exactly where your memory is going.
redis_audit.sh
1
2
3
4
5
6
7
#!/bin/bash
# Find the Top 5 memory-hogging keys in the current DB
echo "Scanning for top memory consumers..."
redis-cli --scan | xargs -I {} redis-cli MEMORY USAGE {} | paste - - | sort -nr -k 2 | head -n 5 | awk '{printf " %s bytes\t%s\n", $2, $1}'
# Summarize by data type
redis-cli --bigkeys | grep -E "summarized|payload"
By following this plan, you transform your infrastructure from a fragile house of cards into a resilient mesh:
This is the standard for high-performance, professional web services in 2026.
]]>