Using LM Studio and Gemma as a Local Engine for Coding Agents

Using LM Studio and Gemma as a Local Engine for Coding Agents

I did not start this experiment because I wanted a nicer chatbot on my laptop.

What I wanted was much more specific: a local model endpoint I could plug into agent-style workflows for code review, repo questions, bounded refactors, and private documentation-heavy tasks. Something that felt close enough to the OpenAI-compatible APIs many tools already expect, but without sending every prompt, diff, and internal doc set to the cloud.

That turned out to be possible. It was also a lot less smooth than the demo videos make it look.

The core setup that worked best for me was LM Studio plus a GGUF build of Gemma 4 26B. Once I got the settings under control, it became a usable local engine for agent clients. Not perfect. Not something I would trust blindly. But good enough that I would actually use it for real work.

This post is the version I wish I had found before I started.

Why I wanted this locally

There were a few reasons I kept pushing on local setup instead of giving up and using cloud models for everything.

First, privacy. If I am experimenting with agent workflows against private repositories, internal notes, or ugly half-finished local docs, I want the option to keep all of that on my machine.

Second, cost. Agent loops are noisy. They read files, retry, summarize, call tools, and sometimes spiral. That is manageable when you are testing locally. It gets annoying fast when every bad prompt is burning money.

Third, iteration speed. I wanted to try weird things: change prompts, swap harnesses, feed in local notes, break the loop, fix the loop, try again. Local models are slower in raw quality terms, but they are very forgiving for this kind of experimentation.

And fourth, I wanted to understand the boundary between “local model” and “local agent.” Those are not the same thing, and a lot of confusion in this area comes from treating them as if they are.

The stack I settled on

After trying a few variations, this is the shape that felt the most practical:

• LM Studio as the local server
• Gemma 4 26B in GGUF format
• LM Studio’s OpenAI-compatible local endpoint
• An agent client on top, such as OpenCode, OpenHands, or a custom harness

That last bullet matters more than people think.

LM Studio gives you model hosting, local inference, and a familiar API surface. That is useful. But it is not the agent. It is the model backend.

The agent still has to do the hard part:

• manage message history
• decide when to call tools
• execute those tools
• feed tool results back into the model
• handle long outputs
• sanitize weird model output
• know when the task is actually done

If that loop is weak, a good local model still feels unreliable. If the loop is solid, even an imperfect local model becomes surprisingly usable.

If you want the setup guide, read the runbook

The setup details ended up being long enough that I split them into a separate post:

LM Studio local agent runbook: Pi and OpenCode step by step

That runbook covers:

• starting the LM Studio server
• checking the local endpoint
• wiring the model into ~/.pi/agents/models.json
• wiring the same endpoint into OpenCode
• doing a cheap verification pass before larger agent tasks
• the runtime and cache-related failures I would check first

Why I ended up preferring GGUF here

For this specific setup, GGUF was the path of least resistance.

I am not making a universal claim that GGUF is always better than every other format. I am saying that in my own tests, it was the easiest way to get Gemma into a stable LM Studio workflow for agent-style tasks.

The key word there is stable.

I care less about benchmark bragging rights than about whether the model can survive long prompts, diffs, and repeated turns without drifting into nonsense or getting stuck in some half-broken internal state. GGUF plus LM Studio gave me a workflow I could keep using. That was enough.

What broke first

The first wave of problems had very little to do with “intelligence” and a lot to do with runtime behavior.

1. Long prompts hanging at 0%

This was the most annoying one.

I would send a large diff or a code-heavy prompt and LM Studio would sit there looking busy while doing nothing useful. In practice, the model was not making forward progress. It just looked stuck.

The fix that helped most was turning off Unified KV Cache and setting the context length manually instead of leaving it on auto.

That was the first clue that local agent work is often about runtime stability, not just model selection.

2. Reasoning tag leakage

Gemma can leak internal reasoning markers or thought-channel fragments into the visible output. If you are just chatting in a UI, that is ugly. If you are piping the response into an agent workflow, it is worse than ugly. It can break downstream parsing or contaminate structured output.

I saw this most often when the prompt mixed code review instructions with tool-like formatting expectations.

Adding stop tokens helped. Tightening the system prompt helped. But the deeper lesson was that local agent clients need output sanitation anyway. You should not assume the model will always hand you clean final text.

3. Context pressure shows up fast

A local coding model can look fine on a short task and then fall apart the moment you give it a real repository diff, a stack trace, and a few tool results in the same conversation.

This is where local setups stop feeling like “cheap cloud replacements” and start feeling like engineering systems with actual constraints. Context is not just a number on the model card. It is a budget, and agent loops spend that budget aggressively.

4. Good answers are easier than reliable behavior

This one took me a while to admit.

On a single prompt, Gemma could often produce a respectable answer about code. That made me optimistic. But agent workloads are harsher than single prompts. They require consistency across turns, clean tool-use behavior, and enough formatting discipline that the harness can keep going.

A model that gives one good answer is not automatically a good agent backend.

Troubleshooting notes from my own setup

These are the problems I actually ran into while trying to make this usable for agents.

One thing I wish more local-LLM writeups did: include the ugly error text.

When you are debugging this stack, vague advice is not that helpful. Concrete failure signatures are.

Qwen 35B on MLX ran out of memory

This was one of the first practical limits I hit.

On paper, a larger local model always sounds attractive. In practice, once I pushed into Qwen 35B-class MLX setups, memory pressure became the story. The model might load partway, fail during inference, or behave badly once context and KV usage started climbing.

That pushed me toward setups that were slightly smaller but much easier to keep alive for repeated agent turns.

This is one reason I think local agent work should be evaluated as a system, not just a model preference. A model that looks stronger in theory is not useful if it keeps falling over in the workflow you actually want.

KV cache support on LM Studio + MLX was not fully there for me

This was another source of confusion.

I expected KV cache behavior to be a straightforward performance win. In this setup, it was not. My experience was that KV caching on the MLX path inside LM Studio was not something I could treat as fully reliable for agent-style workloads.

That matters because agent sessions are exactly the kind of workload where you want caching to help. Long turns, repeated context, retries, and follow-up prompts should benefit from it. But if that layer is unstable, it stops being an optimization and starts becoming a source of weird failures.

One example I hit looked like this:

Error: Error in iterating prediction stream:
NotImplementedError: RotatingKVCache Quantization NYI

If you run into that kind of error, I would treat it as a runtime capability gap first, not something you are supposed to fix with prompting.

In plain English: the stack is telling you that the KV/cache path you are trying to use is not fully implemented for that configuration yet.

Shared KV support for Gemma was not there in LM Studio runtime

This deserves to be called out separately because it wasted a lot of time for me.

Gemma plus shared KV sounded like exactly the kind of optimization that should help agent loops. In practice, in LM Studio runtime, I had to treat shared KV as not supported for the setup I was using.

That matters because if you assume shared KV is working, you can end up chasing fake explanations for behavior that is really coming from unsupported runtime features.

My rule here became simple: if Gemma is behaving strangely and shared KV is part of the configuration, remove that variable first.

0% prompt stalls were tied to bad KV cache state

This was the most visible symptom.

When I saw LM Studio sit at 0% on a prompt that should have started normally, the issue often traced back to invalid or broken KV caching state. Turning off Unified KV Cache and resetting the session usually helped more than trying to tweak the prompt itself.

That is why I would treat 0% prompt stalls as a runtime problem first, not a prompting problem first.

If I hit this again, my default response would be:

1. stop the current run
2. clear or avoid the problematic cache path
3. disable Unified KV Cache
4. retry with a smaller prompt to confirm the model is healthy again

That sequence saved time because it avoided debugging the wrong layer.

Another important lesson here: the prompt is often innocent.

If the runtime state is broken, rewriting the prompt five times is just a way to waste an afternoon.

Gemma think-tag leakage is real

This one showed up often enough that I think it belongs in the troubleshooting section, not just the general setup notes.

Gemma can leak internal reasoning or think-tag markers into visible output. Depending on the client, this may show up as raw reasoning blocks, partial channel markers, or output that looks like it forgot to separate internal thought from the final answer.

That is annoying in a chat window. In an agent loop, it is worse because it can:

• break parsing
• contaminate structured output
• confuse downstream tool-call handling
• waste context on junk you never wanted to keep

What helped me:

• add stop tokens for the leaked markers
• keep the output format expectations simple
• avoid mixing too many formatting rules into one prompt
• sanitize model output in the harness instead of assuming the model will cleanly separate reasoning every time

I would treat this as a normal engineering concern when using Gemma for agents, not as a weird one-off edge case.

Simplifying Pi prompts matters more than I expected

Another lesson from the agent side: the prompt budget disappears fast.

If Pi is carrying a huge system prompt, a long skill catalog, MCP server descriptions, tool instructions, and model-specific behavior rules, you can burn a surprising amount of context before the real task even starts.

For local models, that overhead hurts more.

The practical fix was to simplify the Pi prompt aggressively:

• trim skills that are not needed for the current workflow
• avoid loading MCP/tool descriptions that the task will not use
• keep the system prompt direct and task-specific
• prefer short operational rules over a giant “do everything” instruction block

The goal is not to make the agent dumb. The goal is to stop spending half the context window on scaffolding.

For local coding models in particular, I think this matters a lot. Smaller prompt overhead usually means:

• less wasted context
• fewer formatting mistakes
• less chance of the model getting distracted by meta-instructions
• more room for the actual code, diff, logs, and tool results

If the model is struggling, one of the highest-leverage fixes is often not “find a smarter model.” It is “stop stuffing the prompt with framework overhead the model does not need.”

A short troubleshooting checklist I would use again

If I were setting this up from scratch and it started failing, this is the order I would check things:

1. Confirm LM Studio is actually serving the model you think it is serving.
2. Check whether the exact model ID in the client matches the runtime.
3. Remove shared KV assumptions from the setup.
4. Disable Unified KV Cache.
5. Treat MLX KV/cache errors as runtime limitations first.
6. Retry with a much smaller prompt.
7. Strip down the Pi or agent system prompt so the model gets more room for the real task.
8. If Gemma output is leaking think tags, fix that in both prompt design and output sanitation.

That sequence is not elegant, but it is the one I trust more now.

The settings that made it usable

These were the settings and habits that moved the setup from “interesting demo” to “something I would actually plug into a tool.”

Disable Unified KV Cache

This was the biggest quality-of-life improvement in my setup for long prompts and review-style workloads.

If you are seeing prompt hangs, weird stalls, or long code-heavy requests that never quite start, this is the first thing I would change.

Set context length manually

I had better results when I chose an explicit context size instead of trusting auto mode.

That gave me more predictable behavior, especially when I was switching between smaller code questions and much larger review tasks.

Add stop tokens for leaked thought markers

If your model output is spraying thought tags into normal text, stop tokens are worth trying immediately.

This is not a complete fix. It is more like putting guardrails around a messy edge. Still worth doing.

Use conservative expectations for long diffs

This is less a setting and more a survival rule: do not assume the local model should ingest your entire diff, your entire style guide, and a huge pile of tool output in one go.

Chunk work where you can. Summarize aggressively. Keep the agent loop disciplined.

Treat sampling as workload-specific

I would not hard-sell one magical sampling profile for every coding task. In practice, I found that code review, repo Q&A, and action-oriented tool use want slightly different behavior.

For me, the more important point was not the exact number. It was avoiding the temptation to keep turning the model into a “creative” assistant when what I actually needed was predictable structured behavior.

Where agents come in

This is the part that is usually missing from local LLM articles.

Running Gemma in LM Studio does not give you a coding agent. It gives you a model server.

To get agent behavior, something above that server has to implement a loop like this:

1. Send the current task and available tools to the model.
2. Parse whether the model wants to answer directly or call a tool.
3. Execute the tool if requested.
4. Feed the result back into the conversation.
5. Repeat until the model produces a final answer or hits a stopping rule.

That sounds obvious when written out. It is also where a lot of local setups quietly fail.

If the harness does not execute the tool correctly, the model looks dumb.

If the harness appends tool results in a format the model does not handle well, the model looks confused.

If the harness keeps dumping huge command output back into context, the model looks forgetful.

The local endpoint is only one piece of the system.

Where this fits with OpenHands, OpenCode, and custom harnesses

What I like about LM Studio in this role is that it can behave like a local OpenAI-compatible backend. That makes experimentation easier because a lot of agent tooling already assumes that style of interface.

So the setup I kept coming back to was:

• LM Studio hosts the model locally
• the agent client points at http://localhost:1234/v1
• the agent loop handles tools, memory pressure, and retries

That works well for:

• OpenCode-style local coding sessions
• OpenHands-style experiments where you want a local model backend
• custom agents that already know how to talk to OpenAI-compatible endpoints

What it does not solve is model capability mismatch. If your agent framework expects extremely clean function calling, long-context reliability, or excellent multi-step planning under pressure, local Gemma may still feel rough compared to stronger cloud models.

That is not a failure of LM Studio. It is just the reality of the stack.

What this setup is actually good for

Once I stopped expecting a local model to be a universal drop-in replacement for the best cloud systems, the setup got much more useful.

Here is where I think it makes sense.

PR review and diff reading

This is one of the better use cases because the task is bounded and the output format is easy to evaluate.

The model can read a diff, point out obvious risk, summarize a change, and flag suspicious sections. You still need judgment. You still need tests. But it is useful.

Repo Q&A

If you want to ask questions about a codebase, architecture notes, or internal docs that you do not want to upload elsewhere, local is attractive.

This gets even better when the agent harness is disciplined about retrieval and does not just dump huge blobs of text into the prompt.

Bounded refactors

I would trust this more for “rename this pattern in these files” than for “redesign the authentication system.”

That distinction is important. Local models can be very handy for narrow, repetitive engineering work. They are much less trustworthy when the task becomes open-ended or architectural.

Private knowledge-heavy tasks

If the task depends on local notes, internal guides, or a private personal knowledge base, a local endpoint starts to feel compelling even when the model is not state of the art.

Sometimes privacy and convenience matter more than squeezing out the last 10% of model quality.

Where it still falls short

This setup is useful, but I would not oversell it.

It is still easier to get bad agent behavior than good agent behavior

Single-turn demos flatter local models. Multi-step agent loops expose every weakness.

You notice formatting issues, small reasoning slips, and context handling problems much more when the model has to survive several turns in a row.

Long-context work remains fragile

Even when the model technically accepts a large context, that does not mean it uses it well.

If your workflow depends on feeding in very large diffs, long logs, and many tool results without careful pruning, you will probably have a bad time.

Tool use can be brittle

Some failures are obvious. The model outputs malformed JSON. Or it leaks thought tags into a tool call. Or it starts narrating instead of acting.

Some failures are subtler. The tool call shape is technically valid but not useful. The model calls the wrong tool. The model forgets why it called the tool in the first place.

Again, this is why the harness matters so much.

It is not the best choice for every job

If I need maximum reliability on a complex, high-context, multi-step coding task, I would still reach for a stronger cloud model first.

The local setup wins when privacy, cost control, experimentation, or offline access matter enough to justify the trade.

My practical recommendation

If your goal is to build a local engine for agent usage, I think LM Studio plus Gemma is a reasonable place to start.

Not because it is perfect. Because it is accessible.

You get a local server, an OpenAI-compatible endpoint, and a model that is capable enough to make the experiment real. That is a good combination for people who want to learn how local agents actually work instead of just reading about them.

I would recommend it to:

• developers experimenting with local coding agents
• people who want to review private code or docs locally
• anyone building a custom harness and needing a local backend to test against

I would not recommend it as your only serious option if:

• you need highly reliable long-context reasoning
• you need polished tool calling with very little cleanup
• you are trying to match the strongest hosted coding models head-on

My own conclusion is pretty simple.

LM Studio plus Gemma can absolutely work as a local engine for agent workflows. It is good enough for real experiments and some real tasks. But the useful mental model is not “I installed a local genius on my laptop.”

It is “I built a constrained local backend, then did the engineering work required to make an agent around it behave.”

That framing is less glamorous. It is also much closer to the truth.