For a small education or SaaS team, these symptoms create a difficult choice. Keep adding features and accept the growing cost, or pause delivery and attempt a large modernization project with uncertain scope.
In many cases, the best first step is neither. It is a bounded investigation that separates the real constraints from the stories the team has started telling itself about the system.
“Legacy Rails application” is not yet a diagnosis. It describes an age or technology choice, but not the business problem.
A useful investigation starts with questions such as:
This changes the conversation from “Should we rewrite?” to “What evidence would justify the next change?”
Look at how a feature moves from idea to production.
Warning signs include:
A modernization plan should improve the team’s ability to make safe changes, not only replace old code with new code.
Trace one slow or important user workflow through the system:
1
2
3
4
5
6
7
8
9
browser or mobile client
↓
controller / API endpoint
↓
application services and authorization
↓
database queries and external services
↓
background work, notifications, or reporting
The goal is not to collect every possible metric. It is to find where the workflow spends time, where failures disappear, and where ownership becomes unclear.
Reports, imports, notifications, and large exports often expose platform weaknesses first. A synchronous request may appear simple, but it creates a poor failure boundary when the work is large or unpredictable.
A healthier pattern is usually:
The exact implementation depends on the platform, but the architectural question is general: does the user-facing request have to remain open while the system performs work that belongs in an asynchronous workflow?
When a system is slow or unreliable, the team needs enough evidence to make the next decision.
Useful evidence may include:
A dashboard is not automatically observability. Evidence is useful when it helps somebody choose an action.
Learning platforms often combine course content, assessment, reporting, users, integrations, billing, and administration. Over time, these concerns can become tightly coupled.
That does not mean every concern should immediately become a separate service. It does mean the team should identify which boundaries are causing real delivery or operational pain.
A good first boundary may be a reporting workflow, an integration, or a clearly owned domain module—not a wholesale migration to microservices.
A useful modernization plan answers four questions:
If a proposal cannot answer the fourth question, it is not yet a reliable plan.
The answer may be a code change, a database improvement, better job instrumentation, a deployment safety measure, a clearer ownership boundary, or a decision not to change something yet.
For a small team, a first review can be deliberately narrow:
This is more useful than producing a generic architecture diagram disconnected from the team’s actual constraints.
I would be cautious about:
The goal is not to defend legacy software. The goal is to make the next decision safer and more valuable.
If a Rails or LMS platform is becoming slow, fragile, or difficult to extend, start by writing down one workflow that is causing measurable pain. Record who is affected, how often it happens, what the current workaround is, and what evidence you already have.
That short exercise often reveals whether the next step should be a focused technical investigation, a product decision, an operational improvement, or no major change at all.
I am developing a fixed-scope Rails/LMS Reliability and Modernization Audit for teams that need this kind of evidence-led first review. The aim is to understand the system and produce a practical roadmap—not to sell a rewrite before the problem is understood.
]]>A Rails test suite can work perfectly on a MacBook and fail before the first example runs on an Ubuntu build machine. That sounds like an application bug, but sometimes the application has not even started yet. The failure is in a native dependency loaded during Rails boot.
I recently hit this with a Rails application using Active Storage and ruby-vips. The error was essentially:
1
2
3
4
libvips's unfuzzed operations are not safe to use with untrusted content,
and Active Storage cannot disable them.
Disabling them requires libvips 8.13 or later and ruby-vips 2.2.1 or later.
The Ruby dependency was already new enough. The native library on the Ubuntu host was not.
The important lesson is this:
ruby-vipsis a Ruby wrapper, not the image-processing engine itself. The wrapper still loads the nativelibvipsshared library installed by the operating system or by a separate native build.
On the Ubuntu ARM build host used for this investigation, the versions were:
1
2
3
Ubuntu: 22.04.5 LTS, arm64
libvips: 8.12.1
ruby-vips: 2.3.0
ruby-vips satisfied the Ruby-side requirement, but Ubuntu’s standard Jammy repository provided libvips 8.12.1. Active Storage requires libvips 8.13 or later for the security behavior it needs, so Rails aborted during initialization.
After installing libvips 8.18.3 under a separate system prefix, the same check reported:
1
2
ruby-vips 2.3.0
libvips 8.18.3
The focused spec then passed:
1
3 examples, 0 failures
This is not because Ubuntu cannot run libvips, and it is not because macOS is inherently more compatible. The difference is usually the package source and release cadence.
Ubuntu LTS releases intentionally keep major system packages stable for the lifetime of the release. On Ubuntu 22.04, the standard ARM repository can therefore provide libvips 8.12.1 even though newer libvips releases exist upstream.
That is a useful server property: security updates and compatibility fixes are preferred over continuously changing feature versions. It also means that a build requiring a newer native library may outgrow the distribution package.
Check the actual package before assuming anything:
1
2
3
4
cat /etc/os-release
uname -m
apt-cache policy libvips42 libvips-dev
vips --version
On Debian-family systems, the runtime package is commonly named libvips42 and the headers/build metadata are in libvips-dev.
On macOS, libvips is commonly installed through Homebrew:
1
2
brew install vips
vips --version
Homebrew is a rolling package ecosystem rather than an Ubuntu-LTS snapshot. A MacBook that was set up or updated recently may therefore have a libvips version new enough for the Rails dependency, while an Ubuntu build machine still has the older LTS package.
The right comparison is not “Mac versus Linux.” Compare these instead:
1
2
3
4
5
6
Ruby version
ruby-vips gem version
native libvips version
architecture
package source
library search path
A MacBook may also be using a Homebrew-provided ARM64 or x86_64 library, while the Ubuntu builder is using an ARM64 distribution package. Both can be valid installations with different versions.
There are three separate pieces in this failure:
1
2
3
4
5
6
7
8
9
10
Rails / Active Storage
│
▼
ruby-vips Ruby gem
│ FFI / native loading
▼
libvips shared library
│
▼
Ubuntu package or source installation
Active Storage uses image-processing backends and checks whether the native library has the security controls it needs. This check can happen while the Rails application boots, which is why the test runner can fail before reporting any examples.
ruby-vips gemThe gem provides Ruby classes and bindings. Check it through Bundler, not only through the system Ruby:
1
2
bundle info ruby-vips
bundle exec ruby -e 'require "vips"; puts Vips::VERSION'
The project may use a different Ruby from /usr/bin/ruby, especially when it uses mise, rbenv, or asdf.
The wrapper loads the shared library found through the dynamic linker. Check the version that the Ruby process actually sees:
1
bundle exec ruby -e 'require "vips"; puts Vips::LIBRARY_VERSION'
This is more useful than checking only vips --version. The command-line binary and Ruby process can resolve different libraries if /usr/local, Homebrew, or a custom LD_LIBRARY_PATH is involved.
Before compiling anything, check whether the configured Ubuntu repositories already provide a sufficient version:
1
2
sudo apt update
apt-cache policy libvips42 libvips-dev
If the candidate is 8.13 or newer, install both the runtime and development packages:
1
sudo apt install libvips42 libvips-dev
Then verify both the CLI and the Ruby binding:
1
2
vips --version
bundle exec ruby -e 'require "vips"; puts Vips::LIBRARY_VERSION'
Installing only libvips-dev is not enough. The build headers may be present while the runtime still loads the old shared library. Conversely, installing only the runtime package may leave native extensions or build checks without the headers and pkg-config metadata they need.
When Ubuntu’s package is too old, building a newer upstream release is a practical option for a dedicated build host. The official libvips installation guide recommends Meson and Ninja for source builds.
The following example installs libvips 8.18.3 into /usr/local. Choose a release that is appropriate for the project and verify its checksum or signature according to your environment’s policy.
The exact optional dependencies determine which image formats libvips supports. Start with the core build tools:
1
2
3
4
5
6
7
8
9
10
11
12
13
sudo apt update
sudo apt install \
build-essential \
meson \
ninja-build \
pkg-config \
libglib2.0-dev \
libexpat1-dev \
libjpeg-turbo8-dev \
libtiff5-dev \
libpng-dev \
libwebp-dev \
libarchive-dev
You can add development packages for formats your application needs. Inspect the Meson summary carefully; a successful compilation does not necessarily mean every optional image loader was enabled.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
mkdir -p "$HOME/tmp/libvips-build"
cd "$HOME/tmp/libvips-build"
curl -fL --retry 3 \
-o libvips-8.18.3.tar.xz \
https://github.com/libvips/libvips/releases/download/v8.18.3/libvips-8.18.3.tar.xz
tar -xf libvips-8.18.3.tar.xz
cd libvips-8.18.3
meson setup build \
--prefix=/usr/local \
--libdir=lib \
--buildtype=release
The --libdir=lib option is useful on Debian-family systems because it avoids putting the library under a multiarch path such as lib/aarch64-linux-gnu. Either layout can work; consistency matters more than the exact choice.
Read the Meson summary before continuing:
1
meson configure build
1
2
3
4
meson compile -C build
meson test -C build
sudo meson install -C build
sudo ldconfig
ldconfig refreshes the dynamic linker cache so processes can find the newly installed shared library. Without it, the vips executable or Ruby process may continue loading the distribution copy.
Verify the result:
1
2
3
4
5
/usr/local/bin/vips --version
vips --version
bundle exec ruby -e \
'require "vips"; puts "ruby-vips #{Vips::VERSION}"; puts "libvips #{Vips::LIBRARY_VERSION}"'
The important output is the Vips::LIBRARY_VERSION value observed by the same Bundler environment that runs the Rails tests.
LD_LIBRARY_PATH is neededA custom installation may be in a library directory that is not yet in the linker configuration. For a temporary test, set:
1
export LD_LIBRARY_PATH=/usr/local/lib:/usr/local/lib/aarch64-linux-gnu
Then rerun the Ruby check and the spec:
1
2
3
4
DATABASE_HOST=/path/to/local/postgres/socket \
CI=true \
LD_LIBRARY_PATH=/usr/local/lib:/usr/local/lib/$(uname -m)-linux-gnu \
bundle exec rspec spec/tasks/remove_copy_suffixes_spec.rb
For a permanent machine configuration, prefer a linker configuration file instead of putting LD_LIBRARY_PATH into every CI command:
1
2
3
4
printf '%s\n' /usr/local/lib > /tmp/usr-local-lib.conf
sudo mv /tmp/usr-local-lib.conf /etc/ld.so.conf.d/usr-local-lib.conf
sudo ldconfig
ldconfig -p | grep libvips
Use the actual directory where libvips.so was installed. Do not add a path blindly; confirm it first:
1
find /usr/local -name 'libvips.so*' -print
If the version still looks wrong, inspect the executable and the Ruby process rather than guessing.
For the CLI:
1
2
command -v vips
ldd "$(command -v vips)" | grep -E 'vips|glib|expat'
For Ruby, first locate the FFI-loaded library and then inspect the process while it is running. A simpler first check is usually enough:
1
bundle exec ruby -e 'require "vips"; abort "old libvips" unless Vips::LIBRARY_VERSION >= "8.13"; puts Vips::LIBRARY_VERSION'
Also check for multiple installations:
1
find /usr /usr/local -name 'libvips.so*' -o -name 'vips' 2>/dev/null
Common causes of a mismatch include:
/usr/bin/vips is from apt but Ruby loads /usr/local/lib.ldconfig was not run.LD_LIBRARY_PATH points to an older build.Once the native version is correct, run the smallest failing spec first:
1
2
3
4
DATABASE_HOST=/path/to/local/postgres/socket \
CI=true \
LD_LIBRARY_PATH=/usr/local/lib:/usr/local/lib/$(uname -m)-linux-gnu \
bundle exec rspec spec/tasks/remove_copy_suffixes_spec.rb
Then run the repository’s normal test command. Keep the environment explicit so a local interactive shell does not hide a missing CI dependency.
A successful native check does not prove the whole application is correct. It only proves that the process can load a compatible libvips. The Rails test suite still needs its database, browser, JavaScript, and application-level dependencies.
No. There are several reasonable strategies:
Best when the organization has a trusted repository or image-building process that supplies the required version. It keeps upgrades integrated with package management, but introduces repository and support decisions.
Best when CI is containerized or image-based. The native version becomes part of a reproducible builder image instead of an undocumented manual server change.
Best for a single dedicated host where you want to avoid replacing Ubuntu-managed files. /usr/local is separate from /usr, and the install can be removed or replaced deliberately. Document the linker path and the exact source release.
Best when the build host is already due for an OS refresh. A newer Ubuntu release may provide a newer libvips package, but an OS upgrade is a much larger change than a single library upgrade.
The least desirable option is to overwrite files owned by apt under /usr/lib manually. That makes future package upgrades and incident recovery harder.
When the MacBook passes and Ubuntu fails, record this information on both machines:
1
2
3
4
5
6
uname -m
ruby --version
bundle --version
bundle info ruby-vips
bundle exec ruby -e 'require "vips"; puts Vips::VERSION; puts Vips::LIBRARY_VERSION'
vips --version
Then compare:
ruby-vips gem version.libvips version loaded by Ruby.This turns “it only fails on Ubuntu” into a concrete dependency difference.
Native dependencies live below the Ruby dependency file. Bundler can resolve ruby-vips successfully while the operating system still provides an incompatible libvips. macOS and Ubuntu can therefore run the same Rails code with different native behavior because their package ecosystems move at different speeds.
The durable fix is not to disable the safety check. It is to make the build environment explicit: use a compatible native libvips, verify the version from inside the Bundler process, install it through a repeatable package or image workflow, and document the linker configuration.
Once those layers are checked separately, the failure is straightforward: the application was not broken; the Ubuntu builder was loading an older native library than the Rails dependency allowed.
I wanted a local model that could handle routine Hermes work on a small ARM server: 12 GB of memory and two virtual CPUs. The target was not a leaderboard score. I wanted to know whether the model could follow short instructions, return JSON, write a small function, fix a bug, and produce a short plan without exhausting the machine.
The same question comes up whenever hardware changes. A model that feels good on a 16 GB Apple Silicon laptop may behave very differently on a two-core Linux VM. Rather than asking whether a model needs a fixed amount of memory, I wanted a measurement process I could repeat on the next machine.
The server used for this test had:
1
2
3
4
5
6
Architecture: aarch64
CPU: 2 x Neoverse-N1 OCPU
Memory: 11 GiB usable
Swap: 8 GiB
Runtime: llama.cpp / llama-server
Quantization: Q4_K_M GGUF
The model server exposed an OpenAI-compatible endpoint on loopback:
1
http://127.0.0.1:18080/v1
Binding to loopback matters. This was a local experiment, not an internet-facing inference service. An endpoint that accepts prompts and may return private code should not be exposed unnecessarily.
The llama.cpp command was deliberately simple:
1
2
3
4
5
6
7
8
9
10
11
12
llama-server \
-m /path/to/model.gguf \
--alias local-model \
--host 127.0.0.1 \
--port 18080 \
-c 32768 \
-t 2 \
-np 1 \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--metrics \
--no-webui
The exact flags are not universal. The important parts are the model path, loopback binding, one request slot, a context size that fits the machine, and metrics that let the test record speed rather than relying on impressions.
I use the Hugging Face CLI to download one named file into a predictable directory:
1
2
3
4
5
6
mkdir -p "$HOME/local-llm/models/Qwen3-4B-Instruct-2507-GGUF"
hf download \
bartowski/Qwen_Qwen3-4B-Instruct-2507-GGUF \
Qwen_Qwen3-4B-Instruct-2507-Q4_K_M.gguf \
--local-dir "$HOME/local-llm/models/Qwen3-4B-Instruct-2507-GGUF"
A repeatable directory layout makes the server command portable:
1
2
3
4
5
6
7
local-llm/
├── models/
│ └── <model-name>/
│ └── <quantization>.gguf
├── src/llama.cpp/
├── start-<model-name>.sh
└── (optional) hermes-local wrapper
Download only the quantization you intend to test. A model repository often contains many variants, and downloading a whole repository makes it harder to know which file was actually measured.
Before starting anything, record the hardware and check for an existing server:
1
2
3
4
5
uname -m
nproc
lscpu | grep -E 'Architecture|Model name|CPU\\(s\\)' | head
free -h
ps -eo pid,etime,%cpu,%mem,rss,cmd | grep -E 'llama-server|ollama|mlx' | grep -v grep || true
Then start the server and wait for its health endpoint:
1
2
3
4
5
./start-qwen3.sh &
until curl -fsS http://127.0.0.1:18080/health | grep -q '"status":"ok"'; do
sleep 2
done
Do not treat a process that exists as a ready model. llama-server can spend a while loading weights. During that period, /health returns a 503 loading response. The benchmark should start only after the endpoint reports healthy.
After loading, check memory again:
1
2
ps -o pid,etime,%cpu,%mem,rss,vsz,cmd -C llama-server
free -h
The file size is not the memory footprint. The runtime also needs model metadata, working buffers, the KV cache, and the context window. On this machine, a 2.5 GB Q4 model used roughly 7.5 GB RSS with a 32K context.
A useful smoke benchmark should exercise the tasks the local model is expected to do. I used seven small cases:
The test sends each case through the OpenAI-compatible chat completions endpoint with temperature zero and checks the result programmatically:
1
2
3
4
5
6
payload = {
"model": "local-model",
"messages": [{"role": "user", "content": prompt}],
"temperature": 0,
"max_tokens": max_tokens,
}
The checks should be explicit. For example, the bug-fixing case should verify that the response contains a guard for an empty list and returns zero. A human can see that two answers are equivalent, but a script makes the result reproducible and exposes formatting differences.
The benchmark also records:
1
2
3
4
5
6
wall time
prompt tokens and prompt tokens/sec
generated tokens and generation tokens/sec
model output
finish reason
errors from the checker
The raw output belongs beside the script. A summary without the individual responses is not enough to diagnose a failure.
One of the first results looked like a very poor E2B score. Several answers were empty even though the model was clearly working. Inspecting the raw response showed that the model had filled the completion budget with reasoning_content and never reached a visible answer.
That is a serving and evaluation issue, not a clean measure of model capability. Small reasoning models need separate handling for:
A test that only reads message.content can report a false failure when the useful answer is in another response field. Conversely, a model that regularly consumes its entire budget thinking is still a poor Hermes fallback if the agent runtime cannot handle that behavior reliably.
For a fair comparison, either disable extended thinking for short operational tasks or allocate enough tokens for both reasoning and the final answer. Then run the same test again.
The first comparison included Gemma 4 E2B and E4B, followed by three smaller general or coding models. These were observed results from this machine, not claims about every runtime or quantization.
| Model | GGUF file | Raw pass rate | Bug fix | Generation speed | Peak RSS |
|---|---|---|---|---|---|
| Gemma 4 E2B | 3.11 GB | 1/7 | Failed in this setup | 8.7 tok/s | 4.7 GB |
| Gemma 4 E4B | 4.98 GB | 4/7 | Passed | 4.7 tok/s | 7.9 GB |
| Qwen2.5-Coder 3B | 1.93 GB | 3/7 | Passed | 8.6 tok/s | about 7.2 GB |
| Phi-4-mini | 2.49 GB | 3/7 | Semantically passed, formatting failed | 5.8 tok/s | about 7.2 GB |
| Qwen3-4B-Instruct-2507 | 2.50 GB | 5/7 | Passed | 5.9 tok/s | about 7.5 GB |
The score is only one signal. Qwen2.5-Coder produced correct code but wrapped it in Markdown fences, so a strict code-only checker marked it wrong. Phi-4 calculated the right arithmetic result but was cut off before completing the response. Qwen3 returned valid JSON, passed the bug-fix and planning cases, and was the best general candidate in this run.
The same E2B family model performs differently on a 16 GB MacBook using the MLX runtime. There, E2B reached 6/7, missing only the planning case. That comparison is useful, but it does not contradict the Linux result. The machines differ in CPU, memory bandwidth, runtime, quantization format, prompt template handling, and response parsing.
The decision should use both capability and resource headroom.
An always-on model must leave room for the agent runtime, the operating system, logs, Telegram or gateway processes, and temporary command output. A model that uses nearly all available memory may work in a quiet benchmark and then trigger swapping during a real coding task.
For this 12 GB class machine, I would use these practical bands:
1
2
3
4
under 5 GB RSS: plausible always-on candidate
5–7 GB RSS: test carefully with the real agent workload
7–9 GB RSS: prefer on-demand or reduce context
above 9 GB RSS: unsafe for a shared host
These are operating guidelines, not model requirements. Context size can move a model between bands. A 16K context may be a better default than 32K when the model is handling short coding tasks.
For an on-demand workflow:
1
2
3
4
./start-qwen3.sh
# wait for /health to report status=ok
hermes chat --model local-qwen3 ...
# stop the server after the task
For an always-on fallback, add a supervised service only after measuring idle memory, request memory, startup behavior, and recovery from a failed request. Do not turn a benchmark command into a system service before those checks pass.
The reusable part is a small model manifest plus one benchmark runner. For each candidate, record:
1
2
3
4
5
6
7
8
name: qwen3-4b-instruct-2507
source: bartowski/Qwen_Qwen3-4B-Instruct-2507-GGUF
file: Qwen_Qwen3-4B-Instruct-2507-Q4_K_M.gguf
alias: local-qwen3
context: 16384
temperature: 0
max_tokens: 256
expected_use: routine coding fallback
The runner should accept the model path and alias rather than embedding one model name in the test. Run candidates sequentially, not concurrently, unless the goal is specifically to measure multi-model contention. Save one JSONL file per run with the hardware summary and llama.cpp version.
Keep the following separate:
This separation prevents a failing download, a bad chat template, and a weak model from all looking like the same problem.
Once the local server is healthy, the most useful operational improvement is a command that makes model selection explicit. Relying on the normal Hermes routing path is convenient, but it can obscure whether a task actually ran locally or went to a cloud fallback.
I added a hermes-local wrapper for tasks that must run on the persistent local worker:
1
hermes-local "Reply exactly LOCAL_OK"
The wrapper pins the request to the local-qwen3 model and checks the loopback endpoint before starting Hermes. It deliberately does not fall back to a cloud model. If Qwen3 is unavailable, the command fails clearly instead of silently changing the execution environment.
For a bounded implementation task in an isolated worktree:
1
2
3
4
hermes-local \
--in /path/to/isolated-worktree \
--max-turns 20 \
"Implement the prepared task in IMPLEMENTATION.md and run its verification command."
The command defaults to the file,terminal toolsets and no extended reasoning. Those defaults are intentional: the local model is an implementer, not the architecture owner. The cloud model should write the specification, while the local worker performs the small edit and deterministic test run.
A safe delegation contract has three parts:
IMPLEMENTATION.md.This separation makes the local model useful without pretending that a small CPU model should own ambiguous debugging, security review, architecture, or production operations.
A seven-prompt benchmark is a filter. It is not proof that a model can run a coding agent.
The next test should use a temporary repository and ask the model to:
The runner should capture the complete transcript, tool calls, command results, elapsed time, and final diff. It should fail if the model claims success without a real passing test command.
That test measures the agent loop as well as the model. A model can pass direct prompts and still fail when it needs to plan, call a tool, consume the result, and continue for several turns.
On this two-OCPU, 12 GB machine, Qwen3-4B-Instruct-2507 is the best general candidate from this small comparison. I would start it with a 16K context, keep it on-demand, and test the real Hermes implementation workflow before making it a permanent fallback.
On an Apple Silicon laptop with 16 GB unified memory, Gemma 4 E2B through MLX is a reasonable choice if the local runtime produces the stronger result observed there. The model name alone does not determine the result. The runtime and hardware are part of the deployment.
The workflow is the part worth keeping:
1
2
3
4
5
6
7
8
9
10
inspect the host
→ download one exact quantization
→ start a loopback server
→ wait for a real health signal
→ pin tasks explicitly with hermes-local
→ run fixed capability checks
→ record output, speed, and memory
→ inspect reasoning-channel behavior
→ run a real tool-loop task
→ choose always-on or on-demand
That process can be copied to a laptop, VM, ARM board, or larger server. The answer will change with the resources. The measurement method does not have to.
]]>Part of the AI & Agents reading path.
A useful coding agent is more than a language model with a terminal attached. It needs a workspace, tools, approval boundaries, a way to delegate, and a recovery plan for the cases where the first attempt stalls.
This post documents my current Hermes setup on an Apple Silicon Mac and the practical lesson from the experiment:
Local Gemma is already good at deciding which engineering workflow a request needs. Automatic per-request model switching is not yet active in this Hermes installation, so explicit local-session selection is the reliable path today.
The examples use a Rails application under a placeholder path. They are written to be safe defaults: inspect first, show the plan, approve dependency changes, then verify the result.
The setup has three model paths:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
┌──────────────────────────────────────────────────────────┐
│ Main Hermes conversation │
│ Cloud model when broad reasoning or difficult context │
└───────────────────────┬──────────────────────────────────┘
│ delegation, when requested/used
▼
┌──────────────────────────────────────────────────────────┐
│ Local Gemma through LM Studio │
│ google/gemma-4-e2b │
│ http://127.0.0.1:1234/v1 │
└───────────────────────┬──────────────────────────────────┘
│ file + terminal tools
▼
┌──────────────────────────────────────────────────────────┐
│ Local project workspace │
│ /path/to/<application> │
└──────────────────────────────────────────────────────────┘
Hermes is the runtime. LM Studio provides an OpenAI-compatible local endpoint. Gemma makes decisions and proposes actions. Hermes, not Gemma, executes tools and applies approval policy.
That separation matters. A model does not run bundle update by itself. The agent runtime executes the command, captures the output, gives it back to the model, and continues the loop.
The local endpoint and model currently used are:
1
2
3
Endpoint: http://127.0.0.1:1234/v1
Model: google/gemma-4-e2b
Runtime: LM Studio, local device
The Hermes configuration contains the local provider and alias:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
fallback_providers:
- provider: custom
model: google/gemma-4-e2b
base_url: http://127.0.0.1:1234/v1
api_mode: chat_completions
delegation:
model: google/gemma-4-e2b
provider: custom
base_url: http://127.0.0.1:1234/v1
model_aliases:
local-gemma:
provider: custom
model: google/gemma-4-e2b
base_url: http://127.0.0.1:1234/v1
Check the effective configuration rather than assuming it:
1
2
3
4
5
hermes config path
hermes config get fallback_providers
hermes config get delegation
hermes config get model_aliases
hermes config check
Check LM Studio independently:
1
2
3
lms ls
lms ps
curl -sS http://127.0.0.1:1234/v1/models
The endpoint should list google/gemma-4-e2b. If it does not, Hermes cannot use the local model even if the YAML looks correct.
The most predictable form of cost control is not automatic routing. It is selecting the local model explicitly for a session:
1
2
3
4
5
hermes chat \
--model local-gemma \
--in "/path/to/my-rails-app" \
--toolsets file,terminal \
--reasoning low
For a one-shot inspection:
1
2
3
4
5
6
hermes chat \
--model local-gemma \
--in "/path/to/my-rails-app" \
--toolsets file,terminal \
--reasoning low \
-q 'Inspect this Rails app. Identify the package managers, current dependency state, test commands, and any uncommitted changes. Do not modify files or run upgrades.'
This makes the model choice visible and auditable. It also avoids accidentally sending a routine maintenance request to an expensive cloud provider.
Use the real application path in place of /path/to/my-rails-app:
1
printf '%s\n' /path/to/projects/*
Do not assume that every directory under /path/to/projects is a Rails application. Let the agent inspect for Gemfile, Gemfile.lock, package.json, yarn.lock, config/application.rb, and the repository state.
A dependency upgrade is not a trivial “run two commands” task. It can change a lockfile, update transitive dependencies, expose incompatible Ruby versions, alter JavaScript build output, or require application changes.
Start with a read-only request:
1
2
3
4
5
6
hermes chat \
--model local-gemma \
--in "/path/to/my-rails-app" \
--toolsets file,terminal \
--reasoning low \
-q 'Inspect this Rails repository for a routine dependency maintenance task. Determine whether it uses Bundler and Yarn, report the Ruby/Rails/Node/package-manager versions, inspect git status, identify the relevant upgrade commands, and propose a verification plan. Do not modify files and do not run bundle or yarn upgrade commands.'
A good response should first discover facts. Typical inspection commands include:
1
2
3
4
5
6
7
8
9
pwd
git status --short --branch
ruby --version
bundle --version
node --version
yarn --version
bundle check
bundle outdated --only-explicit || true
yarn outdated || true
The exact commands depend on the repository. In particular, do not use Yarn if the project actually uses npm or pnpm, and do not assume that bundle update is equivalent to updating one named gem.
Once the plan is reviewed, run a separate approved session or continue the same session with a clear boundary:
1
2
3
4
5
6
hermes chat \
--model local-gemma \
--in "/path/to/my-rails-app" \
--toolsets file,terminal \
--reasoning low \
-q 'Now apply the approved dependency-maintenance plan. Update only the agreed Bundler and Yarn dependencies. Before each mutating command, show the exact command and its scope. Afterward inspect the diff and run the repository verification commands. Do not deploy, push, delete files, or change application code unless explicitly required and approved.'
The desired workflow is:
1
2
3
4
5
6
7
8
inspect
→ identify package managers and scope
→ show exact commands
→ approve
→ update dependencies
→ inspect lockfile diff
→ run tests and build checks
→ report failures and remaining risk
For a narrow update, the commands may look like these, but the agent should verify the project first:
1
2
3
bundle update rails
yarn upgrade-interactive --latest
A broad upgrade is much riskier:
1
2
bundle update
yarn upgrade
Do not use broad commands merely because they are shorter. Lockfile churn makes failures harder to diagnose and rollbacks less focused.
Keep normal Hermes approvals enabled. Avoid --yolo for dependency work:
1
2
3
4
5
# Safer default: approval prompts remain active
hermes chat --model local-gemma --in "/path/to/my-rails-app"
# Avoid this for upgrades unless you have a very specific reason:
# hermes chat --yolo ...
A useful prompt makes the boundary explicit:
1
You may inspect files and run read-only diagnostics. Do not run mutating commands until I approve the exact command. Treat bundle update, yarn upgrade, database migrations, deployment, git commit, git push, file deletion, and changes outside this repository as mutating operations.
The local model is inexpensive, but an unsafe command is still unsafe. Lower inference cost should buy more inspection and smaller changes, not fewer controls.
Delegation is different from automatic routing.
The configured delegation model can be checked with:
1
hermes config get delegation
The important fields are:
1
2
3
4
5
6
7
8
9
10
delegation:
model: google/gemma-4-e2b
provider: custom
base_url: http://127.0.0.1:1234/v1
max_iterations: 20
max_concurrent_children: 1
max_spawn_depth: 1
orchestrator_enabled: false
subagent_auto_approve: false
default_toolsets: '["file", "terminal"]'
The local Gemma workflow test showed that it can classify common engineering tasks:
1
2
3
4
5
6
7
bug fixing → bug fixing
missing test suite → test writing
README update → documentation
duplicate code cleanup → refactoring
API vulnerability scan → security review
new CSV export → feature implementation
vague cleanup request → clarification
Measured result from seven prompts:
1
2
3
Workflow routing: 7/7 correct
Response structure: 7/7 compliant
Detailed plan checks: 6/7 passed
The one detailed miss was in the security-review case: Gemma selected the correct workflow and asked for the missing API code, but did not explicitly include remediation in its plan. That is a useful limitation to know. Correct routing does not imply complete execution planning.
The raw test script and result are kept with the Hermes setup:
1
2
3
<hermes-config-dir>/scripts/test_local_gemma_workflows.py
<hermes-config-dir>/docs/LOCAL-GEMMA-WORKFLOW-TEST.md
<hermes-config-dir>/test-results/local_gemma_workflow_results.json
Run it again with:
1
2
cd <hermes-config-dir>
python3 scripts/test_local_gemma_workflows.py
The config contains a placeholder:
1
2
3
4
5
smart_model_routing:
enabled: false
max_simple_chars: 160
max_simple_words: 28
cheap_model: {}
In this installed version, the option is written by setup but there is no active runtime implementation that reliably classifies every incoming task and switches the main model. Turning the flag on would create false confidence.
Until that runtime support exists, these are the practical routing policies:
Use local-gemma for:
Use the cloud model when the task involves:
Start locally. If Gemma cannot progress, stop rather than letting it improvise. Capture the current state and restart with a cloud model using the local output as evidence:
1
2
3
4
5
hermes chat \
--model local-gemma \
--in "/path/to/my-rails-app" \
--toolsets file,terminal \
--reasoning low
Then, only if needed:
1
2
3
4
5
hermes chat \
--in "/path/to/my-rails-app" \
--toolsets file,terminal \
--reasoning medium \
-q 'Continue the dependency-maintenance investigation. Use the prior inspection notes below. Do not repeat completed commands. Identify the smallest safe next step and explain why it needs stronger reasoning. [paste notes]'
Before starting:
1
2
cd "$HOME/public_html/my-rails-app"
git status --short --branch
Ask the agent to establish:
Before mutating:
After mutating:
1
2
3
4
5
6
7
8
git diff --stat
git diff -- Gemfile Gemfile.lock package.json yarn.lock
git status --short --branch
bundle check
yarn check --integrity || true
bundle exec rails test || bundle exec rspec
yarn test || true
yarn build || true
Use the commands the repository actually defines. The final report should separate:
1
2
3
4
5
6
7
Changed:
Tests run:
Tests passed:
Tests failed:
Warnings:
Remaining risk:
Rollback:
This prompt works well for routine Rails maintenance:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
You are operating in a Rails repository. Work as a cautious maintenance engineer.
Goal: inspect and, only after approval, perform the requested dependency update.
Rules:
1. Start with read-only inspection.
2. Identify the repository root and package managers from files, not assumptions.
3. Check git status before touching anything.
4. Report Ruby, Rails, Bundler, Node, Yarn/npm/pnpm versions when available.
5. Propose the smallest command that satisfies the request.
6. Do not run bundle update, yarn upgrade, migrations, deployment, commit, push, deletion, or commands outside the repository without explicit approval.
7. Before a mutating command, show the exact command and explain its scope.
8. After changes, inspect the diff and run the project's own tests, lint, and build checks.
9. Never claim a test passed unless you actually ran it.
10. If the request is ambiguous or the repository state is unsafe, stop and ask for clarification.
Requested task: inspect the app and prepare a plan for a Bundler and Yarn upgrade.
The model is only one part of the system. Reliable orchestration comes from combining:
1
2
3
4
5
6
7
model choice
+ workspace boundary
+ tool access
+ approval policy
+ iterative tool loop
+ verification
+ rollback
Local Gemma is attractive because it removes cost and network dependency from routine work. But the largest reliability gains do not come from model size alone. They come from giving the model a narrow task, a real workspace, explicit constraints, and a requirement to verify its own changes.
For my current setup, the practical rule is simple:
Start routine engineering work with local Gemma, keep the scope narrow, approve mutations, verify everything, and escalate to a cloud model only when the local path demonstrates that it needs help.
This is not fully automatic orchestration yet. It is deliberate orchestration, and it is already useful.
]]>这几天 GitHub 上一批 Agent 相关项目突然变得很热。表面上看，它们都在讲 “AI coding agent” 或 “agent skills”。但放在一起看，我觉得更准确的判断是：
Agent 正在从聊天框，变成一种有工作台、有运行时、有工程方法论的执行系统。
这不是又一轮 prompt trick 的热闹。真正值得注意的是基础设施在补齐。
以前我们问模型一个问题，模型回答。后来我们给它工具，让它能读文件、跑命令、查网页。再后来我们给它 memory，让它不必每次从零开始。
现在的问题变成了：
1
一个 Agent 要长期可靠地干活，周围到底需要什么？
这五个项目刚好可以拆成三层。
代表项目是 cloudflare/computer。它的描述很直接：Give your agent a computer。
这里的 “computer” 不是一台完整虚拟机的营销说法，而是一个 Agent 需要的最小工作台：
它的文档里提到 SQLite-backed virtual filesystem，也提到通过 workspace.runtime 选择不同执行后端。这个抽象很重要，因为 Agent 如果只有聊天记录，就没有真正的工作现场。
一个严肃的 Agent 不应该只会说：
1
我会创建一个文件。
它应该真的有地方创建文件、修改文件、运行检查、留下结果：
1
2
3
4
5
workspace
-> files
-> commands
-> outputs
-> receipts
这也是我越来越觉得 Agent 系统里 “workspace” 应该成为一等对象的原因。聊天记录是交互界面，不是完整工作台。
对本地优先的系统来说，这个 workspace 未必需要在 Cloudflare 上。它可以是本地目录、git worktree、容器、数据库、QMD collection，或者这些东西的组合。关键不是技术选型，而是这个概念本身：Agent 需要一个可恢复、可审计、可清理的工作现场。
代表项目是 PrimeIntellect-ai/prime-agent。它把自己描述成 self-improving RLM agent for coding workflows and long-running autonomous tasks。
这里最值得看的不是 “它也能写代码”。能写代码已经不是新鲜事。
真正值得看的是它围绕长任务做的结构：background agents、daemon、worker、kernel、persistence、heartbeats、schedules、subagents、skills。换句话说，它不是把 Agent 当成一次性对话，而是当成一个可以脱离终端、继续推进任务、之后再被重新接上的运行实体。
这解决的是一个很现实的问题：
1
2
长任务不是一次回答。
长任务是一条会中断、会等待、会失败、会恢复的轨迹。
如果 Agent 只是一个聊天窗口，那么断线、超时、上下文丢失、人离开电脑，都会把工作变得脆弱。长任务需要的是运行时边界：
1
2
3
4
5
6
goal
-> current frontier
-> background execution
-> heartbeat
-> evidence receipt
-> reattach / resume
这和普通 “把 context window 变大” 不是一回事。更长上下文可以让模型看到更多历史，但它不能自动解决：
所以我更关心的是 goal register、frontier、gate、receipt 这些词，而不是单纯的 “agent memory”。Memory 很容易变成一个什么都往里塞的垃圾场。长任务需要的其实是更结构化的状态。
另外三个项目都和 skills 有关，但侧重点不同。
addyosmani/agent-skills 更像生产级工程技能包。它不只是放一堆提示词，而是把 skills、agents、hooks、commands、references、evals 和 docs 组织在一起。它强调技能应该 specific、verifiable、battle-tested、minimal。
mattpocock/skills 的气质不太一样。它像是从一个真实工程师的 .agents 目录里提炼出来的工作习惯：handoff、grilling、writing-for-agents、架构和交付流程。它有一种很实用的味道：少一点平台感，多一点 “我真的每天这样用”。
obra/superpowers 则更像一套 agentic skills framework 加软件开发方法论。它强调的不只是单个技能，而是如何用技能组织开发流程、子 Agent、计划、review 和修复循环。
这三个项目合在一起说明了一件事：
1
2
Skill 不应该只是“更好的提示词”。
Skill 应该是可复用的工程行为单元。
一个好的 Agent skill 至少应该回答：
如果一个 skill 只是在说 “请写高质量代码”，那它几乎没有价值。真正有价值的 skill 应该把资深工程师脑内的流程压缩下来：如何澄清需求，如何写计划，如何做最小改动，如何跑验证，如何请求 review，如何交接给另一个 Agent。
我觉得原因很简单：大家已经意识到，单个模型能力再强，也不能单独构成可靠系统。
一个能干活的 Agent 至少需要这三件事：
1
2
3
workspace -> 它在哪里工作
runtime -> 它如何持续工作
methodology -> 它按什么方式工作
没有 workspace，Agent 只能在对话里想象文件和结果。
没有 runtime，Agent 很难处理长任务、失败恢复和异步等待。
没有 methodology，Agent 可能会写代码，但不会像工程师一样交付。
所以这五个项目看起来分散，其实在补同一张图：
1
2
3
4
5
agent request
-> workspace with files/tools
-> runtime with state/recovery
-> skills with engineering discipline
-> verified artifact
这也是我觉得 “Agent 基建” 这个词比 “AI 编码工具” 更准确的原因。
我现在更愿意把 Agent 系统看成一个分层架构，而不是一个聪明模型外面包一点工具。
对一个本地优先、长期使用的个人 Agent 来说，比较理想的形态应该是：
1
2
3
4
5
6
7
8
stable preferences -> memory
rich notes/research -> QMD / docs
repeatable procedures -> skills
repeated deterministic -> scripts / commands
scheduled checks -> cron / no-agent jobs
long-horizon work -> goal register + receipts
code work -> worktree / sandbox / CI
public outputs -> reviewed blog/social pipeline
模型负责判断和综合。工具负责观察和执行。脚本负责重复。技能负责方法。文档负责长期知识。Git 负责历史。人负责边界、价值判断和授权。
这比 “让 Agent 自动化一切” 更保守，但也更可靠。
这类项目很容易让人兴奋。星标暴涨会制造一种 “必须马上安装” 的冲动。
我不太认同这个顺序。
对 Agent 基础设施，正确顺序应该是：
1
2
3
4
先读架构
再看边界
再做小实验
最后才考虑采用
尤其是能执行代码、写文件、联网、长期运行的系统，一定要先看清楚：
越像基础设施，越不能只看 demo。
如果只选几个方向继续看，我会这样排：
我不一定会直接采用任何一个项目。但我会把它们当成一组很好的设计样本。
因为真正的问题不是 “哪个 Agent 框架会赢”。真正的问题是：
1
2
一个长期和人一起工作的 Agent，
需要哪些基础设施才能可靠地完成真实工作？
这五个项目给出的答案越来越清楚了：
它需要一台电脑。
它需要一个运行时。
它需要一套工程方法论。
而模型，只是这个系统里最聪明、也最不应该单独承担全部责任的那一层。
I spent an afternoon auditing a small always-on VM that runs my personal AI agent, a couple of VPNs, and some dev infrastructure. It started as a routine setup review. It ended with me closing a database that required no password, on a machine with a public IPv4 address.
The fix list was useful. The process of finding the list was more useful. Here is both.
The box had felt fine for months. SSH was key-only. Nothing had ever been breached. The agent answered messages, the VPNs worked, scheduled jobs ran. Nothing felt broken, so nothing got looked at.
Then I actually looked.
Three quick probes changed the mood:
1
2
3
4
5
6
7
8
# what is listening, and on which interfaces?
ss -tlnp
# can I connect to the dev postgres without a password?
psql -h <host> -p <nonstandard_port> -U postgres -c "select 1"
# does redis ask for a password?
redis-cli -p <nonstandard_port> config get requirepass
The answers were: a lot, yes, and no.
Two containers — a dev Postgres and a dev Redis — were published on all interfaces with trust auth and an empty password. The Postgres prompt sat there and printed a row when I asked nicely. The Redis had no requirepass at all. Both were reachable from any network path the host was on, and the host has a public IP.
Redis with no auth is not a sleepy little cache. It is a write target: SSH key planting, cron overwrites, ransomware. The fact that it held zero keys was luck, not design.
The usual reasons, in the usual order:
docker compose publishes ports to 0.0.0.0 unless you say otherwise.POSTGRES_HOST_AUTH_METHOD: trust because it was “just dev”.None of those are exotic mistakes. That is the point. A dev database with no password is one of the most common setups in the industry, and on a machine with a public address it is a live finding, not a vibe.
I ran these before touching anything:
1
2
3
4
5
6
ss -tlnp # listeners and bind addresses
ufw status # is there a host firewall?
systemctl --failed # units that died and stayed dead
systemctl list-units # what is actually running
docker ps # container ports
ls -la <agent_config>/scripts # script permissions (group-writable is a smell)
Plus an agent-side security check: which automation jobs run with full tool access, and which command classes are auto-approved without prompting.
The port scan, the failed units, the auth probes, and the permission check each produced at least one finding. Nothing was a single smoking gun; everything was small and accumulated.
The smallest change with the largest effect: bind the published ports to loopback.
1
2
3
4
5
6
7
services:
db:
ports:
- "127.0.0.1:<db_port>:5432" # was "<db_port>:5432"
redis:
ports:
- "127.0.0.1:<redis_port>:6379" # was "<redis_port>:6379"
Local tooling still works. Nothing outside the host can connect. Verify with ss and confirm there is no 0.0.0.0 line left.
Rebinding fixes reachability. It does not fix the fact that the database had no password, so I set one and switched the auth method to SCRAM.
The gotcha: the official Postgres image only applies POSTGRES_HOST_AUTH_METHOD and POSTGRES_PASSWORD when it initializes a new data directory. My volume already existed, so the container recreated happily with trust still in pg_hba.conf. Config said one thing; enforcement said another.
For an existing volume you have to do it by hand:
1
2
3
4
5
6
7
8
9
10
11
# set the password
docker exec <db-container> psql -U postgres -c \
"ALTER USER postgres PASSWORD '<generated>';"
# rewrite pg_hba.conf: only the host lines, local socket admin stays
docker exec <db-container> sed -i \
's/^\(host.*\)trust$/\1scram-sha-256/' \
/var/lib/postgresql/data/pg_hba.conf
# reload
docker exec <db-container> psql -U postgres -c "SELECT pg_reload_conf();"
Then prove it both ways:
1
2
PGPASSWORD='' psql -h 127.0.0.1 -p <db_port> -U postgres -c "select 1" # must fail
PGPASSWORD='<generated>' psql -h 127.0.0.1 -p <db_port> -U postgres -c "select 1" # works
Redis is the same story in one line:
1
command: ["redis-server", "--requirepass", "${REDIS_PASSWORD}"]
The cloud security group is not a firewall I can see. I wanted defense in depth that I can see, so I added targeted drops on the host: inbound SMTP, rpcbind, and NFS ports that no service actually needs.
Two rules for myself:
I applied the drops with an idempotent script (delete existing instances, then apply) behind a small systemd oneshot unit so it survives reboots:
1
2
3
# /usr/local/sbin/fw-hardening.sh — the delete-then-apply pattern
iptables -D INPUT -p tcp --dport 25 -j DROP 2>/dev/null
iptables -A INPUT -p tcp --dport 25 -j DROP
I learned why idempotency matters the hard way: the first version checked-then-inserted, and when the unit ran a second time at enable --now, I ended up with duplicate rules. The delete-then-apply form makes re-running safe, and rollback is just the delete lines.
PermitRootLogin no drop-in instead of relying on the default.chmod 755.None of these are exciting. They are the difference between a host that was accidentally secure and one that is deliberately boring.
The agent-side audit found the same disease I had just cured in the containers: things running with more power than they need.
I kept one sandboxed script-execution tool auto-approved, deliberately, because it is core to how the agent works day to day. That is a tradeoff, and it should be a decision, not a default. Everything else went behind an approval prompt.
The same review caught a cost problem that is adjacent to security: a local language model server was running all the time, holding about 6.6 GB of RSS on a 12 GB host, as the last-resort fallback for the agent. It is used for bounded tasks a few times a week.
It is now on-demand: stopped and disabled by default, started for a task, stopped when the task ends. Service states are now explicit:
1
2
3
must-run -> SSH, VPN, the agent control plane, the firewall unit
on-demand -> local model server, databases, container workloads
retired -> whatever is backed up and documented, then removed
The on-demand model server also makes the fallback chain honest: it is the last entry, reached only if the primary and every cloud fallback fail, and if it is not running it fails fast instead of pretending to be a safety net.
127.0.0.1:.passwordless login must fail, not “the YAML looks right”.None of this was clever. It was a checklist, applied honestly, with tests that prove the fix actually worked. That is the entire trick.
]]>Part of the AI & Agents reading path.
A small thing happened while I was wiring a few Hermes hosts together.
I had a workflow that I could have left as a chat instruction: collect a little status from two worker machines, check that the shared knowledge repo does not contain secrets, commit the update, push it, sync the repo back to the workers, and verify that they received it.
That is exactly the kind of thing an LLM agent can do interactively.
But it is also exactly the kind of thing an LLM agent should not keep doing interactively forever.
The first run needs judgment. What should be shared? What should stay private? What is the source of truth? Which machines are allowed to decide, and which should only report evidence?
After that, the shape is mostly deterministic.
1
2
3
4
5
6
collect status
-> scan for secrets
-> commit
-> push
-> rsync
-> verify
So the workflow became a Python script. Then the script became a no-agent cron job.
That little promotion is one of the most important patterns I have learned from using LLM agents seriously: the LLM is not the whole system. The system is the loop around it, and the loop should get cheaper and safer every time it repeats.
The simplest rule is:
1
2
LLM for judgment.
Scripts for repetition.
A model is useful when the work is ambiguous:
But a model is wasteful when the work is deterministic:
That work belongs to tools, scripts, SQL, shell commands, APIs, or cron. The model can interpret the result if needed. It should not be paid to rediscover the mechanics every time.
A useful stack looks more like this:
1
2
3
4
5
6
7
judgment / ambiguity -> LLM
repeated procedure -> workflow doc or skill
deterministic action -> Python script
fast manual handle -> slash command
scheduled deterministic -> no-agent cron
rich searchable knowledge -> notes / docs / QMD
stable preference -> memory
The hard part is not knowing these layers exist. The hard part is noticing when a task has changed layers.
Natural language is a very good starting point. It lets me say:
1
Review this workflow and turn it into something reusable.
I do not need to know the exact files or all the edge cases yet. The agent can inspect, ask questions, propose a shape, and do the first implementation.
But if I ask the same thing again and again, the prompt is trying to tell me something. It wants to move down the stack.
1
2
3
4
5
6
chat prompt
-> documented workflow
-> skill
-> script
-> command
-> cron / webhook
That does not mean everything should become fully autonomous. Some workflows should stop at a checklist. Some should become a skill. Some should become a dry-run script. Some should become cron only after they are boring.
The point is to stop paying tokens for rediscovery.
If I have to explain the same constraints every week, the system has not learned. It has only remembered long enough to finish the current chat.
A weak agent loop ends with:
1
I changed it.
A stronger loop ends with:
1
I changed it, ran the verification, and this is what happened.
For code, that means tests or builds. For infrastructure, it means service state and connectivity checks. For publishing, it means the page is actually live. For scheduled posts, it means the queue contains the expected text and time.
The loop should be explicit:
1
2
3
4
5
6
understand
-> gather context
-> act
-> verify with real output
-> report
-> promote the durable lesson
This changes the feel of working with an agent. The important output is not the confident explanation. The important output is the verified state change.
One easy mistake is to treat memory as the place where everything should go.
That creates a different problem. If everything is memory, memory stops being useful. It becomes a noisy, always-on prompt tax.
I prefer to separate the types of knowledge:
1
2
3
4
5
6
memory -> stable facts and preferences needed every turn
skills -> reusable procedures
notes/docs -> rich context and longer reasoning
scripts -> deterministic actions
Git repos -> versioned operational history
session logs -> temporary task history
A stable preference belongs in memory. A procedure belongs in a skill. A long explanation belongs in a document. A repeated action belongs in a script. A change history belongs in Git.
That separation matters because each layer has a cost. Memory is paid every turn. A document is retrieved when needed. A script runs without tokens. A Git commit gives history without forcing every future prompt to carry the whole story.
Good agent design is partly information budgeting.
Some work should not happen at the moment I have the idea.
A voice note can be captured quickly now and processed later. A knowledge base can be embedded on a schedule. A weekly review can cluster notes in a batch. A code intelligence index can run overnight or weekly instead of during every tiny question.
The pattern is:
1
2
capture quickly now
process deeply later
This is not only about saving money. It also improves attention. Not every thought deserves an immediate deep workflow. Some thoughts should be captured cheaply and allowed to meet other related thoughts later.
For LLM systems, batching is a kind of patience.
When there are multiple agents or multiple hosts, it is tempting to make them all equally smart and equally authoritative.
That sounds powerful, but it can create three conflicting memories, three sets of assumptions, and three bots trying to be the final decision-maker.
A cleaner pattern is:
1
2
3
4
5
6
7
primary agent
-> owns memory, policy, promotion, final synthesis
worker agents
-> run local checks
-> collect evidence
-> report concise results
Workers do not need the whole brain. They need the right local tools and a clear role.
The primary can ask a worker to check a host, run a small test, inspect a file, or verify a backup target. The worker returns evidence. The primary decides what becomes durable knowledge.
This is less glamorous than a swarm demo, but it is easier to trust.
Human approval is sometimes treated as a weakness in agent systems. I think that is backwards.
For some actions, the human is not just an input device. The human is the owner of responsibility.
These actions should have explicit gates:
The goal is not to make the human approve every shell command. The goal is to automate everything around the meaningful decision so the approval is small, informed, and real.
1
2
3
4
5
6
agent prepares
agent verifies preconditions
agent explains risk
human approves the boundary crossing
agent executes
agent verifies outcome
That is not slower than unsafe automation. It is faster than cleaning up after a confident mistake.
Agents are very good at moving information between places.
That is useful and dangerous.
A private note can become a public blog post. A shell output can become a status report. A host runbook can become a shared repo. A social post can quote a detail that should have stayed internal.
So anything that leaves its original boundary should be scanned or reviewed for:
The standard should be boring: sanitize by default, especially when promoting internal operational work into public writing.
A script that can only execute is usually not ready to be trusted.
Useful automation should have escape hatches:
1
2
3
4
5
--dry-run
--no-push
--no-rsync
--skip-remote
--execute only after review
Dry-run mode is not just for safety. It makes the script easier to reason about. It shows what the automation believes it is about to do.
That is exactly the kind of surface an LLM can review well: not a vague intention, but a concrete planned action.
If a failure happens once, fix it.
If it happens twice, encode the lesson.
That might mean:
The best agent systems do not become smarter only because the model improves. They become smarter because the environment accumulates guardrails.
A repeated mistake should not become a heroic debugging story every time. It should become harder to repeat.
The chat box is where the interaction starts, but it is not the whole interface.
The real interface is a living system:
1
2
3
4
5
6
7
8
human intention
-> LLM judgment
-> tools and scripts
-> versioned knowledge
-> scheduled checks
-> verification
-> human approval where needed
-> improved procedures
That is why I think the interesting question is no longer only “how do I prompt this model?”
The better question is:
How should this work evolve after the first successful run?
If the answer stays as a prompt forever, it will keep costing attention and tokens.
If the answer becomes a skill, a script, a cron job, a guardrail, or a documented decision, the system has actually learned.
That is the practical promise of LLM agents for me: not replacing judgment, but turning repeated judgment into better tools around future judgment.
The model is powerful.
But the compounding value is in the workflow you leave behind.
]]>When people first meet an LLM, the interaction looks simple.
You type something. The model replies.
1
human -> prompt -> model -> answer
That picture is not wrong, but it is too small. It describes the chat box, not the work.
The more I use LLMs inside daily engineering and life systems, the less I think the interesting part is the prompt by itself. The interesting part is the structure around the model.
We started with prompts. Then context. Then agents. Then memory. Then tools. Now I keep coming back to two bigger shapes: loops and graphs.
Loop engineering is about how an AI system keeps doing useful work.
Graph engineering is about how an AI system understands relationships.
Both matter. They are not the same thing.
Prompt engineering was the first vocabulary most of us learned.
Ask clearly. Give examples. Specify the format. Tell the model what role to take. Add constraints. Say what good output looks like.
That still matters. A vague request gets vague work. A precise request gives the model more handles.
1
2
3
4
5
6
Bad:
Help me with this bug.
Better:
Read the failing test, explain the likely root cause,
make the smallest safe change, then run the focused test.
But prompt engineering has a ceiling.
A prompt is a moment. It depends on what I remember to say this time. It lives inside one conversation. If I have to retype the same careful instruction every week, I have not built a system. I have built a little ritual.
That is fine when I am exploring. It is not enough for reliable work.
The next layer is context engineering.
The question becomes: what should the model know before it answers?
That might be:
This is where notes, search, session history, RAG, and code tools enter the picture.
The model is not only generating from training data. It is being placed inside a local situation.
1
2
3
4
5
6
7
request
+ current files
+ logs
+ notes
+ memory
+ docs
-> better answer
Context engineering changes the interaction from asking a clever stranger to asking a temporary teammate who has read the room.
But context also has a problem. Too little context makes the model guess. Too much context makes the model drown. Wrong context can be worse than no context, because it gives confidence to the wrong path.
So the skill is not “add more context”.
It is choosing the right context for the next action.
A plain LLM can describe a command. An agent with tools can run it.
That changes the interaction.
1
2
3
4
model says: run the test
agent does: run the test
agent reads: real output
model decides: next step
The model does not need to pretend it knows whether the test passes. It can check. It does not need to invent the file contents. It can read the file. It does not need to guess the current branch. It can ask Git.
Tool engineering is partly about giving the model capabilities. More importantly, it removes the temptation to hallucinate.
If the system can observe the world, it should observe the world.
A single LLM conversation can be useful. A long-running assistant needs memory.
But memory is not “save everything”. That is another way to drown the model.
Different things belong in different places:
1
2
3
4
5
6
7
stable facts -> memory
reusable procedure -> skill or workflow
documents/notes -> searchable knowledge base
code relationships -> code graph
one-off progress -> session history
commands -> scripts or slash commands
scheduled work -> cron or webhook
This separation matters because each type of knowledge has a different half-life.
My preference for concise status reports is stable. A failed test output from this morning is not. A workflow for preparing a blog post may be reusable. The exact draft path is temporary. A production deploy rule may be durable. A branch name may be stale tomorrow.
Good memory engineering is not about making the AI remember everything.
It is about making it forget less stupidly.
Once the model can read context, use tools, and remember durable lessons, the question changes.
What loop should it run?
For me, a useful agent loop looks something like this:
1
2
3
4
5
6
7
capture
-> retrieve
-> plan
-> act
-> verify
-> report
-> remember/promote
This is loop engineering.
It is the difference between a model that answers and a system that works.
A good loop observes before acting. It reads the file, checks the branch, inspects the error, searches the notes, or asks for the missing decision.
It acts in small steps. It does not rewrite the whole system when a focused patch would do.
It verifies with reality: tests, builds, live URLs, logs, API responses, file checks, timestamps. Not vibes.
It knows when to stop. Infinite autonomy is not intelligence. Sometimes the correct next step is to report a blocker or ask for approval.
It promotes repeated work. If I keep asking the same thing, the loop should move from chat into a workflow, then maybe a command, script, API, cron job, or product feature.
This is why I think loop engineering is the default best practice for an assistant like Hermes. Hermes is not only a model endpoint. It is a tool-using, memory-backed, scheduled, multi-surface agent. The natural unit is not the prompt. It is the verified loop.
Graph engineering answers a different question.
What is connected to what?
A graph can represent code symbols, documents, people, decisions, projects, dependencies, concepts, tasks, APIs, or memories.
1
2
3
4
5
6
concept: loop engineering
-> related to: agent design
-> related to: verification
-> related to: memory promotion
-> used in: weekly review workflow
-> supported by: tools, skills, cron
In code, a graph can tell me which functions call another function, which route reaches which service, or what might break if I change a model field.
In notes, a graph can show that an idea from a voice note connects to a blog draft, a work process, and a product decision.
In memory, a graph can connect a person, a preference, a repo, and a repeated workflow.
This is useful because LLMs are very good at language, but real work is full of relationships. The answer often depends less on one document and more on the path between several things.
Graph engineering gives the agent better maps.
But a map is not the journey.
This is the distinction I keep coming back to.
For an AI assistant, graph engineering should usually support the loop. It should not replace the loop.
The graph helps the system retrieve the right context:
1
2
3
4
What code path is affected?
What note is connected to this idea?
What previous decision applies here?
What dependency might be touched?
The loop decides what to do with that context:
1
2
3
4
5
6
inspect
plan
change
test
report
promote
So my current rule is:
1
2
loop engineering for action
graph engineering for understanding
If the task is “fix this”, “draft this”, “review this”, “prepare this”, or “monitor this”, I want a loop.
If the task is “what is related?”, “what depends on this?”, “where is this used?”, or “what context matters?”, I want a graph.
The best systems combine them, but they should not confuse them.
There is one more loop that matters: the human loop.
A lot of AI demos quietly assume that the best system is the one that removes the human as much as possible. Sometimes that is true. I do not want to manually rotate log files or retype the same invoice calculation forever.
But for many tasks, the human is not just a source of commands. The human is the owner of judgment.
Deploying to production, changing customer-facing infrastructure, exposing secrets, sending emails, publishing posts, making family decisions, spending money: these are not just technical actions. They carry responsibility.
A good agent should know which loop it is inside.
1
2
3
4
5
6
7
8
safe deterministic task
-> automate
repeated but judgment-heavy task
-> workflow + review
sensitive external side effect
-> explicit approval
The goal is not maximum autonomy everywhere.
The goal is appropriate autonomy.
This is the part I think people underestimate.
LLMs are not normal software components. They are not databases. They are not search engines. They are not humans either.
They are strange language engines that can temporarily inhabit a context, make plausible plans, call tools, explain uncertainty, and then lose the shape of the work unless we build structure around them.
So the interaction is not simply instruction and response.
It is setting up conditions for situated reasoning.
1
2
3
4
5
6
7
Give the model the right role.
Give it the right context.
Give it tools to observe reality.
Give it a loop that checks its work.
Give it memory for durable lessons.
Give it graphs for relationships.
Give it human approval where responsibility matters.
Then the model becomes less like an oracle and more like a moving part in a thinking system.
That feels like the shift.
Not “AI will answer everything”.
More like: we are learning how to build systems where language, tools, memory, graphs, and human judgment keep passing state between each other.
If I had to explain the stack from bottom to top, I would describe it like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Prompt engineering
Say the thing clearly.
Context engineering
Bring the right facts into the room.
Tool engineering
Let the model observe and act.
Memory engineering
Preserve durable lessons without polluting the present.
Graph engineering
Map relationships between code, notes, people, tasks, and decisions.
Loop engineering
Turn all of the above into repeated, verified work.
Human judgment
Decide what should be automated, reviewed, approved, or refused.
Each layer solves a different failure mode.
Prompts reduce ambiguity. Context reduces ignorance. Tools reduce hallucination. Memory reduces repetition. Graphs reduce isolation. Loops reduce drift. Human judgment reduces irresponsible automation.
That is a more useful mental model than arguing whether “prompt engineering is dead”.
Prompting did not die. It became one layer in a larger system.
For Hermes, and probably for most serious personal agents, I think the best practice is:
1
loop-first, graph-assisted, human-governed
Use loops to make the assistant reliable.
Use graphs to make the assistant aware of relationships.
Use memory and skills to make it improve over time.
Use tools and verification to keep it grounded.
Use human approval where the cost of being wrong is not just a bad answer.
That is the basic understanding I want to keep building.
The interesting question is no longer only “what should I type into the LLM?”
It is:
What kind of interaction am I building around it?
]]>For a long time, the practical mental model was simple: one Ampere A1 VM with 4 OCPUs and 24 GB of memory, or the same total split across smaller instances. That is enough to run a surprising amount of personal infrastructure: VPNs, a small Kubernetes cluster, databases, background workers, an AI agent control plane, local experiments, and whatever else slowly accumulates on a machine that is always on.
The current public shape is smaller. Oracle’s Always Free documentation now says Ampere A1 VM instances get the first 1,500 OCPU hours and 9,000 GB-hours per month for free. Oracle describes that as equivalent to 2 OCPUs and 12 GB of memory for Always Free tenancies. The Oracle Cloud Free Tier page says the same thing more directly: Arm-based Ampere A1 cores and 12 GB of memory, usable as one VM or two VMs.
That changes the engineering question.
It is no longer: “What can I fit on a very generous free ARM box?”
It is: “What deserves to be always-on if the honest budget is 2 cores and 12 GB?”
The useful public references are:
The important lines, as of this writing:
That last pair is the tension.
I do not think most people read “Always Free” as a benchmark number. They read it as a promise.
Not necessarily a promise that every quota will be frozen forever. Cloud providers change instance families, limits, abuse controls, capacity policies, and commercial packaging all the time. But “Always Free” creates a stronger expectation than “trial” or “promotional credit”. If a provider trains users to build around a free shape, and that shape later becomes smaller, the change feels less like ordinary pricing maintenance and more like a contract-shaped disappointment.
There is a legal version of that discussion, and I am not trying to make it here. The practical engineering version is enough:
1
2
3
Marketing promise: this is Always Free
Operational reality: capacity and limits can change
User response: design the system as if the shape is not guaranteed
That sounds cynical, but it is just the cloud version of not treating someone else’s free tier as your only production plan.
The fair reading is probably this: the account can still have Always Free resources for an unlimited period, but the resource envelope is not an immutable interface. If you need a particular shape, you should budget for it or have a downgrade plan.
The old practical shape was 4 OCPUs and 24 GB of RAM running continuously.
At Oracle’s published Ampere A1 rates:
1
2
OCPU: $0.01 per OCPU-hour
Memory: $0.0015 per GB-hour
A full 4 OCPU / 24 GB machine for an average 730-hour month is roughly:
1
2
3
4 OCPUs * 730 hours * $0.01 = $29.20
24 GB * 730 hours * $0.0015 = $26.28
Gross monthly shape cost = $55.48
If the first 1,500 OCPU-hours and 9,000 GB-hours are free, then keeping the old 4 / 24 shape continuously means paying for the part above the new free envelope:
1
2
3
4
5
6
Billable OCPU-hours: (4 * 730) - 1,500 = 1,420
Billable GB-hours: (24 * 730) - 9,000 = 8,520
OCPU overage: 1,420 * $0.01 = $14.20
Memory overage: 8,520 * $0.0015 = $12.78
Estimated monthly overage = $26.98
For a 31-day month the estimate is about $28.04. For a 30-day month it is about $26.22.
That is not an outrageous bill. In fact, for a 4-core ARM VM with 24 GB of RAM, it is still cheap.
But it changes the category. A free hobby host becomes a small paid server. That may be perfectly fine, but it should be an explicit decision, not something discovered after the bill arrives.
I decided to treat 12 GB as the real design constraint.
The point was not to recreate the old host badly. The point was to decide what the host is actually for.
For me, the always-on role is:
1
2
3
4
5
small ARM host
-> private access paths
-> basic shell and development access
-> lightweight automation
-> scheduled maintenance
Everything else has to justify itself.
That means the host should keep only the boring access and automation layer always running. Heavier platform services, databases, container workloads, local model servers, and one-off experiment processes should become explicitly on-demand.
That is the important shift: not “can I squeeze it all in?” but “should this be always-on?”
The safe version of downsizing is not to uninstall everything in a panic.
For each service, I want one of three states:
1
2
3
must-run -> enabled and monitored
on-demand -> stopped and disabled, data retained
retired -> backed up, documented, then removed later
For this host, the first pass was mostly “on-demand”: stop services, disable boot-time autostart, keep the data.
A service table is more useful than a vague note:
| Component | New default | Start when needed | Stop after use |
|---|---|---|---|
| k3s | stopped + disabled | sudo systemctl start k3s |
sudo systemctl stop k3s && sudo k3s-killall.sh |
| MySQL | stopped + disabled | sudo systemctl start mysql |
sudo systemctl stop mysql |
| Multipass | stopped + disabled | sudo systemctl start snap.multipass.multipassd.service |
sudo systemctl stop snap.multipass.multipassd.service |
| WARP | stopped + disabled | sudo systemctl start warp-svc.service |
sudo systemctl stop warp-svc.service |
| Local LLM server | on-demand | start the model server for a bounded task | stop it when the task ends |
The Kubernetes line matters. Stopping k3s is not always enough, because container shims, mounts, CNI state, and iptables rules can linger. k3s-killall.sh is useful because it clears runtime state without uninstalling k3s or deleting persistent data.
That gives me a reversible downgrade. If I need the cluster for an experiment, I can start it. If I do not, it is not allowed to quietly consume RAM every day.
Before changing the cloud shape, I want evidence that the smaller role is already true on the larger host.
The checklist is boring on purpose:
1
2
3
4
5
6
7
8
9
10
11
12
hostname
uname -m
nproc
free -h
swapon --show
df -h /
systemctl is-active <must-run-services> || true
systemctl is-enabled <must-run-services> || true
systemctl is-active <on-demand-services> || true
systemctl is-enabled <on-demand-services> || true
ss -tulpn
The expected result is:
The resize itself is still a stop/start event. If the host runs the automation path that talks to me, I should expect that path to be offline during the shape change unless I intentionally fail it over first.
A smaller server is not just a smaller server. It is also a forcing function for deciding what is precious.
Before resizing, I want backups for the things that would be annoying or risky to reconstruct:
The backup should be encrypted or permission-restricted, checksummed, and copied somewhere other than the host being resized. It should also be treated as sensitive. A restore archive for VPN, SSH, and agent configuration is not a blog artifact; it is effectively infrastructure authority in a tarball.
The goal is not to make the host precious. It is the opposite: make the host disposable enough that a failed resize is inconvenient, not existential.
The Oracle change is annoying because the old free ARM shape was genuinely useful. A 4 OCPU / 24 GB always-on machine is enough to hide architectural laziness for a long time.
The smaller 2 OCPU / 12 GB shape is less forgiving, but it is still useful if the host has a clear job.
My rule now is:
1
2
free tier = great control plane
free tier != dumping ground
If I need the old shape continuously, the honest cost is roughly $27 to $28 per month above the new Always Free envelope. That is cheap enough to consider and expensive enough to make me clean up first.
So the plan is to downsize the host, not the responsibility:
That is a healthier design anyway. The cloud provider changed the limits; the useful response is to make the machine’s purpose smaller and clearer.
]]>Then CI moved again.
This is the story of a flaky test suite that wasn’t one problem. It was a small cluster of assumptions that only became visible under parallel CI and a newer browser.
Our Rails app runs its feature specs in CI with parallel_tests — seven RSpec processes sharing one database setup. A normal merge to main triggers the full suite.
After merging a small mailer change, the CI run came back red:
1
2
X execution expired
JUnit Test Report: ./spec/features/admin/registrations_spec.rb#32
execution expired is Ruby’s Timeout::Error. The spec that failed was downloading a CSV and waiting for the file to appear. I looked at the spec locally, ran it, and it passed in five seconds.
Fine. Flaky spec. Rerun the failed job and move on.
The rerun failed again. This time on two different specs:
1
2
3
4
X execution expired
JUnit Test Report: ./spec/features/admin/registrations_spec.rb#109
X execution expired
JUnit Test Report: ./spec/features/admin/programs_spec.rb#167
Same error. Different lines. None of the three specs had anything to do with the mailer change I had just merged. (I’m deliberately using placeholder spec names and leaving out run IDs here — the numbers and names don’t matter, the shape does.)
By now the shape of the problem was hard to miss. Every failure was:
All three specs used the same test helper:
1
2
3
4
5
6
7
8
9
10
module DownloadHelpers
TIMEOUT = ENV.fetch('DOWNLOAD_TIMEOUT', 60).to_i
PATH = Rails.root.join('tmp', 'downloads').freeze
def wait_for_download
Timeout.timeout(TIMEOUT) do
sleep 0.5 until downloaded?
end
end
end
And here is the part that made it intermittent: in rails_helper.rb, every single feature spec clears the download folder before it runs:
1
2
3
config.before(:each, type: :feature) do
DownloadHelpers.clear_downloads
end
clear_downloads does FileUtils.rm_r on everything in tmp/downloads.
Now think about what happens with seven parallel processes. Process 3 clicks “Download” and starts polling for the file. Process 5 starts a feature spec, runs before(:each), and deletes every file in the shared tmp/downloads folder. Process 3’s file vanishes before the poll sees it. It waits, polls, waits, and dies 60 seconds later with execution expired.
It was a race. A shared mutable directory, seven processes, and a cleanup step that assumes it owns the folder. No wonder the failure moved around — it depended entirely on which processes happened to overlap in that window.
This is the part that made the whole thing confusing at first. Every spec passed on my machine, every time.
Because on my machine there is exactly one RSpec process. It clears the download folder, downloads the file, reads it, and clears again. There is nobody else deleting files out from under it. The race needs two processes, and locally there is only one.
The same commit that was green in one CI run was red in another. That should have been the tell. A genuinely broken spec fails the same way every time. A spec that fails differently each run is usually a victim of something else.
The app already had the answer waiting in config/application.rb. Under parallel_tests, each process gets a TEST_ENV_NUMBER, and the app already uses it to give each process its own Redis database:
1
2
3
if ENV['RAILS_ENV'] == 'test' && ENV['TEST_ENV_NUMBER'].to_s != ''
# shift each process onto its own redis db
end
The download helper just needed the same treatment:
1
2
3
4
5
6
7
8
module DownloadHelpers
test_env_number = ENV.fetch('TEST_ENV_NUMBER', nil).to_s
PATH = Rails.root.join(
'tmp',
'downloads',
test_env_number.empty? ? '' : "process_#{test_env_number}"
).freeze
end
Process 3 downloads into tmp/downloads/process_3, process 5 clears tmp/downloads/process_5. They never touch each other’s files. I also made clear_downloads create the directory first, because Chrome will not reliably create a missing nested download folder on its own.
One commit, one file changed:
1
test: isolate download dir per parallel test process
CI came back green. The flaky suite stopped failing in that particular way, because it had never really been flaky — it had been a shared-state bug that only existed when the suite ran in parallel.
But that was not the end of the story.
After the download-folder fix, the failures changed shape. They were no longer clean 60-second timeouts. Instead, feature specs sometimes died while Capybara was interacting with dynamic widgets:
1
2
unknown error: unhandled inspector error:
{"code":-32000,"message":"Node with given id does not belong to the document"}
That is Chrome telling Selenium that a DOM node disappeared between lookup and action. In older or simpler cases, Selenium reports this as a normal stale-element error, and Capybara knows how to retry it inside its synchronization loop. This one came through as a generic Selenium unknown error, so Capybara treated it as fatal.
The tempting fix is dangerous: catch the message and let Capybara retry forever until the global wait expires.
That makes the suite quieter, but it also makes CI slower. A real failure now burns the whole wait window before it tells you anything useful.
The fix I wanted was narrower:
1
2
3
4
5
6
7
8
9
10
11
12
13
module CapybaraChromeStaleNodeFix
CDP_STALE_NODE_MESSAGE = 'does not belong to the document'
MAX_CDP_STALE_NODE_RETRIES = 2
CDP_STALE_NODE_RETRY_WINDOW_SECONDS = 0.5
def catch_error?(error, errors = nil)
if chrome_stale_node_error?(error) && retry_chrome_stale_node_error?
true
else
super
end
end
end
The important part is not the exact constants. It is the shape of the guard:
That is the difference between stabilizing a transient browser race and hiding a broken spec behind a slower timeout.
We also had explicit helpers like wait_for_ajax and wait_for_page_to_settle. They used Capybara.default_max_wait_time, which is reasonable for user-facing assertions but too generous for a tiny internal settle check.
If the page is basically ready and we are only waiting for current AJAX requests or a progress bar to disappear, we should not spend the same budget as a real browser assertion.
So those helpers moved to a small explicit wait:
1
2
3
4
5
6
7
8
CAPYBARA_FAST_WAIT_SECONDS = 1.0
CAPYBARA_POLL_INTERVAL_SECONDS = 0.05
def wait_for_ajax(timeout_seconds = CAPYBARA_FAST_WAIT_SECONDS)
wait_until_with_timeout(timeout_seconds, 'Timed out waiting for ajax requests to finish') do
finished_all_ajax_requests?
end
end
I also avoided wrapping browser calls with Ruby’s Timeout.timeout. Interrupting Selenium in the middle of a WebDriver call is not a great failure mode. A monotonic deadline loop is boring, but boring is good here.
One CI run after the retry change still failed, this time with two ordinary-looking assertion failures.
The first was an announcement-editing spec. The spec tried to remove all selected program filters by mutating Selectize’s hidden input directly with JavaScript, then submitting the form. The visual labels disappeared, but the submitted form value could still contain the old selected IDs under CI timing.
That is not a product bug. It is a test that bypassed the UI and then expected UI state, Selectize state, and form state to agree.
The fix was to exercise the path a user actually takes: remove the visible labels, wait for the selected options to disappear, and submit the specific admin form.
The second was a registration spec. After submitting a sign-in form, it immediately asserted that the database association existed. Locally it usually did. In CI, the browser had not always completed the redirect/session flow before the assertion ran.
The fix was not another sleep. It was to wait for the visible post-login page state first:
1
2
3
fill_in_sign_in_form(user.email)
expect(page).to have_text('Programs')
expect(user.reload.registration_items.count).to eq 1
Feature specs should synchronize on what the user can see. If the user-visible page is not there yet, the test has no business asserting the side effect as if the flow had finished.
A different spec failing every rerun is itself a clue. A broken test fails the same way every time. A moving failure points at shared state, browser timing, or a helper assumption that many tests share.
Check what the tests share before blaming the tests. Parallel test suites run the same code in the same working directory. Files, folders, ports, Redis keys, environment variables, browser helpers — anything shared is a candidate. In our case the app had already solved this for databases and Redis; the downloads folder was simply the one thing nobody had partitioned yet.
Timeouts that always hit the same limit are a smell. A 60-second timeout that always burns the full 60 seconds is not a slow test. It is a test waiting for something that will never arrive. Retrying a browser error through the full Capybara wait can create the same problem in a different form.
Local green runs do not clear CI. Single-process local runs cannot reproduce races that need concurrency. If a spec only fails under parallel CI, try reproducing it with two processes and a shared resource, or just look for the shared resource directly.
Make retries bounded and specific. A retry should describe the transient failure it is meant to absorb. If it catches too much, or waits too long, it becomes a performance bug and hides useful failures.
Synchronize feature specs on user-visible state. Waiting for a page heading, menu, or result is better than asserting the database immediately after a click. Capybara is good at waiting for the browser; use that instead of racing the browser from Ruby.
The first fix was ten lines in a test helper. The later fixes were smaller but more subtle: a bounded browser retry, shorter explicit waits, and two feature specs changed to synchronize on the UI instead of internal state.
The expensive part was not the code. It was accepting that “flaky CI” was not a single diagnosis. It was a queue of small incorrect assumptions, and each green run only proved that we had removed the current one.
]]>