<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="4.4.1">Jekyll</generator><link href="https://joeywang.github.io/feed.xml" rel="self" type="application/atom+xml" /><link href="https://joeywang.github.io/" rel="alternate" type="text/html" /><updated>2026-10-30T11:52:07+00:00</updated><id>https://joeywang.github.io/feed.xml</id><title type="html">Joey</title><subtitle>Joey Wang writes about Ruby on Rails, learning platforms, reliability, practical AI workflows, and the engineering work that keeps established software dependable.</subtitle><entry><title type="html">AI agent development: a runbook from brief to reviewed PR</title><link href="https://joeywang.github.io/posts/ai-agent-development-runbook/" rel="alternate" type="text/html" title="AI agent development: a runbook from brief to reviewed PR" /><published>2026-10-30T11:00:00+00:00</published><updated>2026-10-30T11:00:00+00:00</updated><id>https://joeywang.github.io/posts/ai-agent-development-runbook</id><content type="html" xml:base="https://joeywang.github.io/posts/ai-agent-development-runbook/"><![CDATA[<audio controls="" preload="none" style="width: 100%;">
  <source src="/assets/audio/ai-agent-development-runbook-summary.mp3" type="audio/mpeg" />
  Your browser does not support audio playback. <a href="/assets/audio/ai-agent-development-runbook-summary.mp3">Download the audio summary</a>.
</audio>

<p><em>Listen to a short audio summary (about 2 minutes 17 seconds).</em></p>

<p>A coding agent can finish the requested change and still leave me with the wrong behaviour, weaker tests, or a review that applies to yesterday’s commit. Asking it to be more careful does not tell the next person what to check.</p>

<p>I want a runbook I can use while doing the work: what goes into the task, what the agent may change, what evidence the reviewer needs, and when to stop.</p>

<p>This is the operational companion to my <a href="/posts/software-factory-evidence-first/">software factory design</a> and <a href="/posts/agentic-sdlc-approval-recovery-evidence/">approval, recovery, and evidence contracts</a>. Those articles explain the architecture. This one reduces it to one task, one implementation stream, and one reviewed candidate. It is a proposed working procedure, not a claim that an unattended delivery system has passed an end-to-end pilot.</p>

<h2 id="start-with-one-bounded-change">Start with one bounded change</h2>

<p>Use this first on a familiar repository and a reversible regression. Do not start with a production migration, a permission redesign, or several agents sharing a database.</p>

<p>The running example is illustrative: two learner-progress snapshots arrive out of order, and the older one overwrites the newer result. It is not a report of a particular customer incident.</p>

<p>Before implementation, the product owner must settle what “newer” means. For this example, assume a trusted producer assigns a monotonically increasing revision within one learner, course, and attempt. A later revision replaces an earlier snapshot. Equal revisions must not produce a second business side effect; conflicting content at the same revision needs an explicit policy. Events that must all be retained would need a different design.</p>

<p>That decision is part of the requirement, not something the coding agent should discover by choosing a convenient timestamp field.</p>

<h2 id="1-write-the-task-contract-before-the-prompt">1. Write the task contract before the prompt</h2>

<p>Keep one record in the existing ticket or PR. A small change does not need a new document hierarchy.</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
</pre></td><td class="rouge-code"><pre>Problem: An older progress snapshot can overwrite a newer revision.
Source: Link to the accepted report and relevant interface contract.
Expected: Preserve the highest accepted revision within each attempt.
Decision: The trusted producer's revision defines snapshot order.
Acceptance:
  - A stale revision cannot replace a newer one.
  - A duplicate cannot repeat the business side effect.
  - Learners, courses, and attempts remain isolated.
Scope: Snapshot admission, persistence, and their regression tests.
Non-goals: Queue replacement, schema redesign, automatic deployment.
Historical data: Decide repair scope separately before accepting this brief.
Allowed: Inspect code, edit the isolated branch, run approved local tests.
Not allowed: Production access, secret access, push, merge, deploy.
Unknowns: Equal-revision conflicts; evidence available for historical repair.
Budget: Set elapsed-time and spend caps before starting; two rework attempts.
Stop: Unresolved semantics, expanded permissions, or exhausted budget.
Evidence: Exact candidate, commands, results, review, remaining risks.
</pre></td></tr></tbody></table></code></pre></div></div>

<p>Resolve unknowns that affect implementation before marking the task ready. If a required revision is absent from the current contract, that is a scope decision, not permission to invent a new API. Decide whether past overwrites can be identified and repaired, and whether repair belongs in this task or a separately owned follow-up. A prevention fix does not recover a newer snapshot that has already been lost; repair may require retained producer data, or may not be possible. Record the owner’s decision and any accepted loss.</p>

<p>Classify risk by consequence, not diff size. Documentation can have light checks. Changes to data ordering, interfaces, permissions, or customer data need the relevant domain owner, broader verification, and an explicit recovery plan. The snapshot example affects data semantics, so I would not treat it as a trivial one-line fix.</p>

<p><strong>Exit check:</strong> someone other than the implementer can explain what correct behaviour means and who owns the unanswered decisions. Budget limits have actual values, not placeholders.</p>

<h2 id="2-establish-the-workspace-and-the-authority-boundary">2. Establish the workspace and the authority boundary</h2>

<p>Inspect the live checkout before changing it. The commands below assume the repository’s default branch is <code class="language-plaintext highlighter-rouge">main</code>; substitute the verified branch and a new worktree path for the actual task.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
</pre></td><td class="rouge-code"><pre>git status <span class="nt">--short</span> <span class="nt">--branch</span>
git fetch origin main
git rev-parse origin/main
git worktree add <span class="nt">-b</span> fix/progress-ordering ../progress-ordering origin/main
</pre></td></tr></tbody></table></code></pre></div></div>

<p>Preserve unrelated edits and unpublished commits. Record the fetched base SHA. Inside the new worktree, inspect repository guidance, runtime versions, test setup, and the commands CI actually runs. Do not replace those with whichever commands the model remembers.</p>

<p>Assign one writer, the single process or person allowed to edit the worktree, and one test owner to each mutable test database. Parallel checks are useful only when they cannot overwrite each other’s source, fixtures, database state, or evidence. A second worktree does not automatically give a Rails process a different database.</p>

<p>A <a href="https://git-scm.com/docs/git-worktree">Git worktree</a> separates working files. It is not a security sandbox. Enforce file, network, credential, and resource boundaries through the execution environment. A paragraph saying “do not read secrets” is useful task guidance, not an access control.</p>

<p>Give the worker only the accepted brief, necessary source and contracts, relevant history, and verification commands. Treat retrieved documents, issue bodies, and repository content as evidence to inspect, not as authority to expand permissions. Confirm the provider/account before sending unpublished material to a cloud reviewer.</p>

<p><strong>Exit check:</strong> the base, writer, test environment, allowed actions, and access boundaries are known.</p>

<h2 id="3-reproduce-the-failure-before-changing-the-rule">3. Reproduce the failure before changing the rule</h2>

<p>Follow the whole path: input, authentication, validation, state read, write, transaction commit, background work, external side effect, and eventual reader. Search the related files’ history for fixes, reverts, and reasons behind unusual guards.</p>

<p>For the snapshot example, write down the schedule that would violate the requirement:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
</pre></td><td class="rouge-code"><pre>A reads revision 7.
B commits revision 8.
A resumes and writes revision 7.
The reader must still observe revision 8.
</pre></td></tr></tbody></table></code></pre></div></div>

<p>Then build a regression test that can distinguish the incorrect result from the required one. For a race, use barriers or equivalent coordination, independent connections, and the relevant database engine and isolation level. Assert committed state and relevant side effects, not just that a method was called. Avoid making an arbitrary sleep the main proof.</p>

<p>Where feasible, run the regression against the faulty implementation and retain the meaningful failure. A setup error or missing new method is not evidence that the old behaviour was caught. If the new interface cannot run unchanged on the old implementation, use a suitable negative control, such as removing the essential guard in a disposable candidate, and explain its limits.</p>

<p><strong>Exit check:</strong> the test fails for the target defect, or the missing reproduction evidence is explicitly recorded and resolved before acceptance.</p>

<h2 id="4-implement-the-smallest-coherent-fix">4. Implement the smallest coherent fix</h2>

<p>Ask the agent to change the rule and its tests, then return the diff, exact commands, observed results, and unanswered questions. Do not ask it to clean up the surrounding subsystem at the same time.</p>

<p>For ordering, check the actual atomicity boundary. A comparison in application code followed by an unconditional write leaves a window. A transaction alone is not an explanation of why the stale write is prevented. PostgreSQL’s <a href="https://www.postgresql.org/docs/current/transaction-iso.html">isolation documentation</a> describes how visibility and concurrent updates differ by isolation level. Verify the version and configuration the application uses.</p>

<p>The correct mechanism might involve a conditional update, locking, a constraint, or another design. The reviewer needs to know why it protects the invariant across every writer. If an external report can still arrive out of order after the database is correct, the task is not finished at the first boundary.</p>

<p>Run focused tests first, then the repository’s applicable broader checks. Preserve failed results and their causes. Do not delete assertions, increase timeouts, change snapshots, or narrow CI merely to make the run green. A legitimate change to those protections needs explanation and risk reassessment.</p>

<p><strong>Exit check:</strong> a stable candidate satisfies the local checks without silently changing the acceptance contract.</p>

<h2 id="5-review-through-six-gates">5. Review through six gates</h2>

<p>Give an independent reviewer, human or agent, the original brief, accepted decisions, base and candidate SHAs, complete relevant diff, surrounding code, and actual test evidence. Do not lead with the implementer’s conclusion that the fix is correct. The reviewer may need unchanged callers and test setup, not just changed lines.</p>

<p>Google’s <a href="https://google.github.io/eng-practices/review/reviewer/looking-for.html">review guidance</a> asks whether the intended behaviour is good for users and whether tests would fail when the code breaks. I use the following six gates in the handoff.</p>

<h3 id="requirement-and-design">Requirement and design</h3>

<p><strong>1. Is the requirement right?</strong> Match behaviour to the accepted source. In the example, confirm revision authority and attempt boundaries. Do not approve an internally consistent implementation of the wrong ordering rule.</p>

<p><strong>2. Is the design sound?</strong> Inspect ownership, state transitions, atomicity, compatibility, permissions and data exposure, failure windows, and recovery. Ask what happens if the process stops after the database commit but before the external report.</p>

<h3 id="implementation-and-test-protection">Implementation and test protection</h3>

<p><strong>3. Is the implementation correct?</strong> Trace every writer and reader of the protected state, including callbacks, jobs, repair scripts, permissions, and cross-component contracts. A small diff can bypass a large amount of behaviour.</p>

<p><strong>4. Did test protection weaken?</strong> Inspect deleted or undiscovered tests, weaker assertions, skips, excessive mocks, snapshot updates, coverage exclusions, swallowed failures, and CI filters or matrix reductions. Fewer tests can be legitimate. Unexplained loss of a required behaviour is not.</p>

<p>For each removal, record what it protected, why it is no longer needed, and the replacement evidence. If the requirement removed the behaviour, link that decision.</p>

<h3 id="coverage-and-historical-risk">Coverage and historical risk</h3>

<p><strong>5. Do the tests cover the behaviour and detect its failure?</strong> Map acceptance criteria to assertions, including rejected input, boundaries, duplicate work, committed state, and integration boundaries. Coverage percentages can identify gaps; executing a line does not show that the assertion checks the right result.</p>

<p>For important guards, try a bounded mutation: remove the revision comparison or isolation condition and see whether tests fail meaningfully. <a href="https://testing.googleblog.com/2021/04/mutation-testing.html">Google’s mutation-testing discussion</a> also explains the cost of unproductive mutants. This is a targeted check, not a demand for a perfect mutation score.</p>

<p><strong>6. Does the change reopen an old failure mode?</strong> Search relevant incident notes, fixes, reverts, flaky-test explanations, and compatibility history. Record the scope searched. No search hits means no evidence found in that scope, not proof that no history exists.</p>

<p>Two patterns recur across design, coverage, and historical review. For time-sensitive changes, distinguish calendar boundaries and time zones from clock jumps, delayed jobs, and paused processes. Freezing application time does not control an independent database or remote clock. For concurrent changes, write the harmful schedule and test its final state. Repeating a test until it happens to pass is neither analysis nor recovery.</p>

<p><strong>Exit check:</strong> confirmed defects have a location, trigger, consequence, and evidence. Hypotheses and unverified gaps stay visible rather than becoming invented bugs or silent passes.</p>

<h2 id="6-route-rework-then-review-the-new-candidate">6. Route rework, then review the new candidate</h2>

<p>A code defect goes back with its reproduction. An ambiguous requirement goes to the decision owner. An unavailable service is an environment blocker, not an invitation to rewrite the application.</p>

<p>For an initial pilot, I would allow at most two consecutive automated rework attempts before a human reassesses the approach. That is a trial budget, not a proven optimum. A request for broader permission stops the current run regardless of attempts remaining.</p>

<p>Keep the reviewer read-only. If it discovers a fix, return the finding to implementation rather than quietly repairing and approving its own candidate. Once the writer exits, rerun affected checks and review the new SHA. Record the actual tested object, including a synthetic merge commit when CI uses one. Reconcile relevant base changes before acceptance.</p>

<p><strong>Exit check:</strong> review and evidence apply to the candidate being considered, not an earlier version that happened to pass.</p>

<h2 id="7-hand-off-an-inspectable-pr-not-a-transcript">7. Hand off an inspectable PR, not a transcript</h2>

<p>Use a short closing record:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
</pre></td><td class="rouge-code"><pre>Decision: Ready for human review / changes required / insufficient evidence
Candidate: Repository, base SHA, head SHA, tested merge SHA if applicable
Acceptance: Each criterion -&gt; implementation -&gt; assertion or check
Review: Six gates, confirmed findings, unresolved decisions
Execution: Exact commands, environment, exit status, controlled log references
Protection: Removed or weakened tests and their justification
Recovery: Code rollback, data repair, compensation, or forward fix as needed
Next action: Owner, permitted action, target, and stop condition
</pre></td></tr></tbody></table></code></pre></div></div>

<p>Keep raw logs and private material out of the PR unless they are necessary and safe. A redacted summary with access-controlled references is usually more useful than a session dump.</p>

<p>Check the actual required jobs, not only the green summary. GitHub <a href="https://docs.github.com/en/pull-requests/how-tos/merge-and-close-pull-requests/troubleshooting-required-status-checks">documents</a> that some skipped or neutral results can satisfy required checks, while an entirely filtered workflow may remain pending. A final gate should establish that the tests required for this change really ran, or have a valid, explicit non-applicability decision. Do not expose privileged credentials to untrusted PR code.</p>

<p>Review completion does not authorise push, merge, or deployment. Follow the approval contract for each action. After an authorised external write, read back the exact target. If the response is lost, reconcile before repeating it. Generating a request ID locally does not make the remote operation idempotent unless the service honours that ID.</p>

<p>If release is in scope, verify the delivered behaviour in the permitted target environment and link the observed result. If release is not authorised, leave a reviewed PR and state that limit. “Ready for review” and “released” are different outcomes.</p>

<p><strong>Exit check:</strong> the closing record identifies the tested candidate, maps each acceptance criterion to completed verification, names unresolved gaps, and states the next permitted action. Unresolved critical evidence means insufficient evidence, not ready for review.</p>

<h2 id="try-the-runbook-before-automating-it">Try the runbook before automating it</h2>

<p>Start with one task and one reviewer, not a dashboard or a fleet. Exercise a known defect, a required check failure, and a candidate change after review in a disposable environment. Confirm that each is caught or stops the workflow where intended.</p>

<p>Record missed defects, false alarms, human review and repair time, elapsed time, tool cost, and abandoned attempts. Without a comparable baseline, the first task establishes one. A successful run does not prove a productivity gain or a lower defect rate.</p>

<p>I would automate a check after I can explain what evidence it produces and what it refuses to accept. The useful outcome is not an agent that keeps working. It is a change whose purpose, protection, and remaining risks another person can inspect.</p>]]></content><author><name>joey wang</name></author><category term="ai" /><category term="engineering" /><category term="ai" /><category term="agents" /><category term="testing" /><category term="git" /><category term="automation" /><summary type="html"><![CDATA[A practical AI agent development runbook covering task scope, isolated execution, six review gates, regression tests, bounded rework, and release evidence.]]></summary></entry><entry><title type="html">Agentic SDLC: Approval, Recovery, and Evidence Contracts</title><link href="https://joeywang.github.io/posts/agentic-sdlc-approval-recovery-evidence/" rel="alternate" type="text/html" title="Agentic SDLC: Approval, Recovery, and Evidence Contracts" /><published>2026-10-23T21:30:00+00:00</published><updated>2026-10-23T21:30:00+00:00</updated><id>https://joeywang.github.io/posts/agentic-sdlc-approval-recovery-evidence</id><content type="html" xml:base="https://joeywang.github.io/posts/agentic-sdlc-approval-recovery-evidence/"><![CDATA[<p>An approval can be genuine and still apply to the wrong code. A test can pass and still stop at a mocked boundary. An agent can lose its response to an external write even though the write succeeded.</p>

<p>These are the cases I want an agentic development workflow to handle before I give it more autonomy. A longer planning document does not resolve them.</p>

<p>My earlier <a href="/posts/software-factory-evidence-first/">software factory design</a> described the division of work between people, agents, reusable tasks, and individual runs. This article narrows that design to three contracts: what an approval authorises, what recovery must reconcile, and what counts as evidence.</p>

<p>The starting point was a document review, followed by an independent second-model review. That found design questions, not production results. The mechanisms below are proposals for a bounded pilot, not claims that I have deployed a complete enforcement system or measured a productivity gain.</p>

<h2 id="separate-intent-from-authority">Separate intent from authority</h2>

<p>A lifecycle diagram usually looks reassuring: plan, implement, test, review, deploy. The arrows hide a question: does reaching the next state grant permission to perform its action?</p>

<p>It should not.</p>

<p>I can authorise an agent to implement a fix without authorising it to push a branch. I can approve a merge without approving a production deployment. A workflow needs to represent those distinctions even if one person makes every decision.</p>

<p>Anthropic’s <a href="https://claude.com/blog/the-ai-native-sdlc-playbook">AI-native SDLC playbook</a> proposes a connected lifecycle built around committed artifacts and feedback. I find that useful as an organising model. Its claims about throughput are not evidence that my own workflow is faster, and I would not turn every small fix into a separate intent, specification, plan, and decision document.</p>

<p>For a small, reversible change, I would start with one record:</p>

<ul>
  <li><strong>Brief:</strong> the user problem, scope, and non-goals.</li>
  <li><strong>Contract:</strong> acceptance criteria, permitted actions, risk boundaries, and stop conditions.</li>
  <li><strong>Evidence:</strong> the candidate version, observed results, independent review, and outstanding decisions.</li>
</ul>

<p>These can be sections of a ticket. A schema migration or security-sensitive change needs more detail. Calling it a chore does not lower its risk.</p>

<h2 id="bind-approval-to-the-candidate-action-and-target">Bind approval to the candidate, action, and target</h2>

<p>Consider an illustrative sequence. A reviewer approves candidate A. The agent adds a follow-up commit, producing candidate B. The dashboard still shows a green check, and the conversation still contains “approved”.</p>

<p>Nothing has necessarily gone wrong with the people. The record has lost the relationship between the decision and its subject.</p>

<p>An approval should identify the work, the exact candidate, the action, and the target. An evidence record should identify what was actually tested. The following YAML is a proposed shape, not an executable configuration. Angle-bracket values are placeholders, not real identifiers:</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
</pre></td><td class="rouge-code"><pre><span class="na">approval</span><span class="pi">:</span>
  <span class="na">task_id</span><span class="pi">:</span> <span class="s2">"</span><span class="s">&lt;task</span><span class="nv"> </span><span class="s">identifier&gt;"</span>
  <span class="na">repository</span><span class="pi">:</span> <span class="s2">"</span><span class="s">&lt;repository</span><span class="nv"> </span><span class="s">identifier&gt;"</span>
  <span class="na">candidate_sha</span><span class="pi">:</span> <span class="s2">"</span><span class="s">&lt;full</span><span class="nv"> </span><span class="s">commit</span><span class="nv"> </span><span class="s">object</span><span class="nv"> </span><span class="s">ID&gt;"</span>
  <span class="na">spec_version</span><span class="pi">:</span> <span class="s2">"</span><span class="s">&lt;versioned</span><span class="nv"> </span><span class="s">acceptance</span><span class="nv"> </span><span class="s">contract&gt;"</span>
  <span class="na">action</span><span class="pi">:</span> <span class="s2">"</span><span class="s">merge"</span>
  <span class="na">target</span><span class="pi">:</span> <span class="s2">"</span><span class="s">&lt;target</span><span class="nv"> </span><span class="s">branch&gt;"</span>
  <span class="na">expected_target_sha</span><span class="pi">:</span> <span class="s2">"</span><span class="s">&lt;full</span><span class="nv"> </span><span class="s">target</span><span class="nv"> </span><span class="s">object</span><span class="nv"> </span><span class="s">ID&gt;"</span>
  <span class="na">approver_ref</span><span class="pi">:</span> <span class="s2">"</span><span class="s">&lt;authenticated</span><span class="nv"> </span><span class="s">human</span><span class="nv"> </span><span class="s">approval</span><span class="nv"> </span><span class="s">reference&gt;"</span>
  <span class="na">evidence_refs</span><span class="pi">:</span>
    <span class="pi">-</span> <span class="s2">"</span><span class="s">&lt;immutable</span><span class="nv"> </span><span class="s">evidence</span><span class="nv"> </span><span class="s">record&gt;"</span>

<span class="na">evidence</span><span class="pi">:</span>
  <span class="na">tested_sha</span><span class="pi">:</span> <span class="s2">"</span><span class="s">&lt;full</span><span class="nv"> </span><span class="s">commit</span><span class="nv"> </span><span class="s">object</span><span class="nv"> </span><span class="s">ID&gt;"</span>
  <span class="na">environment_ref</span><span class="pi">:</span> <span class="s2">"</span><span class="s">&lt;versioned</span><span class="nv"> </span><span class="s">test</span><span class="nv"> </span><span class="s">environment&gt;"</span>
  <span class="na">config_ref</span><span class="pi">:</span> <span class="s2">"</span><span class="s">&lt;relevant</span><span class="nv"> </span><span class="s">configuration</span><span class="nv"> </span><span class="s">version&gt;"</span>
  <span class="na">command_ref</span><span class="pi">:</span> <span class="s2">"</span><span class="s">&lt;test</span><span class="nv"> </span><span class="s">invocation&gt;"</span>
  <span class="na">result</span><span class="pi">:</span> <span class="s2">"</span><span class="s">passed"</span>
  <span class="na">artifact_ref</span><span class="pi">:</span> <span class="s2">"</span><span class="s">&lt;access-controlled</span><span class="nv"> </span><span class="s">output</span><span class="nv"> </span><span class="s">reference&gt;"</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>A deployment needs a deployment-specific record, including the artifact digest and destination. A merge approval is not a reusable deployment credential.</p>

<p>The equality checks are only the beginning. A field called <code class="language-plaintext highlighter-rouge">approver_ref</code> proves nothing if the executing agent can write it. The execution boundary must authenticate the approval through a record or service that the worker cannot impersonate. The same applies to evidence: a worker-authored <code class="language-plaintext highlighter-rouge">result: passed</code> is not an independent test result.</p>

<p>I would invalidate the affected approval or evidence when its candidate, acceptance contract, or relevant target context changes. That does not mean reapproving the entire task after an unrelated spelling correction. It means defining which changes affect which claims.</p>

<p>The tested object also needs care. A CI run against a synthetic merge candidate is not automatically a run against the PR head, and neither is automatically proof about the eventual deployed artifact. Record the actual object and its relationship to the candidate rather than relabelling it for convenience.</p>

<pre><code class="language-mermaid">sequenceDiagram
    participant H as Human
    participant G as Trusted gate
    participant E as Evidence store
    participant T as Target
    H-&gt;&gt;G: Approve candidate A, action and target
    G-&gt;&gt;E: Read authenticated approval and applicable evidence
    alt Candidate or relevant context changed
        E--&gt;&gt;G: Mismatch or stale evidence
        G--&gt;&gt;H: Refuse and request refreshed review
    else Candidate and context match
        E--&gt;&gt;G: Evidence for candidate A
        G-&gt;&gt;T: Conditional write against expected target
        alt Target moved
            T--&gt;&gt;G: Reject stale precondition
            G--&gt;&gt;H: Reconcile before proceeding
        else Preconditions hold
            T--&gt;&gt;G: Operation reference
            G-&gt;&gt;T: Read back exact target
            T--&gt;&gt;G: Observed state
            G--&gt;&gt;H: Report result or uncertainty
        end
    end
</code></pre>

<p>Checking the branch and then writing later leaves a race. The final operation needs a service-supported precondition or an equivalently protected executor. If the target cannot provide that guarantee, I would not pretend the diagram provides it.</p>

<p>A Git worktree helps keep unrelated file changes apart. It does not confine credentials, network access, or the authority to merge. Those require a separate security boundary.</p>

<h2 id="recover-by-reconciling-not-replaying">Recover by reconciling, not replaying</h2>

<p>A useful handoff records the branch, candidate, assumptions, completed checks, and next action. That is enough to resume many read-only steps. It is not enough to resume an uncertain external write.</p>

<p>Suppose an agent submits an operation and loses the connection before recording the response. There are three states worth distinguishing:</p>

<ol>
  <li>The operation is confirmed successful.</li>
  <li>It is confirmed not to have executed.</li>
  <li>Its outcome is unknown.</li>
</ol>

<p>A timeout belongs in the third category. A temporarily empty lookup may belong there too, especially when the remote system is eventually consistent.</p>

<p>Before an authorised write, I would persist an operation record containing a stable operation ID, the exact target, the request identity, and the applicable approval. If that record cannot be saved, the write should not proceed. Afterward, the record gets the remote identifier and exact-target readback.</p>

<p>On recovery, the first action is a query, not a repeat of the previous command.</p>

<pre><code class="language-mermaid">flowchart TD
    A["Read durable intent and approval"] --&gt; B["Query exact remote operation and target"]
    B --&gt; C{"Known outcome?"}
    C --&gt;|"Confirmed success"| D["Record evidence; do not repeat"]
    C --&gt;|"Confirmed not executed"| E{"Authorised and safe to retry?"}
    E --&gt;|"Yes, within budget"| F["Retry under API contract"]
    E --&gt;|"No"| H["Stop for human reconciliation"]
    C --&gt;|"Unknown"| G{"Server-side idempotency sufficient?"}
    G --&gt;|"Yes, same request and valid approval"| F
    G --&gt;|"No or uncertain"| H
    F --&gt; B
    D --&gt; I["Update handoff and continue"]
</code></pre>

<p>A local operation ID does not make an external API idempotent. The service must support the relevant contract, including what happens when the same identifier is reused with different parameters or outside its retention window. Amazon’s <a href="https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/">Making retries safe with idempotent APIs</a> explains this distinction through caller-provided request identifiers and reconciliation.</p>

<p>Read-only checks can usually have bounded retries. An in-scope test failure can enter a bounded repair loop. An unknown side effect, changed permission boundary, or exhausted budget should stop the run. “Ask a human after every failure” is too expensive; “retry until green” is too permissive.</p>

<h2 id="make-evidence-narrower-and-stronger">Make evidence narrower and stronger</h2>

<p>The question is not whether another agent agrees with the implementation. It is whether the acceptance claim survives a check grounded outside the implementation’s own explanation.</p>

<p>For an illustrative learner-progress fix, I would distinguish these claims:</p>

<ul>
  <li>A unit test proves the local update rule for the supplied cases.</li>
  <li>A contract test checks the payload against the agreed interface.</li>
  <li>A browser test with a mocked reporting service checks the UI and request path up to that mock.</li>
  <li>A controlled integration test with the real service checks a different boundary.</li>
</ul>

<p>Those are useful but different results. Calling the browser test “E2E” does not extend its evidence beyond the mock.</p>

<p>The verifier should start with the original symptom and acceptance criteria, then inspect the necessary code, tests, and contracts. I would avoid leading with the implementer’s conclusion. I would not prevent the verifier from reading relevant context in the name of independence.</p>

<p>A second model is helpful for finding missing assumptions. It cannot turn a document review into runtime verification, and two models can share the same mistaken premise. Useful independent oracles include an agreed interface contract, a preserved regression case, or a known expected result supplied independently of the proposed fix.</p>

<p>Humans still need to inspect risk-sensitive diffs. Tests do not remove the need to review a permission change or a destructive migration. The aim is to make that review focused and well supported, not to declare code review obsolete.</p>

<h2 id="keep-the-audit-trail-smaller-than-the-transcript">Keep the audit trail smaller than the transcript</h2>

<p>I want to reconstruct why an action happened, what authorised it, and what was observed. I do not need every byte the agent saw.</p>

<p>Full session transcripts and command outputs can contain credentials, personal data, request headers, or customer content. Automatically committing them creates another system to secure and eventually clean up.</p>

<p>My default would be an allowlisted, redacted summary with controlled evidence references. Raw artifacts, when needed, belong in restricted storage with an owner, access policy, retention period, and deletion process. A path and a checksum do not protect the referenced file. Secret scanning is useful, but not a substitute for deciding what should be collected.</p>

<p>This also keeps the sources of truth clearer: Git owns code versions, the test system owns its observed result, the approval boundary owns the decision, and the task record links them. A knowledge index is a retrieval aid, not the authority to act.</p>

<h2 id="a-reversible-slice-still-needs-a-recovery-plan">A reversible slice still needs a recovery plan</h2>

<p>A thin vertical slice is a good way to learn quickly. It is not proof that everything it does can be undone.</p>

<p><a href="https://git-scm.com/docs/git-revert">Git revert</a> records changes that reverse earlier commits. It does not retract a notification, restore an external account, or repair data already consumed downstream.</p>

<p>For a consequential slice, I would name the recovery mechanism: code rollback, configuration disable, data restoration, compensating action, or forward repair. I would also identify the point after which an ordinary code rollback is insufficient.</p>

<p>Temporary compatibility layers need an exit condition. An expand-migrate-contract change may deliberately leave old and new paths in place. That is sensible only if someone owns the removal condition and review point. The cleanup can be a later approved slice; it should not be forced into the first change merely to make one PR look complete.</p>

<h2 id="the-pilot-i-would-run-next">The pilot I would run next</h2>

<p>I would use one familiar repository, one real reversible regression, and one implementation stream. No infrastructure changes, data migration, or automatic production release. One task record should be enough.</p>

<p>Before starting, I would agree the implementation scope, acceptance oracle, human review budget, and stop conditions. Then I would reproduce the failure, make a meaningful red test green, run the focused suite and critical-boundary smoke, and ask for independent verification at the exact candidate.</p>

<p>Two failure exercises matter as much as the happy path: reject evidence from an old candidate, and resume safely after an interrupted read-only step. Any later test of uncertain writes should start with a controlled fixture, not a production experiment.</p>

<p>I would record human review time, elapsed time, rework, acceptance coverage, and external mutations. If there is no comparable historical baseline, the first task establishes one. A single successful run cannot demonstrate a lower defect rate or a reliable productivity improvement.</p>

<p>I would stop when the task crosses an unapproved security, data, or API boundary, when evidence cannot be reproduced, or when the agreed budget runs out. The next approval must come from the person who owns the decision, not from the agent’s confidence that another attempt will work.</p>

<p>The useful unit of progress is an accepted outcome with applicable evidence. More agent activity is easy to generate. Authority that survives code changes, recovery that survives lost responses, and evidence that answers the original question are harder. Those are the parts I would prove first.</p>]]></content><author><name>Joey Wang</name></author><category term="ai" /><category term="engineering" /><category term="ai" /><category term="agents" /><category term="testing" /><category term="automation" /><category term="security" /><summary type="html"><![CDATA[An agentic SDLC needs version-bound approvals, safe recovery after uncertain writes, and independent evidence, without turning small changes into paperwork.]]></summary></entry><entry><title type="html">我想建的软件工厂：从需求到可验证交付</title><link href="https://joeywang.github.io/posts/software-factory-evidence-first-zh/" rel="alternate" type="text/html" title="我想建的软件工厂：从需求到可验证交付" /><published>2026-10-23T07:00:00+00:00</published><updated>2026-10-23T07:00:00+00:00</updated><id>https://joeywang.github.io/posts/software-factory-evidence-first-zh</id><content type="html" xml:base="https://joeywang.github.io/posts/software-factory-evidence-first-zh/"><![CDATA[<div class="audio-summary">
  <p><strong>语音摘要：使用本地语音合成生成的中文简要讲解。</strong></p>
  <audio controls="" preload="metadata">
    <source src="/assets/audio/2026-10-23-software-factory-evidence-first-zh-summary.ogg" type="audio/ogg" />
    您的浏览器不支持音频播放。
  </audio>
</div>

<p><a href="/posts/software-factory-evidence-first/">Read in English</a></p>

<p>在<a href="/posts/ai-native-one-person-company-operating-system/">之前的一人公司设计</a>里，我把问题归结为协调：客户反馈、产品决策、代码、运行故障和下一步工作，最终都要回到同一个人的注意力里。</p>

<p>软件工厂是其中更具体的一段：一个已经值得做的问题，怎样变成可以检查、可以接手、必要时可以撤回的软件变更？</p>

<p>我的起点很实际。让一个 coding agent 改代码并不难。难的是让下一次执行知道上一次做到哪里，让审查者看到原始需求而不只是实现者的总结，让失败停在正确的位置，而不是靠更多提示词把流程往前推。</p>

<p>这篇文章整理的是我正在推敲的设计。它还不是一个经过端到端验证、可以无人值守运行的交付平台；文中的风险分级、返工上限和验收规则，都需要通过小规模试点检验。</p>

<h2 id="软件工厂可以从一个很小的循环开始">软件工厂可以从一个很小的循环开始</h2>

<p>Matt Pocock 的 <a href="https://github.com/mattpocock/dictionary-of-ai-coding#software-factory">Dictionary of AI Coding</a> 用一个朴素的定义描述 software factory：通过触发事件自动启动 agent 会话。</p>

<p>按我的理解，一个定时任务，每次处理一种小型维护工作，留下一个等待人工审查的 PR，就已经符合这个定义。工厂不要求人退出所有决策。</p>

<p>这与 <a href="https://github.com/mattpocock/dictionary-of-ai-coding#dark-factory">dark factory</a> 有区别。按照同一份词典，后者指的是某个代码范围内，人不再阅读工厂生成的变更。依此理解，晚上无人看守地执行、早上由人审 PR，仍然是有人审查的软件工厂。</p>

<p>我目前想验证的是前一种模式。自动化负责准备和执行一部分工作，人保留产品选择、关键授权和发布责任。</p>

<h2 id="先把一次请求和一次执行分开">先把一次请求和一次执行分开</h2>

<p>设计里最容易混淆的是“任务”。它可能指一张工单，也可能指实现、审查这样的工作类型，还可能指某个 agent 正在进行的一次尝试。</p>

<p>下面是我对 <a href="https://github.com/mattpocock/skills/pull/1316">Matt Pocock 软件工厂术语草案</a>的改编，用于表达自己的交付约束，不是对原词表的逐项引用。</p>

<ul>
  <li><strong>工单（work item / ticket）</strong>：一次具体请求，保留问题来源、目标和验收条件。</li>
  <li><strong>Task</strong>：可重复使用的工作类型，例如调查、实现、审查和返工。</li>
  <li><strong>Run</strong>：执行某个 Task 的一次尝试，有明确的开始和结束。</li>
  <li><strong>Trigger</strong>：启动 Run 的事件，例如工单被确认 ready、审查失败或人工补全信息。</li>
  <li><strong>Runner</strong>：承载执行的环境，可以是本地进程、工作节点或可丢弃的云机器。</li>
</ul>

<p>因此，一张工单可以经历调查、实现、审查和返工等多个 Run。它的持久状态不应该只存在于其中某次聊天里。</p>

<p>需要人回答问题时，也不必让一个 Run 一直等待。它应该记录问题和已有证据，结束执行，把工单停在“待澄清”或“待决策”。人工更新后，新的事件再启动下一次 Run。</p>

<p>这使恢复不必依赖某个会话仍然活着。</p>

<h2 id="控制平面协调工作事实留在原来的系统">控制平面协调工作，事实留在原来的系统</h2>

<p>我不打算先建一个包办所有事情的新平台。</p>

<p>Git 已经能保存代码历史，CI 能记录检查结果，工单系统能承载需求和责任归属，可观测性系统能报告运行问题。重新复制这些数据，很容易得到几份互相矛盾的“最新状态”。</p>

<p>在这个设计里，Hermes 是协调和审批入口，连接已有系统：</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
</pre></td><td class="rouge-code"><pre>问题来源 / 客户反馈 / 运行信号
               ↓
     工单：请求、范围、验收条件
               ↓
     控制平面：路由、权限、预算
               ↓
       隔离执行 → 候选代码
               ↓
       自动检查 → 独立审查
               ↓
         人的决策与发布授权
               ↓
       正常发布流程 → 结果验证
</pre></td></tr></tbody></table></code></pre></div></div>

<p>工单负责说明要解决什么；Git 负责说明代码是什么；CI 负责说明某个版本跑过哪些检查。Company OS 保存经过审阅的设计和决策，个人知识系统则保持自己的隐私边界。</p>

<p>控制平面记录这些对象的关联与执行历史，但不替它们宣布事实。一个 agent 说“测试通过了”，仍然需要对应的命令、退出状态和结果；说“已经发布”，也需要发布系统和运行环境的证据。</p>

<p>只有当现有系统之间确实缺少某项能力时，才值得增加新的组件。</p>

<h2 id="交接的是证据不只是一段总结">交接的是证据，不只是一段总结</h2>

<p>我希望一张工单能串起下面这些问题：</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
</pre></td><td class="rouge-code"><pre>request：为什么做？要达到什么结果？
scope：允许改什么？明确不做什么？
plan：怎样实现、检查和恢复？
decision：有哪些重要取舍，由谁决定？
change：实际改了什么，执行结果是什么？
audit：这个候选版本是否满足原请求？
debt：哪些剩余风险被明确接受？
</pre></td></tr></tbody></table></code></pre></div></div>

<p>这不意味着每次修改都要创建七个文件。小型文档变更可以把请求留在工单里，把变更和检查留在 PR 里。涉及接口、数据语义或重要业务规则时，才需要更完整的记录。</p>

<p>这些记录应该通过稳定工单 ID 和候选 commit SHA 关联，而不是把同一段描述复制到多个地方。</p>

<p>尤其是审查意见，必须说明它针对哪个版本。审查通过后又改了代码，旧意见就不能直接当作新候选的通行证。试点阶段，我倾向于每个新 SHA 都重新审查；将来是否允许对内容完全相同的 rebase 沿用结论，也应由明确的验证规则决定。</p>

<p>失败结果同样需要保留。最后一版测试通过，不应抹掉此前的失败、原因和修正过程。</p>

<h2 id="风险决定审查强度而不是决定用哪个更聪明的模型">风险决定审查强度，而不是决定用哪个“更聪明”的模型</h2>

<p>我把三个决定分开处理：风险有多高、谁来执行、谁有权验收。</p>

<p>模型名称只能影响执行者选择。它不能扩大权限，也不能替代验收。</p>

<p>当前设计用四档风险组织检查：</p>

<ul>
  <li><strong>L0</strong>：纯文档或格式变更，确认范围、链接和格式，不涉及敏感政策或可执行行为。</li>
  <li><strong>L1</strong>：局部、可逆的代码变更，需要验收条件对应的测试，以及未参与实现的独立审查 Run。</li>
  <li><strong>L2</strong>：跨组件、接口、数据语义或有实质影响的依赖与配置变更，需要影响分析、更广的回归证据和恢复方案；产品或数据契约取舍交给人决定。</li>
  <li><strong>L3</strong>：安全、权限、隐私、生产或客户数据，以及破坏性或难以逆转的操作，需要明确的人类授权、相应责任人审查和操作后的读回验证。</li>
</ul>

<p>工作类型提供默认清单，风险标记只能提高要求。“chore”不是低风险证明，“bug fix”也不能掩盖认证逻辑的变化。一般风险不明时至少按 L2 处理，数据分类不明或触及生产则按 L3 处理。</p>

<p>分级也不只发生在入口。实现中发现要扩大范围、增加依赖、修改测试预期、跳过测试、更新快照或放宽超时，都应该提出范围问题并重新评估，而不是由实现者为了完成任务自行决定。</p>

<p>工厂本身的控制规则也属于被保护的代码。修改触发器、审查规则或权限边界，至少按 L2 对待；削弱关键制动措施时按 L3 处理。</p>

<h2 id="独立审查需要一个明确的候选对象">独立审查需要一个明确的候选对象</h2>

<p>独立审查不能只是让模型再读一遍自己的总结。</p>

<p>我希望审查者拿到原始请求、验收条件、base SHA、candidate SHA、完整 diff 和实际检查结果。审查 Run 不参与实现，也不能一边修代码一边批准自己。</p>

<p>更换一个模型不自动产生独立性。审查者如果只看到实现者筛选过的描述，仍可能继承同一套错误假设。</p>

<p>同样，流程界面里存在 review 状态，并不证明审查是硬门槛。需要验证状态转换和权限控制，确认执行者不能绕过必需审查直接宣布完成。Hermes 的 <a href="https://hermes-agent.nousresearch.com/docs/user-guide/features/kanban">Kanban 文档</a>和 <a href="https://hermes-agent.nousresearch.com/docs/user-guide/features/kanban-worker-lanes">worker lanes 文档</a>描述了卡片、运行记录、交接和完成契约等机制，但具体配置与执行语义仍需在试点版本上检查。</p>

<p>代码工作区的隔离也要说清楚。<a href="https://git-scm.com/docs/git-worktree">Git worktree</a> 可以避免两个执行者把文件修改混在一起，却不限制进程读取宿主机文件、凭据或访问网络。需要安全沙箱时，必须另外设置操作系统层面的文件、网络、凭据和资源约束。</p>

<p>人的批准也必须是一个可以核验的事件，来自自动化不能冒充的身份。由 agent 自己填一个 <code class="language-plaintext highlighter-rouge">approved: true</code> 字段，不能算人工授权。试点中，L2/L3 变更全部由人审阅，较低风险变更仍做抽查；其中 L3 的恢复或回滚路径也需要实际测试。</p>

<h2 id="返工应该回到能解决问题的那一步">返工应该回到能解决问题的那一步</h2>

<p>一个流程失败后，最省事的实现往往是重新启动整个 agent。这样做容易反复调查已经知道的事实，也可能把权限故障误当成代码问题。</p>

<p>我希望每条反馈至少包含：问题类型、期望与实际、证据、严重程度、负责人和下一步。</p>

<p>例如，缺少验收条件应交回需求负责人；产品语义有冲突应交回决策人；测试失败应把最小复现、失败命令和结果交给实现者。外部服务不可用则暂停并记录恢复条件，不应靠继续改代码来消耗重试。</p>

<p>对于代码审查发现的问题，最短路径通常是：</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
</pre></td><td class="rouge-code"><pre>实现 → 检查 → 审查
               ↓ 发现缺口
             定向返工
               ↓
       新候选 → 重新检查与审查
</pre></td></tr></tbody></table></code></pre></div></div>

<p>试点暂定每条触发链最多连续进行两次自动返工，然后交给人重新判断范围或方案。两次只是一个待测的运行约束，不是普遍最优值。基础设施重试计数，也不能默认等同于审查返工计数。</p>

<p>未满足的验收条件不能被改名为“技术债”后关闭。技术债只容纳经过明确接受、有负责人和复查条件的剩余风险。</p>

<h2 id="触发与恢复要按会重复会中断来设计">触发与恢复要按会重复、会中断来设计</h2>

<p>每增加一个触发源，都要回答几个具体问题：</p>

<p>谁启动执行？它可以读写哪些系统？结束后触发什么？人在哪一步重新进入？重复事件如何避免创建重复工作？</p>

<p>例如，外部事件 ID 可以用于创建工单或卡片时的幂等键，但这不等于所有后续操作都已经幂等。创建分支、提交 PR、写入外部记录，都需要自己的去重和读回规则。</p>

<p>Runner 中断时，接手者至少要能从持久记录找到原始请求、当前候选版本、已经完成的外部操作、已有证据和阻塞原因。无法确认某次写入是否成功，就先查询目标状态，而不是直接再写一次。</p>

<p>这也是我把 Runner 看作可替换执行容量的原因。即使机器长期存在，恢复也不应依赖它的聊天上下文或唯一一份本地文件。</p>

<h2 id="第一轮试点先检查会不会停">第一轮试点先检查“会不会停”</h2>

<p>我不准备一开始就接入所有告警、所有仓库，再安排多个 agent 并行工作。对小团队而言，生成变更的速度很容易超过审查能力。</p>

<p>第一步应是手动选择一项非敏感、低风险、范围清楚的工作。在执行前固定验收条件，确认实现者和审查者的实际身份、权限与工作区，然后完整走一遍实现、检查、审查和人工决策。合并与部署仍是独立授权。</p>

<p>随后，在可丢弃环境中主动制造失败：加入一个已知缺陷，检查审查能否拦住；让必需检查失败，检查流程能否拒绝完成；在审查后修改候选，检查旧结论是否失效；重复发送事件，检查是否产生重复工作；让任务缺少必要信息，检查执行能否停止并准确提问。</p>

<p>这些测试通过，只能说明相应控制路径在测试条件下有效。它们不证明整个工厂可靠，更不证明它已经节省成本。</p>

<p>下一步才是少量真实任务，逐个执行，记录成功、失败、阻塞和放弃的案例。观察人工提示、审查、修复和维护的总投入，以及工具费用、漏检缺陷和最终结果。没有可比较的人工基线，就不声称节省了多少时间。</p>

<p>我希望这套设计最终减少的是上下文丢失、重复调查和无效返工。它是否值得继续建设，要看一个被接受的结果总共花了多少成本，以及它是否真的解决了值得解决的问题。</p>

<p>在得到这些证据以前，我宁愿先建设一个速度有限、停止行为经过测试的交付循环，也不急着把更多工作自动送进队列。</p>

<h2 id="参考资料">参考资料</h2>

<ol>
  <li><a href="https://github.com/mattpocock/dictionary-of-ai-coding#software-factory">Matt Pocock: Dictionary of AI Coding：Software factory 与 Dark factory</a></li>
  <li><a href="https://github.com/mattpocock/skills/pull/1316">Matt Pocock: software-factory 术语设计草案，PR #1316</a>。撰稿时该 PR 仍为 Open，并标记为 Draft；它提供设计语言，并非可直接运行的流水线。</li>
  <li><a href="https://hermes-agent.nousresearch.com/docs/user-guide/features/kanban">Hermes Agent: Kanban</a></li>
  <li><a href="https://hermes-agent.nousresearch.com/docs/user-guide/features/kanban-worker-lanes">Hermes Agent: Kanban worker lanes</a></li>
  <li><a href="https://git-scm.com/docs/git-worktree">Git 官方文档: git-worktree</a>：说明多个工作树的机制；工作树隔离不等于安全沙箱。</li>
</ol>

<p><em>本文进一步展开了<a href="/posts/ai-native-one-person-company-operating-system/">此前 Company OS 设计</a>中的交付部分。风险层级、证据结构和试点约束是作者提出的工作方案，不代表上述项目自带的完整保证，也不是已验证的行业标准。</em></p>]]></content><author><name>Joey Wang</name></author><category term="ai" /><category term="engineering" /><category term="ai" /><category term="agents" /><category term="hermes" /><category term="automation" /><category term="testing" /><summary type="html"><![CDATA[软件工厂不只是让智能体更快地写代码，而是把请求转化为可验证、可交接、可恢复的变更：本文梳理工单与执行的区别、持久证据、风险分级、独立审查和人工授权，讨论重复事件、执行中断与定向返工，并提出一个先测试失败时能否停止、再衡量真实交付成本的小规模试点方案，明确区分尚待检验的设计与已经验证的能力。]]></summary></entry><entry><title type="html">Designing a software factory for verifiable delivery</title><link href="https://joeywang.github.io/posts/software-factory-evidence-first/" rel="alternate" type="text/html" title="Designing a software factory for verifiable delivery" /><published>2026-10-23T07:00:00+00:00</published><updated>2026-10-23T07:00:00+00:00</updated><id>https://joeywang.github.io/posts/software-factory-evidence-first</id><content type="html" xml:base="https://joeywang.github.io/posts/software-factory-evidence-first/"><![CDATA[<div class="audio-summary">
  <p><strong>Audio summary: a short English summary, generated with local text-to-speech.</strong></p>
  <audio controls="" preload="metadata">
    <source src="/assets/audio/2026-10-23-software-factory-evidence-first-summary.ogg" type="audio/ogg" />
    Your browser does not support the audio element.
  </audio>
</div>

<p><a href="/posts/software-factory-evidence-first-zh/">中文版</a></p>

<p>In <a href="/posts/ai-native-one-person-company-operating-system/">my earlier design for a one-person company</a>, I kept coming back to coordination. Customer feedback, product decisions, code, operational failures and the next piece of work all compete for the same person’s attention.</p>

<p>The software factory covers a more specific part of that problem: once something is worth doing, how does it become a software change that someone can inspect, take over and, if necessary, roll back?</p>

<p>My starting point is practical. Getting a coding agent to change code is not especially difficult. The harder parts are letting the next execution pick up where the previous one stopped, giving reviewers the original request rather than just the implementer’s summary, and stopping failures in the right place instead of pushing the process forward with more prompts.</p>

<p>This article describes a design I am still working through. It is not an unattended delivery platform that I have validated end to end. The risk tiers, rework limits and acceptance rules below all need testing in a small pilot.</p>

<h2 id="a-software-factory-can-start-with-a-small-loop">A software factory can start with a small loop</h2>

<p>Matt Pocock’s <a href="https://github.com/mattpocock/dictionary-of-ai-coding#software-factory">Dictionary of AI Coding</a> gives software factory a simple definition: automation that starts agent sessions in response to triggers.</p>

<p>My reading of that definition is that a scheduled job handling one kind of small maintenance task and leaving a PR for human review already qualifies. A factory does not require people to leave every decision to automation.</p>

<p>That differs from a <a href="https://github.com/mattpocock/dictionary-of-ai-coding#dark-factory">dark factory</a>. In the same dictionary, a dark factory is an area of a codebase where humans no longer read the changes the factory produces. On that reading, unattended execution overnight followed by human PR review in the morning is still a human-reviewed software factory.</p>

<p>The first model is the one I want to test. Automation prepares and carries out some of the work. People retain responsibility for product choices, consequential approvals and releases.</p>

<h2 id="separate-a-request-from-an-execution">Separate a request from an execution</h2>

<p>The word “task” is easy to overload. It can mean a ticket, a reusable kind of work such as implementation or review, or the particular attempt an agent is making right now.</p>

<p>The distinctions below are my adaptation of <a href="https://github.com/mattpocock/skills/pull/1316">Matt Pocock’s draft software-factory vocabulary</a>, not a verbatim glossary. I use them to express my own delivery constraints.</p>

<ul>
  <li><strong>Work item / ticket</strong>: a specific request that preserves its source, goal and acceptance criteria.</li>
  <li><strong>Task</strong>: a reusable kind of work, such as investigation, implementation, review or rework.</li>
  <li><strong>Run</strong>: one attempt to execute a Task, with a clear beginning and end.</li>
  <li><strong>Trigger</strong>: an event that starts a Run, such as a ticket becoming ready, a failed review or a human supplying missing information.</li>
  <li><strong>Runner</strong>: the execution environment, which might be a local process, a worker node or a disposable cloud machine.</li>
</ul>

<p>A single ticket can therefore go through several Runs for investigation, implementation, review and rework. Its durable state should not live only in one of those conversations.</p>

<p>When a human needs to answer a question, the Run does not have to wait indefinitely. It should record the question and available evidence, end execution, and leave the ticket in a state such as “needs clarification” or “needs decision”. A later event starts a fresh Run after the human updates it.</p>

<p>Recovery then does not depend on keeping a particular session alive.</p>

<h2 id="the-control-plane-coordinates-work-existing-systems-retain-the-facts">The control plane coordinates work; existing systems retain the facts</h2>

<p>I do not want to start by building a new platform that does everything.</p>

<p>Git already records code history. CI records check results. An issue tracker holds requests and ownership. Observability systems report operational problems. Copying all of that data into another system can easily produce several contradictory versions of the “latest state”.</p>

<p>In this design, Hermes is the coordination and approval entry point that connects the existing systems:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
</pre></td><td class="rouge-code"><pre>Problem sources / customer feedback / operational signals
                          ↓
          Ticket: request, scope, acceptance criteria
                          ↓
          Control plane: routing, permissions, budget
                          ↓
          Isolated execution → candidate code
                          ↓
          Automated checks → independent review
                          ↓
          Human decisions and release authorization
                          ↓
          Normal release process → outcome verification
</pre></td></tr></tbody></table></code></pre></div></div>

<p>The ticket explains what needs solving. Git identifies the code. CI records which checks ran against a particular version. Company OS holds reviewed designs and decisions, while the personal knowledge system keeps its own privacy boundary.</p>

<p>The control plane records the relationships between these objects and the execution history. It does not declare facts on their behalf. An agent saying “the tests passed” still needs the corresponding command, exit status and results. Saying “it has been deployed” needs evidence from the deployment system and the running environment.</p>

<p>A new component is worth adding only when a necessary capability is genuinely missing between the existing systems.</p>

<h2 id="hand-over-evidence-not-just-a-summary">Hand over evidence, not just a summary</h2>

<p>I want a ticket to connect the answers to these questions:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
</pre></td><td class="rouge-code"><pre>request:  Why are we doing this? What outcome do we want?
scope:    What may change? What is explicitly out of scope?
plan:     How will we implement, check and recover?
decision: What consequential trade-offs exist, and who decides?
change:   What actually changed? What did execution return?
audit:    Does this candidate satisfy the original request?
debt:     Which remaining risks have been explicitly accepted?
</pre></td></tr></tbody></table></code></pre></div></div>

<p>This does not mean creating seven files for every change. For a small documentation update, the request can stay in the ticket, with the change and checks in the PR. Interface changes, data semantics and important business rules call for a fuller record.</p>

<p>These records should link through a stable ticket ID and candidate commit SHA, rather than repeat the same description in several places.</p>

<p>Review findings, in particular, must identify the version they apply to. If code changes after approval, the old review cannot automatically authorize the new candidate. During the pilot, I would require a fresh review for every new SHA. Any later policy for carrying a review across a content-identical rebase would need explicit verification rules.</p>

<p>Failed results need to survive too. A passing final test run should not erase earlier failures, their causes or the changes made to address them.</p>

<h2 id="risk-determines-review-requirements-not-which-model-seems-smarter">Risk determines review requirements, not which model seems smarter</h2>

<p>I treat risk classification, executor selection and acceptance authority as separate decisions.</p>

<p>A model name can influence who performs the work. It cannot expand permissions or replace acceptance checks.</p>

<p>The current design organizes checks into four risk tiers:</p>

<ul>
  <li><strong>L0</strong>: documentation or formatting only, with scope, link and format checks; no sensitive policy changes or executable behavior.</li>
  <li><strong>L1</strong>: localized, reversible code changes, with tests mapped to acceptance criteria and an independent review Run that did not implement the candidate.</li>
  <li><strong>L2</strong>: changes across components or interfaces, changes to data semantics, or dependency and configuration changes with meaningful impact. These need impact analysis, broader regression evidence and a recovery plan. Product or data-contract trade-offs go to a human decision-maker.</li>
  <li><strong>L3</strong>: security, permissions, privacy, production or customer data, and destructive or hard-to-reverse operations. These need explicit human authorization, review by the appropriate owner and post-action read-back verification.</li>
</ul>

<p>The work type supplies a default checklist; risk flags can only raise the requirements. A “chore” label is not evidence of low risk, and calling something a “bug fix” does not make an authentication change harmless. Unknown general risk defaults to at least L2. Unknown data classification or any production touch defaults to L3.</p>

<p>Classification also continues during execution. Discovering a need to expand scope, add a dependency, change test expectations, skip tests, update snapshots or increase timeouts should raise a scope question and trigger reassessment. The implementer should not make those decisions unilaterally just to finish the job.</p>

<p>The factory’s own controls are protected code too. Changes to triggers, review rules or permission boundaries are at least L2. Weakening a critical stopping mechanism makes them L3.</p>

<h2 id="independent-review-needs-an-explicit-candidate">Independent review needs an explicit candidate</h2>

<p>Independent review cannot mean asking a model to reread its own summary.</p>

<p>I want the reviewer to receive the original request, acceptance criteria, base SHA, candidate SHA, complete diff and actual check results. The review Run does not implement the change and cannot approve its own edits while repairing the code.</p>

<p>Switching models does not automatically create independence. If a reviewer sees only the implementer’s selected account of the work, it can inherit the same faulty assumptions.</p>

<p>Likewise, a review state in a workflow interface does not prove that review is an enforced gate. State transitions and permission controls need testing to establish that an executor cannot bypass a required review and declare completion. Hermes’s <a href="https://hermes-agent.nousresearch.com/docs/user-guide/features/kanban">Kanban documentation</a> and <a href="https://hermes-agent.nousresearch.com/docs/user-guide/features/kanban-worker-lanes">worker lanes documentation</a> describe mechanisms for cards, run history, handoffs and completion contracts, but their configuration and execution semantics still need checking in the version used for the pilot.</p>

<p>Workspace isolation needs a precise definition as well. A <a href="https://git-scm.com/docs/git-worktree">Git worktree</a> can keep two executors’ file edits apart. It does not restrict a process from reading host files or credentials, or accessing the network. A security sandbox requires separate operating-system-level controls over files, network access, credentials and resources.</p>

<p>Human approval must also be a verifiable event from an identity that automation cannot impersonate. An agent filling in an <code class="language-plaintext highlighter-rouge">approved: true</code> field does not count as human authorization. During the pilot, humans review every L2/L3 change and spot-check lower-risk changes. L3 recovery or rollback paths also need actual testing.</p>

<h2 id="route-rework-to-the-step-that-can-resolve-it">Route rework to the step that can resolve it</h2>

<p>After a workflow fails, the easiest implementation is often to restart the entire agent. That can repeat investigations into facts already known, or treat a permissions failure as a code problem.</p>

<p>I want each piece of feedback to include at least the problem type, expected and actual behavior, evidence, severity, owner and next action.</p>

<p>Missing acceptance criteria should go back to the request owner. Conflicting product semantics should go to the decision-maker. A test failure should return a minimal reproduction, the failing command and its results to the implementer. An unavailable external service calls for a pause and recorded recovery conditions, not more code changes to use up the retry budget.</p>

<p>For findings from a code review, the shortest route is usually:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
</pre></td><td class="rouge-code"><pre>Implementation → checks → review
                            ↓ finding
                       Targeted rework
                            ↓
              New candidate → fresh checks and review
</pre></td></tr></tbody></table></code></pre></div></div>

<p>The pilot provisionally allows at most two consecutive automated rework Runs per trigger chain before a human reassesses the scope or approach. Two is an operating constraint to test, not a universally optimal number. Infrastructure retry counts should not be assumed to measure review rework either.</p>

<p>An unmet acceptance criterion cannot be renamed “technical debt” so the ticket can close. Debt covers only explicitly accepted remaining risks with an owner and conditions for revisiting them.</p>

<h2 id="assume-triggers-will-repeat-and-execution-will-stop">Assume triggers will repeat and execution will stop</h2>

<p>Every new trigger source needs answers to a few specific questions:</p>

<p>Who starts execution? Which systems may it read and write? What does completion trigger next? Where does a human re-enter? How do duplicate events avoid creating duplicate work?</p>

<p>An external event ID, for example, can be an idempotency key when creating a ticket or card. That does not make every subsequent operation idempotent. Branch creation, PR creation and writes to external records each need their own deduplication and read-back rules.</p>

<p>If a Runner stops, whoever takes over should at least be able to recover the original request, current candidate, completed external actions, available evidence and blocking reason from durable records. If it is unclear whether a write succeeded, query the target state before writing again.</p>

<p>That is why I treat a Runner as replaceable execution capacity. Even when the machine persists, recovery should not depend on its conversation context or the only local copy of a file.</p>

<h2 id="the-first-pilot-should-test-whether-the-system-stops">The first pilot should test whether the system stops</h2>

<p>I do not plan to begin by connecting every alert and repository, then running several agents in parallel. For a small team, generating changes can easily outpace the capacity to review them.</p>

<p>The first step should be one manually selected, non-sensitive, low-risk piece of work with a clear scope. Fix the acceptance criteria before execution. Confirm the implementer’s and reviewer’s actual identities, permissions and workspaces. Then walk through implementation, checks, review and a human decision. Merge and deployment remain separately authorized actions.</p>

<p>Next, deliberately introduce failures in a disposable environment. Add a known defect and see whether review catches it. Fail a required check and see whether the workflow refuses completion. Change the candidate after review and see whether the old approval becomes invalid. Deliver the same event twice and look for duplicate work. Omit necessary information and see whether execution stops and asks a precise question.</p>

<p>Passing those tests would show only that the corresponding control paths worked under the test conditions. It would not establish that the whole factory is reliable, much less that it has saved money.</p>

<p>Only then would I move to a small number of real tasks, one at a time, recording successes, failures, blockers and abandoned attempts. I would track the total human effort spent prompting, reviewing, repairing and maintaining the system, alongside tool costs, missed defects and final outcomes. Without a comparable manual baseline, I would not claim a particular time saving.</p>

<p>I hope this design will reduce lost context, repeated investigation and unproductive rework. Whether it is worth building further depends on the total cost of an accepted result, and whether that result solves a problem worth solving.</p>

<p>Until I have that evidence, I would rather work toward a delivery loop with limited throughput and tested stopping behavior than automatically send more work into the queue.</p>

<h2 id="references">References</h2>

<ol>
  <li><a href="https://github.com/mattpocock/dictionary-of-ai-coding#software-factory">Matt Pocock: Dictionary of AI Coding: Software factory and Dark factory</a></li>
  <li><a href="https://github.com/mattpocock/skills/pull/1316">Matt Pocock: Draft software-factory vocabulary, PR #1316</a>. When I wrote this, the PR was open and marked Draft. It supplies a design vocabulary, not an executable pipeline.</li>
  <li><a href="https://hermes-agent.nousresearch.com/docs/user-guide/features/kanban">Hermes Agent: Kanban</a></li>
  <li><a href="https://hermes-agent.nousresearch.com/docs/user-guide/features/kanban-worker-lanes">Hermes Agent: Kanban worker lanes</a></li>
  <li><a href="https://git-scm.com/docs/git-worktree">Git documentation: git-worktree</a>: explains multiple working trees; workspace separation is not a security sandbox.</li>
</ol>

<p><em>This article develops the delivery portion of <a href="/posts/ai-native-one-person-company-operating-system/">my earlier Company OS design</a>. The risk tiers, evidence structure and pilot constraints are proposed working rules. They are neither complete guarantees supplied by the projects above nor validated industry standards.</em></p>]]></content><author><name>Joey Wang</name></author><category term="ai" /><category term="engineering" /><category term="ai" /><category term="agents" /><category term="hermes" /><category term="automation" /><category term="testing" /><summary type="html"><![CDATA[A software factory design with durable evidence, risk-based reviews and human approval, tested first on whether it stops safely rather than how fast it runs.]]></summary></entry><entry><title type="html">The Engineer Who Owns the Outcome</title><link href="https://joeywang.github.io/posts/the-engineer-who-owns-the-outcome/" rel="alternate" type="text/html" title="The Engineer Who Owns the Outcome" /><published>2026-10-10T08:32:00+00:00</published><updated>2026-10-10T08:32:00+00:00</updated><id>https://joeywang.github.io/posts/the-engineer-who-owns-the-outcome</id><content type="html" xml:base="https://joeywang.github.io/posts/the-engineer-who-owns-the-outcome/"><![CDATA[<div class="audio-summary">
  <p><strong>Audio summary:</strong> a short spoken version of this post.</p>
  <audio controls="" preload="metadata">
    <source src="/assets/audio/2026-10-10-the-engineer-who-owns-the-outcome-summary.ogg" type="audio/ogg" />
    Your browser does not support the audio element.
  </audio>
</div>

<p>In <a href="/posts/ai-era-developer-role-coder-to-architect/">an earlier post on the changing developer role</a>, I described how AI shifts a developer’s work from writing code toward designing and validating systems. I still think that is right, but it leaves a harder question open: if a model can produce the code, sketch the architecture, and propose the workflow, what is the engineer responsible for?</p>

<p>My answer is not another skill for the list. It is responsibility for an outcome that matters: knowing which problem is worth solving, making the result trustworthy where it runs, and finding out whether it helped. Code is part of that work. It is not the result.</p>

<h2 id="a-fluent-draft-is-not-a-better-support-process">A fluent draft is not a better support process</h2>

<p>Take a hypothetical support team considering AI-generated replies. A model that writes fluent drafts is a promising component. It is not evidence that support got better.</p>

<p>Someone still has to decide what information a reply may rely on, which policies apply, when a human must review it, how odd cases are handled, and how staff correct mistakes. Then someone has to find out whether it helped. Replies may be more accurate. Or the team may be sending more of them, faster, with the same errors.</p>

<p>These are not cleanup tasks for after implementation. They cross code, data, operating rules, and people, and together they decide whether the feature becomes a system people can use and trust. Ordinary software is no different: automate a step without knowing why it exists and you get an ineffective process that runs faster. “Can we build this?” is the easy question. “What should improve, for whom, and how would we know?” is the one that matters.</p>

<h2 id="designing-the-workflow-is-not-the-moat">Designing the workflow is not the moat</h2>

<p>It is tempting to say the human part is designing the workflow. I do not think that holds. Models can already propose steps, integrations, handoffs, and failure cases. Calling workflow design permanently beyond AI repeats the old mistake of equating engineering with typing code, one level up.</p>

<p>What a model cannot do is answer for the choice. A plausible design still has to be tested against a specific organization, its users, and the cost of being wrong. What can run automatically, and what must stay reviewable or reversible? Who approves the change, and who looks after it once the demo is over? Someone has to make those calls and change course when the evidence disagrees. That is responsibility, not design skill.</p>

<h2 id="trust-is-engineering-work">Trust is engineering work</h2>

<p>Trust in an AI-enabled system does not come from a capable model. It comes from clear limits on what the system may do, checks suited to the risk, a way for people to inspect and correct its work, and a sensible response when it is uncertain or fails.</p>

<p>For one task, the right limit is a draft a person approves. For another, the system can act alone within a narrow, reversible scope. The choice depends on what an error costs, how good the checks are, and who lives with the result. This is not paperwork around the real engineering. It is the engineering. A demo that works once but cannot be checked, corrected, or operated is not finished.</p>

<p>Evidence is the other half of trust. Lines generated, agent steps completed, and impressive screenshots describe activity, not value. The measure has to follow the goal: fewer errors, faster turnaround, less repetitive work, or whatever the people affected care about. If I cannot say what evidence would show a change helped, I am not ready to call it done.</p>

<h2 id="responsibility-is-not-the-same-as-authority">Responsibility is not the same as authority</h2>

<p>“Own the outcome” can sound like asking one engineer to answer for everything. That is not what I mean. Engineers rarely control the policy, the staffing, or whether the organization adopts a change. Product owners, domain experts, operators, leaders, and users all shape whether it succeeds. Holding someone accountable for decisions they cannot make is not ownership. It is blame looking for a target.</p>

<p>So the line I draw is this. Within my authority, I own the quality of the system, the honesty of what we claim about it, and the evidence for whether it works. Beyond that authority, my job is to make risks and trade-offs visible to the people who can decide, and to say plainly when a decision is not mine to make. The organization owes the other half: explicit ownership, involvement from the people affected, and enough authority for whoever is accountable for a risk to address it. An engineer can make consequences legible. They cannot replace the organization around the system.</p>

<h2 id="the-case-for-craft">The case for craft</h2>

<p>There is a fair objection here. If the job is outcomes and trust, does deep technical craft matter less? Is domain expertise just one input among many?</p>

<p>The objection gets something important right. Craft and domain knowledge are how anyone notices that generated output is plausible but wrong: code that passes the obvious tests and mishandles the case that matters, a shortcut that breaks a real constraint, a metric that improves while users get worse service. Without that expertise, owning the outcome is a promise nobody can keep, because the failures that matter look correct.</p>

<p>Where I differ is on what follows. Expertise does not matter less. It matters more when it is aimed at the whole outcome instead of measured by how much code it produces.</p>

<h2 id="the-question-that-stays">The question that stays</h2>

<p>AI will change the mix of engineering work. Some implementation gets faster, some tasks disappear, new failure modes appear, and the balance will differ by team and domain. Under all of that is the question I want to be able to answer for any system I put my name to: did it make something that matters better, within limits people understand, and can I show it?</p>

<p>Getting a model to produce code does not answer that. It is where the question starts.</p>]]></content><author><name>Joey Wang</name></author><category term="ai" /><category term="engineering" /><category term="ai" /><category term="career" /><category term="automation" /><summary type="html"><![CDATA[When AI can draft the code and the workflow, an engineer's lasting job is owning the outcome: a system people can trust, and evidence that it helped.]]></summary></entry><entry><title type="html">Sidekiq Queue Strategies: Match Urgency and Tolerance to the Job</title><link href="https://joeywang.github.io/posts/sidekiq-queue-strategies/" rel="alternate" type="text/html" title="Sidekiq Queue Strategies: Match Urgency and Tolerance to the Job" /><published>2026-09-27T19:00:00+00:00</published><updated>2026-09-27T19:00:00+00:00</updated><id>https://joeywang.github.io/posts/sidekiq-queue-strategies</id><content type="html" xml:base="https://joeywang.github.io/posts/sidekiq-queue-strategies/"><![CDATA[<div class="audio-summary">
  <p><strong>Audio summary:</strong> a short spoken version of this post.</p>
  <audio controls="" preload="metadata">
    <source src="/assets/audio/2026-09-27-sidekiq-queue-strategies-summary.ogg" type="audio/ogg" />
    Your browser does not support the audio element.
  </audio>
</div>

<p>A Sidekiq queue called <code class="language-plaintext highlighter-rouge">default</code> is easy to start with. It is also an easy place to hide a growing set of contradictory requirements.</p>

<p>An email, a payment reconciliation, a search-index refresh, a nightly report, and a cache warm-up can all be valid background jobs. They do not have the same urgency. They do not have the same tolerance for delay. They do not have the same answer to the question: “What happens if this job is enqueued twice?”</p>

<p>The queue name is not the strategy. It is only the first label.</p>

<p>The useful design work is to classify each job before deciding where it runs:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
</pre></td><td class="rouge-code"><pre>urgency       How quickly does it need to start?
tolerance     How bad is a delay or a missed run?
duplicates    Is a second copy harmless, wasteful, or dangerous?
retry         Should failure wait, retry, or alert somebody?
concurrency   Can several copies run safely at the same time?
</pre></td></tr></tbody></table></code></pre></div></div>

<p>Once those answers are written down, Sidekiq’s queue modes and uniqueness options become much easier to use without turning <code class="language-plaintext highlighter-rouge">sidekiq.yml</code> into a collection of guesses.</p>

<h2 id="start-with-urgency-and-tolerance">Start with urgency and tolerance</h2>

<p>I find it useful to separate two properties that are often treated as one.</p>

<p><strong>Urgency</strong> is about time to start. A job may be urgent because a user is waiting for its result, or because a delayed action will make another workflow incorrect.</p>

<p><strong>Tolerance</strong> is about how much delay or loss the business can accept. A low-urgency job may still have low tolerance for being lost. A nightly export can wait for an hour, but it may still need an operator-visible failure if it has not completed by morning.</p>

<p>That gives a rough first classification:</p>

<table>
  <thead>
    <tr>
      <th>Class</th>
      <th>Urgency</th>
      <th>Delay tolerance</th>
      <th>Typical jobs</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Critical</td>
      <td>high</td>
      <td>low</td>
      <td>payment capture, access changes, time-sensitive notifications</td>
    </tr>
    <tr>
      <td>Interactive</td>
      <td>high</td>
      <td>short</td>
      <td>user-requested exports, document generation, webhook follow-up</td>
    </tr>
    <tr>
      <td>Important</td>
      <td>medium</td>
      <td>moderate</td>
      <td>search indexing, CRM sync, progress aggregation</td>
    </tr>
    <tr>
      <td>Bulk</td>
      <td>low</td>
      <td>high</td>
      <td>historical backfills, analytics imports, cache warming</td>
    </tr>
    <tr>
      <td>Periodic</td>
      <td>deadline-based</td>
      <td>depends on deadline</td>
      <td>daily reports, cleanup, reconciliation</td>
    </tr>
  </tbody>
</table>

<p>This is not a universal taxonomy. It is a prompt to make the trade-off visible.</p>

<p>A job’s class should also answer what happens when the system is busy. If bulk work consumes every thread, should an access-change job wait behind it? If the answer is no, the queues need separate capacity, not only different names.</p>

<h2 id="one-queue-is-a-policy-decision">One queue is a policy decision</h2>

<p>A single queue is not neutral. It says that all jobs compete for the same capacity and that the order in which they are checked is good enough.</p>

<p>That may be perfectly reasonable for a small application. It becomes a problem when a slow or high-volume producer can delay a different kind of work.</p>

<p>A simple separation might look like this:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
</pre></td><td class="rouge-code"><pre>critical   payments, security, account changes
interactive user-facing work with a short response expectation
normal     ordinary application jobs
bulk       imports, backfills, expensive maintenance
</pre></td></tr></tbody></table></code></pre></div></div>

<p>The names are less important than the ownership rules. A queue should have a reason to exist, a rough workload profile, and a way to tell whether it is being starved.</p>

<p>Sidekiq supports strict queue ordering and weighted queues. Strict ordering means a process checks queues in the declared order. Weighted queues give a queue more chances to be selected, but they do not guarantee a fixed percentage of completed work. The <a href="https://sidekiq.org/wiki/Advanced-Options">Sidekiq advanced options</a> documentation describes both modes and the limitation that a process cannot mix them.</p>

<p>For example, a weighted process could be configured like this:</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
</pre></td><td class="rouge-code"><pre><span class="na">:queues</span><span class="pi">:</span>
  <span class="pi">-</span> <span class="pi">[</span><span class="nv">critical</span><span class="pi">,</span> <span class="nv">6</span><span class="pi">]</span>
  <span class="pi">-</span> <span class="pi">[</span><span class="nv">interactive</span><span class="pi">,</span> <span class="nv">3</span><span class="pi">]</span>
  <span class="pi">-</span> <span class="pi">[</span><span class="nv">normal</span><span class="pi">,</span> <span class="nv">2</span><span class="pi">]</span>
  <span class="pi">-</span> <span class="pi">[</span><span class="nv">bulk</span><span class="pi">,</span> <span class="nv">1</span><span class="pi">]</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>This is a useful starting point when the queues share a worker pool and the application can tolerate probabilistic selection. It is not a service-level agreement. A busy <code class="language-plaintext highlighter-rouge">critical</code> queue can still consume capacity, and a low-weight queue can still wait for a long time under sustained load.</p>

<h2 id="when-separate-processes-are-clearer">When separate processes are clearer</h2>

<p>If a queue must remain available even when another workload is unhealthy, give it separate process capacity.</p>

<p>For example:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
</pre></td><td class="rouge-code"><pre>process A: critical, interactive
process B: normal
process C: bulk
</pre></td></tr></tbody></table></code></pre></div></div>

<p>Or, if critical work really must be checked first within its process:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
</pre></td><td class="rouge-code"><pre>process A: critical -&gt; interactive
process B: normal -&gt; bulk
</pre></td></tr></tbody></table></code></pre></div></div>

<p>This costs more operationally, but it makes the boundary easier to reason about. A large import cannot fill every worker slot needed for a payment-related job.</p>

<p>Strict ordering has a catch: lower queues can starve while higher queues remain busy. Weighted queues have a different catch: they improve sharing but do not provide a hard deadline. The choice depends on which failure is worse for the application.</p>

<p>I would rather have two deliberately sized processes than six queues that all run through one overloaded pool with no clear capacity model.</p>

<h2 id="make-the-job-itself-safe-to-retry">Make the job itself safe to retry</h2>

<p>Queue selection cannot repair a job that is unsafe to run twice.</p>

<p>Sidekiq retries failed jobs, and a worker can lose its connection after the remote operation has succeeded but before the process records success locally. From the application’s point of view, the job may be attempted again even though the first attempt partially completed.</p>

<p>That is why idempotency belongs in the job design, not only in the queue configuration.</p>

<p>A job that records a payment, sends a message, or applies an external change should have a business-level idempotency key where the downstream system supports one. A job that updates a local projection should usually write the same result again safely rather than assuming it will run exactly once.</p>

<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
</pre></td><td class="rouge-code"><pre><span class="k">class</span> <span class="nc">RebuildSearchDocumentJob</span>
  <span class="kp">include</span> <span class="no">Sidekiq</span><span class="o">::</span><span class="no">Job</span>

  <span class="n">sidekiq_options</span> <span class="ss">queue: :normal</span><span class="p">,</span> <span class="ss">retry: </span><span class="mi">8</span>

  <span class="k">def</span> <span class="nf">perform</span><span class="p">(</span><span class="n">document_id</span><span class="p">)</span>
    <span class="n">document</span> <span class="o">=</span> <span class="no">Document</span><span class="p">.</span><span class="nf">find</span><span class="p">(</span><span class="n">document_id</span><span class="p">)</span>
    <span class="no">SearchIndex</span><span class="p">.</span><span class="nf">upsert</span><span class="p">(</span><span class="n">document</span><span class="p">.</span><span class="nf">id</span><span class="p">,</span> <span class="n">document</span><span class="p">.</span><span class="nf">search_payload</span><span class="p">)</span>
  <span class="k">end</span>
<span class="k">end</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>The important property here is not the number <code class="language-plaintext highlighter-rouge">8</code>. It is that <code class="language-plaintext highlighter-rouge">upsert</code> and the document payload make a retry safe. Retry counts should follow the failure mode and the business deadline, not a copied default.</p>

<h2 id="uniqueness-is-not-idempotency">Uniqueness is not idempotency</h2>

<p>A uniqueness lock answers a different question:</p>

<blockquote>
  <p>Should I allow another copy of this job to be enqueued or executed while this copy is in progress?</p>
</blockquote>

<p>Idempotency answers:</p>

<blockquote>
  <p>If this operation happens again, will the result remain correct?</p>
</blockquote>

<p>A unique lock can reduce duplicate work. It cannot prove that a remote API call happened only once, and it cannot replace a durable business record or an idempotency key.</p>

<p>There are two common ways to get uniqueness in Sidekiq applications:</p>

<ul>
  <li>Sidekiq Enterprise’s built-in unique jobs feature;</li>
  <li>the open source <a href="https://github.com/mhenrixon/sidekiq-unique-jobs"><code class="language-plaintext highlighter-rouge">sidekiq-unique-jobs</code></a> gem.</li>
</ul>

<p>They have different APIs and operational details, so a team should choose one deliberately and read the version-specific documentation. The examples below use <code class="language-plaintext highlighter-rouge">sidekiq-unique-jobs</code> terminology.</p>

<h2 id="choose-the-lock-from-the-duplicate-failure-mode">Choose the lock from the duplicate failure mode</h2>

<p>The gem provides several lock types. The useful question is not “Which lock is strongest?” It is “At what point do duplicates become a problem?”</p>

<h3 id="duplicate-enqueueing-is-wasteful">Duplicate enqueueing is wasteful</h3>

<p>Suppose a record is edited several times in a short period and each edit queues a search refresh. The latest refresh may make earlier refreshes unnecessary.</p>

<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
</pre></td><td class="rouge-code"><pre><span class="k">class</span> <span class="nc">RefreshSearchJob</span>
  <span class="kp">include</span> <span class="no">Sidekiq</span><span class="o">::</span><span class="no">Job</span>

  <span class="n">sidekiq_options</span> <span class="p">\</span>
    <span class="ss">queue: :normal</span><span class="p">,</span>
    <span class="ss">lock: :until_executing</span><span class="p">,</span>
    <span class="ss">on_conflict: :log</span>

  <span class="k">def</span> <span class="nf">perform</span><span class="p">(</span><span class="n">record_id</span><span class="p">)</span>
    <span class="no">SearchIndex</span><span class="p">.</span><span class="nf">refresh</span><span class="p">(</span><span class="n">record_id</span><span class="p">)</span>
  <span class="k">end</span>
<span class="k">end</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>An <code class="language-plaintext highlighter-rouge">:until_executing</code> lock prevents multiple copies from accumulating before the worker starts. This is appropriate when dropping a duplicate is acceptable because another job will produce the same current state.</p>

<p>It is not appropriate for a job where every event must be processed. In that case, model the event or use an outbox rather than treating a duplicate as disposable.</p>

<h3 id="the-whole-operation-should-remain-unique">The whole operation should remain unique</h3>

<p>For a rebuild or synchronization that should not overlap with another copy, <code class="language-plaintext highlighter-rouge">:until_executed</code> holds the lock through execution:</p>

<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
</pre></td><td class="rouge-code"><pre><span class="k">class</span> <span class="nc">SyncCustomerJob</span>
  <span class="kp">include</span> <span class="no">Sidekiq</span><span class="o">::</span><span class="no">Job</span>

  <span class="n">sidekiq_options</span> <span class="p">\</span>
    <span class="ss">queue: :important</span><span class="p">,</span>
    <span class="ss">lock: :until_executed</span><span class="p">,</span>
    <span class="ss">on_conflict: :log</span><span class="p">,</span>
    <span class="ss">retry: </span><span class="mi">6</span>

  <span class="k">def</span> <span class="nf">perform</span><span class="p">(</span><span class="n">customer_id</span><span class="p">)</span>
    <span class="no">CustomerSync</span><span class="p">.</span><span class="nf">call</span><span class="p">(</span><span class="n">customer_id</span><span class="p">)</span>
  <span class="k">end</span>
<span class="k">end</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>This can stop a thundering herd when several triggers request the same synchronization. It also means that a failed job may keep the lock while it retries, depending on the lock and retry behavior. That can be correct, but it can also delay a fresh request longer than the caller expects.</p>

<h3 id="only-concurrent-execution-is-unsafe">Only concurrent execution is unsafe</h3>

<p>Some jobs are safe to enqueue more than once but unsafe to run concurrently. A generated report might be overwritten by two writers, or a third-party endpoint might reject simultaneous updates.</p>

<p>For that case, <code class="language-plaintext highlighter-rouge">:while_executing</code> is a server-side concurrency lock. It does not stop duplicate jobs from entering Redis. It stops them from doing the protected work at the same time.</p>

<p>That distinction matters operationally. A queue can still grow even when only one copy is executing.</p>

<h3 id="a-time-window-matters-more-than-the-exact-job">A time window matters more than the exact job</h3>

<p>For periodic work, <code class="language-plaintext highlighter-rouge">:until_expired</code> can be useful when the business rule is “at most one request per interval” rather than “one copy until successful completion”.</p>

<p>Be careful with long lock periods. A lock can prevent a legitimate recovery or manual rerun. Locks also need an expiry or a cleanup plan, because a process can die while a lock exists.</p>

<p>The Sidekiq Enterprise documentation makes the same broader point about uniqueness: treat it as best effort, use a bounded period, and keep the lock short. The open source gem has its own reaper and configuration choices, but the operational lesson is the same.</p>

<h2 id="conflict-strategy-is-part-of-the-business-rule">Conflict strategy is part of the business rule</h2>

<p>When a duplicate is detected, the result should be intentional:</p>

<ul>
  <li><code class="language-plaintext highlighter-rouge">:log</code> is reasonable when the duplicate is harmless and should be discarded.</li>
  <li><code class="language-plaintext highlighter-rouge">:reject</code> is useful when an operator needs a visible record that a required job was not accepted.</li>
  <li><code class="language-plaintext highlighter-rouge">:reschedule</code> can be appropriate when the work must happen later, but it changes queue pressure and retry statistics.</li>
  <li><code class="language-plaintext highlighter-rouge">:replace</code> fits jobs where the newest request supersedes an older pending request.</li>
</ul>

<p>The correct choice depends on whether the duplicate represents the same desired state or a separate event.</p>

<p>For example, two requests to rebuild the current search document may collapse into one. Two invoice events should not. The first is state convergence; the second is event processing. Giving both jobs the same uniqueness policy is a data-model mistake disguised as a queue setting.</p>

<h2 id="a-practical-classification-table">A practical classification table</h2>

<p>Before adding a new worker, I would write a small record like this:</p>

<table>
  <thead>
    <tr>
      <th>Question</th>
      <th>Example answer for a search refresh</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>User waiting?</td>
      <td>No</td>
    </tr>
    <tr>
      <td>Start within</td>
      <td>A few minutes</td>
    </tr>
    <tr>
      <td>Maximum useful delay</td>
      <td>30 minutes</td>
    </tr>
    <tr>
      <td>Duplicate meaning</td>
      <td>Same desired state</td>
    </tr>
    <tr>
      <td>Concurrent execution safe?</td>
      <td>Yes, if the index uses upsert</td>
    </tr>
    <tr>
      <td>Retry</td>
      <td>Yes, with bounded backoff</td>
    </tr>
    <tr>
      <td>Queue</td>
      <td><code class="language-plaintext highlighter-rouge">normal</code></td>
    </tr>
    <tr>
      <td>Uniqueness</td>
      <td><code class="language-plaintext highlighter-rouge">:until_executing</code></td>
    </tr>
    <tr>
      <td>Conflict</td>
      <td>Log and drop</td>
    </tr>
    <tr>
      <td>Alert</td>
      <td>Queue age or repeated failures</td>
    </tr>
  </tbody>
</table>

<p>For a payment capture, the answers would be different. The job might need a dedicated high-priority process, a durable idempotency key, an explicit failure alert, and no silent duplicate dropping.</p>

<p>Writing the answers down also exposes missing requirements. If nobody knows the maximum useful delay, it is difficult to choose a queue weight responsibly.</p>

<h2 id="watch-the-queue-not-just-the-worker">Watch the queue, not just the worker</h2>

<p>A healthy Sidekiq process can still be serving an unhealthy queue policy. I would monitor at least:</p>

<ul>
  <li>queue latency by queue;</li>
  <li>oldest job age;</li>
  <li>retry and dead-set growth;</li>
  <li>execution time and failure rate by worker;</li>
  <li>duplicate conflicts and lock cleanup;</li>
  <li>concurrency saturation;</li>
  <li>the number of jobs enqueued versus completed.</li>
</ul>

<p>Queue latency is often more useful than total job count. A bulk queue can contain many jobs without being in trouble. A critical queue with a small number of old jobs may already be failing its purpose.</p>

<p>Alerts should follow the class of work. A five-minute delay may be harmless for a backfill and unacceptable for an access change. One global queue-latency threshold usually hides that distinction.</p>

<h2 id="the-strategy-i-would-use-first">The strategy I would use first</h2>

<p>For a Rails application with mixed workloads, I would start conservatively:</p>

<ol>
  <li>Keep the default queue for ordinary work only.</li>
  <li>Create a separate <code class="language-plaintext highlighter-rouge">critical</code> or <code class="language-plaintext highlighter-rouge">interactive</code> queue only when there is a measured reason.</li>
  <li>Move bulk work away from the process that serves user-facing jobs.</li>
  <li>Make every retrying job idempotent at the business boundary.</li>
  <li>Add uniqueness only where duplicate work is a known problem.</li>
  <li>Choose the conflict behavior explicitly rather than accepting a silent default.</li>
  <li>Monitor queue age by class before tuning weights.</li>
  <li>Revisit the classification when the workload or deadline changes.</li>
</ol>

<p>The goal is not to produce a complicated queue topology. It is to stop unrelated jobs from competing as if they had the same urgency and the same failure semantics.</p>

<p>Sidekiq gives us useful mechanics: queues, weights, retries, capsules, and uniqueness locks. The application still has to decide what a job means, how late is too late, and whether a duplicate is an error, a no-op, or a separate event. Those decisions belong in the design of the job, not hidden in a YAML file.</p>

<h2 id="references">References</h2>

<ul>
  <li><a href="https://sidekiq.org/wiki/Advanced-Options">Sidekiq advanced queue options</a></li>
  <li><a href="https://sidekiq.org/wiki/Reliability">Sidekiq reliability and queue prioritisation</a></li>
  <li><a href="https://sidekiq.org/wiki/Ent-Unique-Jobs">Sidekiq Enterprise unique jobs</a></li>
  <li><a href="https://github.com/mhenrixon/sidekiq-unique-jobs"><code class="language-plaintext highlighter-rouge">sidekiq-unique-jobs</code> documentation</a></li>
</ul>]]></content><author><name>Joey Wang</name></author><category term="engineering" /><category term="ruby" /><category term="rails" /><category term="sidekiq" /><category term="background-jobs" /><category term="reliability" /><summary type="html"><![CDATA[A practical way to classify Sidekiq jobs by urgency, failure tolerance, duplication risk, and queue strategy.]]></summary></entry><entry><title type="html">The OAuth Redirect URI Is Not a Suggestion</title><link href="https://joeywang.github.io/posts/google-oauth-redirect-uri-mismatch/" rel="alternate" type="text/html" title="The OAuth Redirect URI Is Not a Suggestion" /><published>2026-09-08T21:44:56+00:00</published><updated>2026-09-08T21:44:56+00:00</updated><id>https://joeywang.github.io/posts/google-oauth-redirect-uri-mismatch</id><content type="html" xml:base="https://joeywang.github.io/posts/google-oauth-redirect-uri-mismatch/"><![CDATA[<div class="audio-summary">
  <p><strong>Audio summary:</strong> a short spoken version of this post.</p>
  <audio controls="" preload="metadata">
    <source src="/assets/audio/2026-09-08-google-oauth-redirect-uri-mismatch-summary.ogg" type="audio/ogg" />
    Your browser does not support the audio element.
  </audio>
</div>

<p>I recently put Google sign-in in front of a self-hosted web dashboard.</p>

<p>The dashboard was already running behind HTTPS and a reverse proxy. The first version used a shared password at the proxy layer. That worked, but it made the dashboard feel like a private server rather than an application with a proper identity boundary.</p>

<p>The target architecture was straightforward:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
</pre></td><td class="rouge-code"><pre>Browser
  -&gt; HTTPS reverse proxy
     -&gt; OAuth2 proxy
        -&gt; local dashboard
</pre></td></tr></tbody></table></code></pre></div></div>

<p>The dashboard itself did not need to learn about Google. The authentication layer could handle the OAuth flow, create a session cookie, and forward an authenticated request upstream.</p>

<p>The first login attempt failed immediately:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre>Error 400: redirect_uri_mismatch
</pre></td></tr></tbody></table></code></pre></div></div>

<p>This error is less mysterious than it looks.</p>

<h2 id="the-redirect-uri-is-a-contract">The redirect URI is a contract</h2>

<p>An OAuth login has a round trip:</p>

<ol>
  <li>The application sends the browser to Google.</li>
  <li>Google authenticates the user.</li>
  <li>Google sends the browser back to the application.</li>
  <li>The application exchanges the authorization code for tokens.</li>
</ol>

<p>The return address in step three is the redirect URI. It is not merely a hint. Google compares it with the list registered on the OAuth client. The comparison is effectively exact.</p>

<p>These are different redirect URIs:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
</pre></td><td class="rouge-code"><pre>https://dashboard.example.com/oauth2/callback
https://dashboard.example.com/auth/callback
https://dashboard.example.com/oauth2/callback/
http://dashboard.example.com/oauth2/callback
</pre></td></tr></tbody></table></code></pre></div></div>

<p>The differences are small to a person reading them. They are not small to the OAuth provider.</p>

<p>The scheme, hostname, path, port, and trailing slash all matter.</p>

<h2 id="the-mistake-was-in-the-boundary-between-two-systems">The mistake was in the boundary between two systems</h2>

<p>The reverse proxy had one public hostname. The authentication proxy had another idea about its callback path. Google had a third value registered in the OAuth client.</p>

<p>Each individual component looked reasonable:</p>

<ul>
  <li>the public site loaded over HTTPS;</li>
  <li>the OAuth client ID was correct;</li>
  <li>the authentication proxy could reach Google;</li>
  <li>the local dashboard was healthy;</li>
  <li>the callback handler existed.</li>
</ul>

<p>But OAuth does not care that the pieces are individually reasonable. It cares that the exact same URI appears in both places.</p>

<p>The useful debugging step was to inspect the authorization request generated by the running service rather than infer it from a configuration file. In the browser’s developer tools, I checked the redirect to Google’s authorization endpoint and decoded only its <code class="language-plaintext highlighter-rouge">redirect_uri</code> parameter. I did not copy or share the full authorization URL: it can contain transient state and identifying configuration.</p>

<p>The value to compare was the decoded callback URI:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre>https://dashboard.example.com/oauth2/callback
</pre></td></tr></tbody></table></code></pre></div></div>

<p>That was the value Google needed to find in the OAuth client’s <strong>Authorized redirect URIs</strong> list. <a href="https://developers.google.com/identity/protocols/oauth2/web-server#redirect-uri">Google’s web-server OAuth documentation</a> requires the URI in the request to match a registered URI for that client.</p>

<h2 id="the-fix">The fix</h2>

<p>I added the exact URI to the Google OAuth client:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre>https://dashboard.example.com/oauth2/callback
</pre></td></tr></tbody></table></code></pre></div></div>

<p>The following details mattered:</p>

<ul>
  <li><code class="language-plaintext highlighter-rouge">https</code>, not <code class="language-plaintext highlighter-rouge">http</code>;</li>
  <li>the public hostname, not <code class="language-plaintext highlighter-rouge">127.0.0.1</code>;</li>
  <li>the authentication proxy’s callback path;</li>
  <li>no extra trailing slash;</li>
  <li>no guessed port;</li>
  <li>no callback path from an earlier design.</li>
</ul>

<p>After saving the OAuth client, the login flow worked without changing the dashboard application itself.</p>

<h2 id="keeping-the-dashboard-out-of-the-public-network">Keeping the dashboard out of the public network</h2>

<p>The authentication proxy and dashboard both remained on private upstream addresses behind the reverse proxy. Only the reverse proxy listened on the public HTTPS interface.</p>

<p>The resulting shape was:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
</pre></td><td class="rouge-code"><pre>Public HTTPS
  -&gt; reverse proxy
     -&gt; authentication proxy
        -&gt; dashboard
</pre></td></tr></tbody></table></code></pre></div></div>

<p>That distinction is important. Google authentication is not a replacement for network boundaries, TLS, rate limiting, or careful proxy configuration.</p>

<p>I kept the existing network allowlist as an additional boundary. Google identity answers “who is this?” The network policy answers “from where may this service be reached?” Those are different questions, and it is useful to keep both.</p>

<h2 id="restricting-the-identity-domain">Restricting the identity domain</h2>

<p>A Google OAuth client can authenticate Google accounts, but authentication alone does not always mean authorization.</p>

<p>If a dashboard is intended only for an organisation, the proxy should enforce the allowed email domain or organisation policy after Google returns the identity. In a generic configuration that might look like:</p>

<div class="language-ini highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre><span class="py">email_domains</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="s">["example.org"]</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>This is not a substitute for checking the actual claims and provider configuration. It is one layer in the access policy.</p>

<p>For a small private dashboard, I prefer an organisation-restricted Google Workspace application where possible, plus an explicit allowed-domain check in the authentication proxy.</p>

<h2 id="things-not-to-put-in-a-blog-post">Things not to put in a blog post</h2>

<p>The working setup contains values that are useful to the machine but not useful to readers:</p>

<ul>
  <li>OAuth client secrets;</li>
  <li>cookie-signing secrets;</li>
  <li>client IDs when they identify a private deployment;</li>
  <li>private hostnames and IP allowlists;</li>
  <li>internal service names;</li>
  <li>account email addresses;</li>
  <li>screenshots containing authorization URLs or session cookies.</li>
</ul>

<p>A technical explanation does not need those values. The important information is the relationship between the components and the verification method.</p>

<p>I also avoid copying an authorization URL into a ticket or chat. It contains state and can contain identifying information even when the client secret is not present.</p>

<h2 id="a-compact-troubleshooting-checklist">A compact troubleshooting checklist</h2>

<p>When Google reports <code class="language-plaintext highlighter-rouge">redirect_uri_mismatch</code>, I now check these in order:</p>

<ol>
  <li>Inspect the actual authorization request generated by the application or proxy.</li>
  <li>Decode the <code class="language-plaintext highlighter-rouge">redirect_uri</code> query parameter.</li>
  <li>Compare it character-for-character with Google Cloud’s registered URI.</li>
  <li>Check HTTPS versus HTTP.</li>
  <li>Check the public hostname versus an upstream or localhost hostname.</li>
  <li>Check the path and trailing slash.</li>
  <li>Check whether a reverse proxy is rewriting the path.</li>
  <li>Check that the OAuth client ID in the running service is the one being edited.</li>
  <li>Save the Google Cloud change and retry in a fresh browser session.</li>
  <li>Never “fix” the problem by weakening TLS or exposing the upstream dashboard directly.</li>
</ol>

<p>This process is faster than starting with the assumption that Google OAuth itself is broken.</p>

<h2 id="the-broader-lesson">The broader lesson</h2>

<p>OAuth failures often happen at system boundaries. The identity provider knows one URL, the reverse proxy advertises another, and the application logs a third. Looking at only one configuration file creates false confidence.</p>

<p>The reliable source of truth is the request that actually leaves the running system.</p>

<p>For redirect errors, inspect the live authorization URL, extract the redirect URI, and compare it exactly with the provider’s registered value. Once I treated the URI as a strict contract instead of a descriptive label, the error became a two-minute configuration fix.</p>

<p>The dashboard was not failing to authenticate Google. Google was correctly refusing to send a credential to an address the application had not explicitly registered.</p>]]></content><author><name>Joey Wang</name></author><category term="engineering" /><category term="oauth" /><category term="google" /><category term="nginx" /><category term="security" /><category term="troubleshooting" /><summary type="html"><![CDATA[What I learned wiring Google OAuth in front of a self-hosted dashboard, and why redirect_uri_mismatch is usually a configuration comparison problem.]]></summary></entry><entry><title type="html">CI Gates, Test Result Caches, and the Code Graph That Might Tell Us What to Test</title><link href="https://joeywang.github.io/posts/ci-gates-test-caches-code-graphs/" rel="alternate" type="text/html" title="CI Gates, Test Result Caches, and the Code Graph That Might Tell Us What to Test" /><published>2026-09-02T17:00:00+00:00</published><updated>2026-09-02T17:00:00+00:00</updated><id>https://joeywang.github.io/posts/ci-gates-test-caches-code-graphs</id><content type="html" xml:base="https://joeywang.github.io/posts/ci-gates-test-caches-code-graphs/"><![CDATA[<audio controls="" preload="metadata" src="/assets/audio/ci-gates-test-caches-code-graphs-summary.ogg">
  Your browser does not support the audio element.
</audio>

<p>A green CI run can still leave me with an uncomfortable question:</p>

<blockquote>
  <p>Did we run the right tests?</p>
</blockquote>

<p>The opposite problem is easier to recognise. A pull request runs every test, every time, even when the change is a README edit or a small isolated helper. The feedback loop gets slower, the bill gets larger, and eventually somebody starts looking for ways around it.</p>

<p>That is how a safety system becomes a delivery tax.</p>

<p>Recently I applied a risk-based CI gate to Turtle, following improvements I had already made in REX. The change was deliberately modest. It did not pretend that a script could understand the entire architecture. It classified changes into test levels, kept RuboCop independent, and used the full suite for high-risk paths and releases.</p>

<p>The more interesting part came afterwards: caching test-result context and asking whether a code graph or an AI agent could help us decide what deserves testing in the first place.</p>

<h2 id="the-old-choice-slow-or-unsafe">The old choice: slow or unsafe</h2>

<p>The usual CI conversation is presented as a binary choice:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
</pre></td><td class="rouge-code"><pre>run everything every time
  -&gt; safe but slow

run only what looks relevant
  -&gt; fast but risky
</pre></td></tr></tbody></table></code></pre></div></div>

<p>Neither is good enough.</p>

<p>Running everything is a reasonable baseline, especially for a small repository. But it hides the fact that not all changes have the same risk. A documentation-only change does not need the same RSpec scope as a database migration, dependency lockfile change, or production release.</p>

<p>Running a guessed subset is more dangerous. File paths are not the same thing as runtime behaviour. A change in a serializer may affect an API response. A configuration change may alter boot behaviour. A seemingly local model callback may be used by a background job, an admin screen, and a mobile endpoint.</p>

<p>The useful goal is not “run fewer tests.” It is:</p>

<blockquote>
  <p>Make the smallest test decision that is still justified by evidence, and make the decision visible.</p>
</blockquote>

<h2 id="a-gate-before-the-test-job">A gate before the test job</h2>

<p>The Turtle workflow now has a small <code class="language-plaintext highlighter-rouge">ci_gate</code> job before RSpec. It classifies the pull request and publishes the decision in the GitHub Actions summary.</p>

<p>The levels are intentionally understandable:</p>

<table>
  <thead>
    <tr>
      <th>Level</th>
      <th>Category</th>
      <th>Typical coverage</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>L0</td>
      <td>Static-only</td>
      <td>RuboCop and other static checks; documentation-only work can skip RSpec</td>
    </tr>
    <tr>
      <td>L1</td>
      <td>Affected</td>
      <td>Fast library, model, and helper specs</td>
    </tr>
    <tr>
      <td>L2</td>
      <td>Smoke</td>
      <td>Fast specs plus request, controller, and view specs</td>
    </tr>
    <tr>
      <td>L3</td>
      <td>Full</td>
      <td>The complete parallel RSpec suite</td>
    </tr>
  </tbody>
</table>

<p>The classifier considers the changed paths, the branch name, and optional labels such as <code class="language-plaintext highlighter-rouge">risk:bugfix</code> or <code class="language-plaintext highlighter-rouge">risk:feat</code>. High-risk paths, including dependencies, database changes, Dockerfiles, workflow files, and test boot configuration, escalate the level. A label can raise the required level, but it cannot lower a level implied by the changed files.</p>

<p>That last rule matters. A label is a useful human signal. It is not permission to bypass evidence.</p>

<p>The release path always selects L3. Production is where optimism becomes an incident, so the test decision should be boring and conservative there.</p>

<p>RuboCop remains a separate required job. The test classifier does not decide whether Ruby style and static checks should run. This is important because a fast test path should not accidentally become a fast-everything path.</p>

<h2 id="what-the-gate-does-not-claim">What the gate does not claim</h2>

<p>The first version is not a perfect test-impact analysis system.</p>

<p>For example, L1 currently runs a broad group of fast specs rather than calculating the exact affected examples. L2 adds request, controller, and view coverage. This is useful prioritisation, but it is still a policy-based approximation.</p>

<p>That is a feature, not an embarrassment.</p>

<p>A transparent approximation is easier to review than a mysterious “AI selected 14 tests” result. The workflow tells us which level was selected and why. If the policy is wrong, we can improve the policy without pretending that the existing result was mathematically precise.</p>

<p>A good first gate should make risk visible before it tries to automate every decision.</p>

<h2 id="caching-test-results-without-caching-confidence">Caching test results without caching confidence</h2>

<p>The second improvement in REX was to cache test-result context. The important distinction is that we do not treat an old green result as proof that a new commit is safe.</p>

<p>The cache key includes the things that can change what a test means:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
</pre></td><td class="rouge-code"><pre>repository
commit and ref
suite and test level
exact command
Ruby / Node / pnpm environment
lockfiles and test configuration
</pre></td></tr></tbody></table></code></pre></div></div>

<p>The key is derived from a canonical JSON description and a SHA-256 digest. If the environment or test command changes, the key changes too.</p>

<p>This is the shape of the context rather than just a directory called <code class="language-plaintext highlighter-rouge">test-cache</code>:</p>

<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
</pre></td><td class="rouge-code"><pre><span class="p">{</span><span class="w">
  </span><span class="nl">"repository"</span><span class="p">:</span><span class="w"> </span><span class="s2">"..."</span><span class="p">,</span><span class="w">
  </span><span class="nl">"sha"</span><span class="p">:</span><span class="w"> </span><span class="s2">"..."</span><span class="p">,</span><span class="w">
  </span><span class="nl">"suite"</span><span class="p">:</span><span class="w"> </span><span class="s2">"rspec"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"level"</span><span class="p">:</span><span class="w"> </span><span class="s2">"full"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"bundle exec rails parallel:spec"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"ruby"</span><span class="p">:</span><span class="w"> </span><span class="s2">"..."</span><span class="p">,</span><span class="w">
  </span><span class="nl">"node"</span><span class="p">:</span><span class="w"> </span><span class="s2">"..."</span><span class="p">,</span><span class="w">
  </span><span class="nl">"files"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="nl">"Gemfile.lock"</span><span class="p">:</span><span class="w"> </span><span class="s2">"..."</span><span class="p">,</span><span class="w">
    </span><span class="nl">"pnpm-lock.yaml"</span><span class="p">:</span><span class="w"> </span><span class="s2">"..."</span><span class="p">,</span><span class="w">
    </span><span class="nl">"config/webpacker.yml"</span><span class="p">:</span><span class="w"> </span><span class="s2">"..."</span><span class="w">
  </span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></pre></td></tr></tbody></table></code></pre></div></div>

<p>The exact commit is part of the identity. That means the cache is not pretending that a result for one source revision is a current result for another revision.</p>

<p>So what is it useful for?</p>

<p>One use is diagnostics. When a replacement run starts, it can find the previous failed RSpec result for the same branch and environment, then run the previously failed examples first. This gives fast feedback about a likely regression while the required suite continues to provide the real gate.</p>

<p>The ordering looks like this:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
</pre></td><td class="rouge-code"><pre>new commit
  -&gt; calculate exact test context
  -&gt; restore matching failure evidence
  -&gt; run previous failed examples first
  -&gt; run the required suite
  -&gt; upload current results
</pre></td></tr></tbody></table></code></pre></div></div>

<p>The failed-example lane is not a replacement for the required suite. It is a short feedback lane. A cached failure is evidence about where to look first, not a verdict about the new code.</p>

<p>That distinction is easy to lose when a CI optimisation is described as “test caching.” We should cache evidence and reproducible context, not confidence.</p>

<p>GitHub’s <a href="https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching">dependency caching documentation</a> makes a related point: cache keys should change when the inputs that produce the cached output change, and workflows can use the cache-hit result to decide whether work can be skipped. For test results, the inputs need to be much richer than one lockfile hash.</p>

<h2 id="where-a-code-graph-can-help">Where a code graph can help</h2>

<p>The next question is harder:</p>

<blockquote>
  <p>How do we know which tests are connected to the changed code?</p>
</blockquote>

<p>This is where code-intelligence and graph tools become interesting. A graph can represent relationships that a directory-based classifier cannot see:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
</pre></td><td class="rouge-code"><pre>changed controller
  -&gt; route
  -&gt; authorization policy
  -&gt; service object
  -&gt; model callback
  -&gt; background job
  -&gt; request and system specs
</pre></td></tr></tbody></table></code></pre></div></div>

<p>A useful system could use those relationships to propose a test set. It might also identify a path that crosses an application boundary, such as a Rails endpoint calling another service or emitting a job consumed elsewhere.</p>

<p>For my own setup, <a href="https://github.com/abhigyanpatwari/GitNexus">GitNexus</a> is an interesting candidate because it exposes dependency, call-chain, execution-flow, and impact-analysis views through a local-first code knowledge graph. The important word is <em>candidate</em>. Its output still needs to be checked against the actual Rails conventions and test suite.</p>

<p><a href="https://codeql.github.com/docs/writing-codeql-queries/about-data-flow-analysis/">CodeQL</a> offers a different model. It is designed for querying code as data and analysing data-flow paths. That is particularly valuable for security and source-to-sink questions, although it is not a drop-in RSpec test selector for a Rails application.</p>

<p>Commercial test-impact systems such as <a href="https://help.launchableinc.com/">Launchable</a> take another approach: combine repository changes with historical test behaviour and use predictive selection to choose tests. That can be powerful, but it introduces a new dependency on historical data quality. A test that has never failed may still be important. A flaky or under-specified test can distort the model.</p>

<p>I would separate the possible tools into three layers:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
</pre></td><td class="rouge-code"><pre>static graph facts
  -&gt; symbols, routes, calls, imports, ownership

historical evidence
  -&gt; test duration, failures, changed files, flaky examples

AI interpretation
  -&gt; explain the likely blast radius and recommend coverage
</pre></td></tr></tbody></table></code></pre></div></div>

<p>The first layer should be as deterministic as possible. The second should be measured rather than invented. The third can help us reason, but should show its evidence and uncertainty.</p>

<h2 id="a-safer-ai-assisted-test-decision">A safer AI-assisted test decision</h2>

<p>I would not begin with an agent that is allowed to skip tests. I would begin with an agent that produces a reviewable proposal:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
</pre></td><td class="rouge-code"><pre>Inputs:
  diff, changed symbols, routes, graph paths, test history

Output:
  selected test categories
  likely affected tests
  high-risk paths not covered
  reasons and confidence
  tests that still must run unconditionally
</pre></td></tr></tbody></table></code></pre></div></div>

<p>For example:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
</pre></td><td class="rouge-code"><pre>Changed: Api::CoursesController#update

Graph evidence:
  route -&gt; controller -&gt; authorization -&gt; Course#update
  controller enqueues CourseIndexJob

Recommended:
  request specs for courses
  authorization specs
  CourseIndexJob specs
  one full smoke path

Mandatory:
  RuboCop
  dependency/security checks
  release-level full suite

Confidence:
  medium: dynamic dispatch and shared concerns are not fully resolved
</pre></td></tr></tbody></table></code></pre></div></div>

<p>The agent is useful because it gathers and explains relationships. It is not the authority that decides whether production is safe.</p>

<p>That authority should remain in explicit CI policy, source-backed tests, and human-reviewed changes to the policy itself.</p>

<h2 id="the-graph-can-be-wrong">The graph can be wrong</h2>

<p>There are several reasons not to overtrust this approach:</p>

<ul>
  <li>Ruby metaprogramming can hide call relationships.</li>
  <li>Rails conventions can connect code without an obvious direct call.</li>
  <li>Dynamic routes and serializers may not be resolved completely.</li>
  <li>Tests can be badly named or cover less than their file path suggests.</li>
  <li>Historical failures can reflect infrastructure rather than application behaviour.</li>
  <li>A generated explanation can sound more certain than the underlying graph.</li>
</ul>

<p>This is why I like the idea of evidence layers. The system should distinguish a parsed route from an inferred relationship, and an observed test failure from a model prediction.</p>

<p>The source code and the test results remain authoritative. The graph is an index. The AI explanation is a navigation and reasoning layer.</p>

<h2 id="what-i-would-implement-next">What I would implement next</h2>

<p>The next practical step is not a full autonomous test selector. It is a report-only mode.</p>

<p>For each pull request:</p>

<ol>
  <li>Parse the diff and identify changed symbols.</li>
  <li>Query the code graph for callers, routes, jobs, and dependencies.</li>
  <li>Look up recent test history and failed examples.</li>
  <li>Produce a proposed test set and a confidence score.</li>
  <li>Compare the proposal with the policy-selected level.</li>
  <li>Run the stricter of the two decisions.</li>
  <li>Store the proposal and outcome for later review.</li>
</ol>

<p>After enough runs, we can measure whether the recommendations are useful:</p>

<ul>
  <li>Did the proposed tests catch failures earlier?</li>
  <li>How often did the graph miss a necessary test?</li>
  <li>How much time did the prioritised path save?</li>
  <li>Did the full suite catch failures that the proposal missed?</li>
  <li>Did the additional complexity make CI harder to understand?</li>
</ul>

<p>Only after that evidence exists would I consider allowing a tool to reduce required coverage for a narrow class of changes. Even then, releases and high-risk paths should remain conservative.</p>

<h2 id="the-principle">The principle</h2>

<p>The best CI system is not the one with the cleverest test selector. It is the one that makes a safe decision quickly, explains why it made that decision, and leaves enough evidence to improve the next decision.</p>

<p>Risk-based gates give us a policy.</p>

<p>Exact-context caches give us reproducible evidence and faster failure feedback.</p>

<p>Code graphs give us a better map of what a change might affect.</p>

<p>AI can help interpret that map, especially when the architecture is difficult to hold in one person’s head. But it should not turn an uncertain guess into a green checkmark.</p>

<p>The useful progression is:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
</pre></td><td class="rouge-code"><pre>full suite baseline
  -&gt; explicit risk policy
  -&gt; exact-context evidence cache
  -&gt; graph-assisted recommendations
  -&gt; measured, reviewable automation
</pre></td></tr></tbody></table></code></pre></div></div>

<p>That is slower than promising magic and much faster than debugging a production regression caused by a test suite that quietly stopped looking in the right place.</p>]]></content><author><name>Joey Wang</name></author><category term="engineering" /><category term="ai" /><category term="ci" /><category term="testing" /><category term="rails" /><category term="github-actions" /><category term="ai" /><category term="code-graph" /><summary type="html"><![CDATA[What I learned from splitting Rails CI by risk, caching exact test results, and exploring whether AI and code graphs can make test selection safer.]]></summary></entry><entry><title type="html">When a Green Deployment Still Serves a 500</title><link href="https://joeywang.github.io/posts/when-a-green-deployment-still-serves-a-500/" rel="alternate" type="text/html" title="When a Green Deployment Still Serves a 500" /><published>2026-09-01T13:00:00+00:00</published><updated>2026-09-01T13:00:00+00:00</updated><id>https://joeywang.github.io/posts/when-a-green-deployment-still-serves-a-500</id><content type="html" xml:base="https://joeywang.github.io/posts/when-a-green-deployment-still-serves-a-500/"><![CDATA[<audio controls="" preload="metadata" src="/assets/audio/when-a-green-deployment-still-serves-a-500-summary.ogg">
  Your browser does not support the audio element.
</audio>

<p>A production deployment can be green and still be broken.</p>

<p>We recently deployed a new version of a Rails learning platform. The image built. The Kubernetes deployments rolled out. The pods became ready. The workers started.</p>

<p>Then somebody opened the login page and got a 500.</p>

<p>That is the kind of incident that makes a team question every green checkmark it has been relying on.</p>

<h2 id="the-symptom">The symptom</h2>

<p>The first request to the application redirected to the login page as expected. The login page itself returned 500.</p>

<p>The Rails error was an asset lookup failure. The page expected a CSS entry such as <code class="language-plaintext highlighter-rouge">mypage.css</code>, but the Webpacker manifest did not contain it.</p>

<p>The Kubernetes view of the world looked healthy:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
</pre></td><td class="rouge-code"><pre>rails       ready
migration   complete
scheduler   ready
worker      ready
</pre></td></tr></tbody></table></code></pre></div></div>

<p>The application view of the world was different:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
</pre></td><td class="rouge-code"><pre>GET /login  -&gt; 500
missing Webpacker asset -&gt; mypage.css
</pre></td></tr></tbody></table></code></pre></div></div>

<p>Both statements were true. The pods were healthy enough to pass their readiness checks. The application was not healthy enough for a user to sign in.</p>

<h2 id="the-first-lesson-readiness-is-not-availability">The first lesson: readiness is not availability</h2>

<p>A readiness probe answers a narrow question: should this pod receive traffic?</p>

<p>It does not answer:</p>

<ul>
  <li>Can the login page render?</li>
  <li>Did the compiled asset manifest contain the entries that the templates reference?</li>
  <li>Can the browser download the CSS and JavaScript?</li>
  <li>Can a user complete the first meaningful action?</li>
</ul>

<p>Our rollout check was necessary, but it was not sufficient. We had verified process health, not application behaviour.</p>

<p>The missing check was small:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
</pre></td><td class="rouge-code"><pre>GET /status -&gt; 200
GET /login  -&gt; 200
critical CSS/JS assets -&gt; 200
</pre></td></tr></tbody></table></code></pre></div></div>

<p>A deployment that cannot pass those checks should not be reported as successful.</p>

<h2 id="what-made-the-failure-harder-to-diagnose">What made the failure harder to diagnose</h2>

<p>The application had recently moved its JavaScript dependencies from Yarn to pnpm. Webpacker 5 still carries assumptions about Yarn.</p>

<p>The production Docker build installed pnpm, but the old Webpacker Rake task still had a Yarn-oriented verification step. In addition, the build copied only part of the Rails configuration before compiling the assets. The pnpm-specific configuration was not reliably available at the point where Webpacker started.</p>

<p>There was another problem layered on top of that. The build command had been written in a form that allowed the asset compilation failure to be ignored:</p>

<div class="language-dockerfile highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre>bundle exec rake webpacker:compile || true
</pre></td></tr></tbody></table></code></pre></div></div>

<p>That transformed a useful build failure into a bad image that looked successful.</p>

<p>The manifest check we added later made the situation visible:</p>

<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
</pre></td><td class="rouge-code"><pre><span class="n">manifest</span> <span class="o">=</span> <span class="no">JSON</span><span class="p">.</span><span class="nf">parse</span><span class="p">(</span><span class="no">File</span><span class="p">.</span><span class="nf">read</span><span class="p">(</span><span class="s2">"public/assets/manifest.json"</span><span class="p">))</span>
<span class="nb">abort</span> <span class="k">unless</span> <span class="n">manifest</span><span class="p">.</span><span class="nf">key?</span><span class="p">(</span><span class="s2">"mypage.css"</span><span class="p">)</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>The right place to discover a missing asset is the builder, before the image is pushed or deployed. Not in a user’s browser after the rollout.</p>

<h2 id="recovery-took-too-long">Recovery took too long</h2>

<p>The first rollback path was also wrong for an incident.</p>

<p>Instead of switching Kubernetes back to an image that had already been built and verified, the rollback went through the production build process again. That meant waiting for dependency installation, native extensions, asset compilation, image creation, and rollout.</p>

<p>Rebuilding an old commit is useful when an old image no longer exists. It is a poor emergency rollback strategy when the known-good image is already in the registry.</p>

<p>The faster operation is an image switch guarded by the current version:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
</pre></td><td class="rouge-code"><pre>expected current image: bad-sha
rollback image:        known-good-sha
</pre></td></tr></tbody></table></code></pre></div></div>

<p>If the cluster is no longer running the expected bad version, the rollback should stop rather than overwrite an unrelated change.</p>

<p>The rollback should then update the application, migration, scheduler, and worker deployments, wait for each rollout, and run the same application smoke checks used after deployment.</p>

<p>A rollback is not complete when <code class="language-plaintext highlighter-rouge">kubectl</code> accepts the command. It is complete when the service is serving the known-good image and the user-facing checks pass.</p>

<h2 id="the-changes-we-made">The changes we made</h2>

<p>We made the build and release path more conservative in four ways.</p>

<h3 id="1-the-asset-build-now-fails-closed">1. The asset build now fails closed</h3>

<p>The production image no longer ignores Webpacker errors. The build must compile the assets and produce a manifest containing the critical login stylesheet.</p>

<p>The pnpm integration is explicit. The builder passes a known Webpack executable path to Webpacker and invokes the compiler without the old Yarn-only verification task.</p>

<p>This means a broken manifest stops the build. That is exactly what we want.</p>

<h3 id="2-the-deployment-has-an-application-smoke-gate">2. The deployment has an application smoke gate</h3>

<p>After the migration and rollout, the release checks:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
</pre></td><td class="rouge-code"><pre>/status
/login
critical CSS asset
critical JavaScript asset
</pre></td></tr></tbody></table></code></pre></div></div>

<p>A pod can be ready while an application is broken. The smoke gate checks the path a user actually takes.</p>

<p>This is not a replacement for feature tests. It is a fast last line of defence between deployment and traffic.</p>

<h3 id="3-rollback-uses-an-existing-immutable-image">3. Rollback uses an existing immutable image</h3>

<p>The rollback procedure now targets an existing image identified by its full commit SHA. It checks the current image before changing anything and waits for all selected deployments to become ready.</p>

<p>It does not rebuild the old source during the incident.</p>

<p>That distinction matters. Build pipelines optimise for repeatability. Emergency rollback optimises for time and a known result.</p>

<h3 id="4-ci-spends-test-time-according-to-risk">4. CI spends test time according to risk</h3>

<p>We also changed the CI model. Not every change needs the same test scope.</p>

<p>A documentation-only change should not start a database and a long RSpec suite. A user-facing feature should run focused tests and a basic browser smoke. A database, dependency, CI, or shared test-support change should escalate to a broader suite.</p>

<p>The current levels are:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
</pre></td><td class="rouge-code"><pre>L0  changed files and static checks
L1  affected tests
L2  basic E2E smoke
L3  full RSpec and integration tests
L4  full E2E
L5  performance tests
</pre></td></tr></tbody></table></code></pre></div></div>

<p>The promotion path from <code class="language-plaintext highlighter-rouge">main</code> to production remains strict. Faster feedback on a feature branch must not become weaker release confidence.</p>

<h2 id="caching-failures-without-lying-to-ourselves">Caching failures without lying to ourselves</h2>

<p>The next part of this work is test-result reuse.</p>

<p>A failed test result is useful evidence. It can tell us which examples to rerun first. But a cached pass is not proof that a new commit is safe.</p>

<p>Our result context therefore includes more than a branch name:</p>

<ul>
  <li>exact commit SHA;</li>
  <li>test suite and command;</li>
  <li>Ruby, Node, and pnpm versions;</li>
  <li>dependency lockfile hashes;</li>
  <li>schema and test configuration fingerprints.</li>
</ul>

<p>A failed-example retry is allowed only when the context matches. The first attempt and retry remain separate in the CI report.</p>

<p>The rule is simple:</p>

<blockquote>
  <p>Use cached failures to find the problem faster. Never use cached passes to bypass the required tests.</p>
</blockquote>

<h2 id="what-i-would-check-next-time">What I would check next time</h2>

<p>When a deployment reports success, I want to see evidence from three layers:</p>

<ol>
  <li><strong>Build:</strong> the image contains the required runtime and compiled assets.</li>
  <li><strong>Platform:</strong> migrations and workloads roll out with the expected image SHA.</li>
  <li><strong>Application:</strong> <code class="language-plaintext highlighter-rouge">/login</code>, critical assets, and one meaningful user path work.</li>
</ol>

<p>If one layer is missing, the release is only partially verified.</p>

<p>I also want a rollback drill. A rollback procedure that has never been exercised is a paragraph in a document, not an operational capability. The team should know which image is known good, how to switch to it, how long the operation takes, and what evidence proves recovery.</p>

<h2 id="the-uncomfortable-conclusion">The uncomfortable conclusion</h2>

<p>The incident was not caused by one exotic failure. It came from ordinary assumptions lining up in the wrong direction:</p>

<ul>
  <li>a package-manager migration left an older build tool assumption behind;</li>
  <li>the build tolerated a failed asset compilation;</li>
  <li>readiness checks did not exercise the login page;</li>
  <li>rollback rebuilt instead of switching to an existing image.</li>
</ul>

<p>Each assumption was understandable in isolation. Together they produced a deployment that was green in the pipeline and broken for users.</p>

<p>The fix is not to add more green badges. It is to make each badge answer a precise question, fail when that question has a bad answer, and keep a fast path back to a version that has already worked.</p>

<p>That is the standard I want from production engineering: not the appearance of safety, but evidence that the system can start, serve a real request, and recover when the next change goes wrong.</p>]]></content><author><name>Joey Wang</name></author><category term="engineering" /><category term="rails" /><category term="ci" /><category term="deployment" /><category term="reliability" /><category term="webpacker" /><category term="incident-response" /><summary type="html"><![CDATA[A REX production incident exposed the gap between a successful Kubernetes rollout and a working application, and what we changed to close it.]]></summary></entry><entry><title type="html">Recovering a hacked WordPress site: why I rebuilt instead of patching</title><link href="https://joeywang.github.io/posts/wordpress-security-incident-rebuild/" rel="alternate" type="text/html" title="Recovering a hacked WordPress site: why I rebuilt instead of patching" /><published>2026-08-31T23:00:00+00:00</published><updated>2026-08-31T23:00:00+00:00</updated><id>https://joeywang.github.io/posts/wordpress-security-incident-rebuild</id><content type="html" xml:base="https://joeywang.github.io/posts/wordpress-security-incident-rebuild/"><![CDATA[<p>A WordPress site I maintained was compromised. The investigation found unauthorised administrator accounts and malicious changes across several parts of the application.</p>

<p>By that point, resetting a password and deleting a suspicious plugin would not have been enough. I needed a reason to trust the code and database again.</p>

<p>I chose to rebuild from a reviewed pre-incident baseline, test a separate recovery candidate, and replace the affected installation through a controlled cutover. The recovery also uncovered ordinary application problems: missing plugin activation, broken images, and a theme widget displaying content it should not have displayed.</p>

<p>Those repairs were part of getting the site back, too.</p>

<p><em>This account omits identifying details, incident dates, infrastructure information, and exploit mechanics. It describes the recovery decisions and checks recorded at the time, rather than claiming anything about the site’s current security.</em></p>

<h2 id="establish-what-the-evidence-supports">Establish what the evidence supports</h2>

<p>The investigation drew on access logs, a copy of the affected code, and database records. Each answered a different question.</p>

<p>The logs helped establish a sequence of administrative activity. The code showed malicious modifications. Database records showed unauthorised accounts and changes to application state. Together, they supported a much stronger conclusion than any isolated request or suspicious filename.</p>

<p>They did not establish the original entry point conclusively.</p>

<p>That distinction matters. Evidence of an attacker using administrative access does not, by itself, explain how they obtained it. A stolen password, an earlier vulnerability, and an existing backdoor are different explanations. I could not honestly choose one just because it made the story easier to tell.</p>

<p>I also could not use a successful recovery to conclude that no information had been accessed. Restoring an application and assessing the impact of a compromise are separate jobs.</p>

<h2 id="preserve-the-affected-state-before-replacing-it">Preserve the affected state before replacing it</h2>

<p>Before destructive cleanup, I preserved copies of the code, database, and available logs. Backup verification and independently stored copies gave me something to return to during the investigation.</p>

<p>There is an important distinction between an evidence copy and a recovery copy. The affected installation was useful for understanding what happened. That did not make it suitable to put back online.</p>

<p>In a live incident, containment may need to happen immediately to protect visitors and prevent further changes. Preserving evidence should support that response, not become an excuse to leave a harmful site running.</p>

<p>A checksum has a similarly limited purpose. It can show that a copied archive matches the original archive. It cannot show that the original archive was free of malicious code.</p>

<h2 id="why-i-chose-a-rebuild">Why I chose a rebuild</h2>

<p>Malicious changes were spread across the installation rather than confined to one removable component. That made an in-place cleanup difficult to verify. Deleting everything I had found would still leave the question of what I had missed.</p>

<p>A separate recovery candidate gave me a clearer basis for comparison. I could review the earlier baseline, check the expected users and application state, and test the candidate before switching the live site over.</p>

<p>An older backup is not automatically trustworthy. It may predate the visible symptoms without predating the compromise. It may also contain outdated software. A sound recovery process needs both a reviewed baseline and supported software from trusted sources; blindly restoring an old archive can restore old vulnerabilities along with the content.</p>

<p>The same caution applies to the database. Replacing application files does not remove an unauthorised account or repair altered settings. The recovery had to consider code and data together.</p>

<h2 id="keep-the-recovery-candidate-separate">Keep the recovery candidate separate</h2>

<p>I prepared the candidate without immediately overwriting the affected installation. This preserved the evidence and allowed checks before cutover.</p>

<p>Code and database state were treated as one release. A tested code directory connected to the wrong database state would not have been the candidate I had verified.</p>

<p>Before destructive database cleanup, I made and verified another backup. This mattered because the recovery procedure changed once the old data was removed. Reversing a directory change would no longer be enough; restoring data would also be necessary.</p>

<p>For a security incident, a rollback plan must be explicit about where it leads. It should provide a safe maintenance state or a reviewed recovery version. Re-enabling a known-compromised installation is not a safe fallback simply because it is convenient.</p>

<h2 id="the-restored-site-still-needed-repair">The restored site still needed repair</h2>

<p>After the code switch, some pages did not behave as expected.</p>

<p>Required plugins were present on disk but were not active in the restored application state. That left page components and a contact form displaying incorrectly. Image delivery also needed attention, and an empty theme widget still displayed built-in default content.</p>

<p>These were useful reminders of how much behaviour sits outside the files themselves. A plugin directory can exist without its functionality being available. A page can return a successful response while showing broken content.</p>

<p>The recorded follow-up checks confirmed that affected page components and the contact form rendered again, and that the checked images loaded correctly. That is a narrower claim than saying every workflow passed. A visible form does not prove submission works or that its email reaches the intended inbox.</p>

<p>For a recovery checklist, I would include representative pages, media, login and permissions, form submission, email delivery, and any external integrations the site depends on. Each needs its own observed result.</p>

<h2 id="what-the-recovery-checks-proved">What the recovery checks proved</h2>

<p>The recovery record showed that the replacement site served the checked pages, expected accounts remained, identified unauthorised accounts were absent, and known malicious components were no longer present in the candidate. Existing WordPress login sessions were invalidated as part of the response.</p>

<p>Those were useful results. They were not proof that every possible persistence mechanism had been eliminated or that all potentially exposed credentials had been replaced.</p>

<p>Likewise, a scan that finds none of the previously identified malicious patterns is evidence about those patterns. It is not a certificate that the whole environment is safe.</p>

<p>I want an incident report to say exactly what was checked and what passed. Recommendations should stay recommendations until someone implements and verifies them. Credential replacement, stronger authentication, least privilege, supported dependencies, and protected backups belong in a general recovery plan, but listing them is not evidence of completion.</p>

<h2 id="make-backups-testable">Make backups testable</h2>

<p>The backup work after the recovery focused on making failures visible and copies verifiable. Archive readability, checksums, and copies held separately from the hosting account are all useful checks.</p>

<p>They still do not replace a restore test.</p>

<p>A backup can be readable while missing something the application needs. An incremental archive can be valid while depending on another archive that has been lost. A database dump can import successfully while the restored application fails to behave correctly.</p>

<p>A small restore fixture was exercised during the follow-up work. That helped test the mechanics, but it was not an end-to-end rehearsal of the entire live site. Keeping that distinction in the record makes the next piece of work clear.</p>

<h2 id="monitor-without-making-automatic-destructive-changes">Monitor without making automatic destructive changes</h2>

<p>A follow-up integrity check was introduced to detect unexpected changes to application files and important administrative state. Its job was to report changes for review, not delete files on its own.</p>

<p>That separation was deliberate. Legitimate updates change files too. A detector needs an understood baseline, reliable alert delivery, and a person who can distinguish an approved change from something that needs investigation.</p>

<p>The baseline itself must come from a reviewed state. Otherwise, monitoring can faithfully tell you that an already-compromised installation has not changed.</p>

<h2 id="what-i-would-carry-into-the-next-recovery">What I would carry into the next recovery</h2>

<p>The most useful decision was to stop treating the task as a search for a few bad files. Once changes were spread across code and accounts, a separate, reviewed recovery candidate was easier to reason about than repeated edits to the affected installation.</p>

<p>The ordinary application checks mattered just as much to completing the work. Users need functioning pages and forms, not merely a server that responds.</p>

<p>I would use the same approach again: contain the incident, preserve what is needed for investigation, rebuild from reviewed sources, test the behaviour people depend on, and record the limits of the evidence. Recovery is easier to review when every claim has a check behind it and every unverified item remains visible.</p>

<h2 id="further-reading">Further reading</h2>

<ul>
  <li><a href="https://wordpress.org/documentation/article/faq-my-site-was-hacked/">WordPress: FAQ — My site was hacked</a></li>
  <li><a href="https://developer.wordpress.org/advanced-administration/security/hardening/">WordPress: Hardening WordPress</a></li>
</ul>]]></content><author><name>Joey Wang</name></author><category term="engineering" /><category term="wordpress" /><category term="security" /><category term="incident-response" /><category term="recovery" /><category term="backups" /><summary type="html"><![CDATA[Lessons from a WordPress recovery: preserve evidence, rebuild from a reviewed baseline, test real behaviour, and be precise about what the checks prove.]]></summary></entry></feed>