These are the cases I want an agentic development workflow to handle before I give it more autonomy. A longer planning document does not resolve them.
My earlier software factory design described the division of work between people, agents, reusable tasks, and individual runs. This article narrows that design to three contracts: what an approval authorises, what recovery must reconcile, and what counts as evidence.
The starting point was a document review, followed by an independent second-model review. That found design questions, not production results. The mechanisms below are proposals for a bounded pilot, not claims that I have deployed a complete enforcement system or measured a productivity gain.
A lifecycle diagram usually looks reassuring: plan, implement, test, review, deploy. The arrows hide a question: does reaching the next state grant permission to perform its action?
It should not.
I can authorise an agent to implement a fix without authorising it to push a branch. I can approve a merge without approving a production deployment. A workflow needs to represent those distinctions even if one person makes every decision.
Anthropic’s AI-native SDLC playbook proposes a connected lifecycle built around committed artifacts and feedback. I find that useful as an organising model. Its claims about throughput are not evidence that my own workflow is faster, and I would not turn every small fix into a separate intent, specification, plan, and decision document.
For a small, reversible change, I would start with one record:
These can be sections of a ticket. A schema migration or security-sensitive change needs more detail. Calling it a chore does not lower its risk.
Consider an illustrative sequence. A reviewer approves candidate A. The agent adds a follow-up commit, producing candidate B. The dashboard still shows a green check, and the conversation still contains “approved”.
Nothing has necessarily gone wrong with the people. The record has lost the relationship between the decision and its subject.
An approval should identify the work, the exact candidate, the action, and the target. An evidence record should identify what was actually tested. The following YAML is a proposed shape, not an executable configuration. Angle-bracket values are placeholders, not real identifiers:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
approval:
task_id: "<task identifier>"
repository: "<repository identifier>"
candidate_sha: "<full commit object ID>"
spec_version: "<versioned acceptance contract>"
action: "merge"
target: "<target branch>"
expected_target_sha: "<full target object ID>"
approver_ref: "<authenticated human approval reference>"
evidence_refs:
- "<immutable evidence record>"
evidence:
tested_sha: "<full commit object ID>"
environment_ref: "<versioned test environment>"
config_ref: "<relevant configuration version>"
command_ref: "<test invocation>"
result: "passed"
artifact_ref: "<access-controlled output reference>"
A deployment needs a deployment-specific record, including the artifact digest and destination. A merge approval is not a reusable deployment credential.
The equality checks are only the beginning. A field called approver_ref proves nothing if the executing agent can write it. The execution boundary must authenticate the approval through a record or service that the worker cannot impersonate. The same applies to evidence: a worker-authored result: passed is not an independent test result.
I would invalidate the affected approval or evidence when its candidate, acceptance contract, or relevant target context changes. That does not mean reapproving the entire task after an unrelated spelling correction. It means defining which changes affect which claims.
The tested object also needs care. A CI run against a synthetic merge candidate is not automatically a run against the PR head, and neither is automatically proof about the eventual deployed artifact. Record the actual object and its relationship to the candidate rather than relabelling it for convenience.
sequenceDiagram
participant H as Human
participant G as Trusted gate
participant E as Evidence store
participant T as Target
H->>G: Approve candidate A, action and target
G->>E: Read authenticated approval and applicable evidence
alt Candidate or relevant context changed
E-->>G: Mismatch or stale evidence
G-->>H: Refuse and request refreshed review
else Candidate and context match
E-->>G: Evidence for candidate A
G->>T: Conditional write against expected target
alt Target moved
T-->>G: Reject stale precondition
G-->>H: Reconcile before proceeding
else Preconditions hold
T-->>G: Operation reference
G->>T: Read back exact target
T-->>G: Observed state
G-->>H: Report result or uncertainty
end
end
Checking the branch and then writing later leaves a race. The final operation needs a service-supported precondition or an equivalently protected executor. If the target cannot provide that guarantee, I would not pretend the diagram provides it.
A Git worktree helps keep unrelated file changes apart. It does not confine credentials, network access, or the authority to merge. Those require a separate security boundary.
A useful handoff records the branch, candidate, assumptions, completed checks, and next action. That is enough to resume many read-only steps. It is not enough to resume an uncertain external write.
Suppose an agent submits an operation and loses the connection before recording the response. There are three states worth distinguishing:
A timeout belongs in the third category. A temporarily empty lookup may belong there too, especially when the remote system is eventually consistent.
Before an authorised write, I would persist an operation record containing a stable operation ID, the exact target, the request identity, and the applicable approval. If that record cannot be saved, the write should not proceed. Afterward, the record gets the remote identifier and exact-target readback.
On recovery, the first action is a query, not a repeat of the previous command.
flowchart TD
A["Read durable intent and approval"] --> B["Query exact remote operation and target"]
B --> C{"Known outcome?"}
C -->|"Confirmed success"| D["Record evidence; do not repeat"]
C -->|"Confirmed not executed"| E{"Authorised and safe to retry?"}
E -->|"Yes, within budget"| F["Retry under API contract"]
E -->|"No"| H["Stop for human reconciliation"]
C -->|"Unknown"| G{"Server-side idempotency sufficient?"}
G -->|"Yes, same request and valid approval"| F
G -->|"No or uncertain"| H
F --> B
D --> I["Update handoff and continue"]
A local operation ID does not make an external API idempotent. The service must support the relevant contract, including what happens when the same identifier is reused with different parameters or outside its retention window. Amazon’s Making retries safe with idempotent APIs explains this distinction through caller-provided request identifiers and reconciliation.
Read-only checks can usually have bounded retries. An in-scope test failure can enter a bounded repair loop. An unknown side effect, changed permission boundary, or exhausted budget should stop the run. “Ask a human after every failure” is too expensive; “retry until green” is too permissive.
The question is not whether another agent agrees with the implementation. It is whether the acceptance claim survives a check grounded outside the implementation’s own explanation.
For an illustrative learner-progress fix, I would distinguish these claims:
Those are useful but different results. Calling the browser test “E2E” does not extend its evidence beyond the mock.
The verifier should start with the original symptom and acceptance criteria, then inspect the necessary code, tests, and contracts. I would avoid leading with the implementer’s conclusion. I would not prevent the verifier from reading relevant context in the name of independence.
A second model is helpful for finding missing assumptions. It cannot turn a document review into runtime verification, and two models can share the same mistaken premise. Useful independent oracles include an agreed interface contract, a preserved regression case, or a known expected result supplied independently of the proposed fix.
Humans still need to inspect risk-sensitive diffs. Tests do not remove the need to review a permission change or a destructive migration. The aim is to make that review focused and well supported, not to declare code review obsolete.
I want to reconstruct why an action happened, what authorised it, and what was observed. I do not need every byte the agent saw.
Full session transcripts and command outputs can contain credentials, personal data, request headers, or customer content. Automatically committing them creates another system to secure and eventually clean up.
My default would be an allowlisted, redacted summary with controlled evidence references. Raw artifacts, when needed, belong in restricted storage with an owner, access policy, retention period, and deletion process. A path and a checksum do not protect the referenced file. Secret scanning is useful, but not a substitute for deciding what should be collected.
This also keeps the sources of truth clearer: Git owns code versions, the test system owns its observed result, the approval boundary owns the decision, and the task record links them. A knowledge index is a retrieval aid, not the authority to act.
A thin vertical slice is a good way to learn quickly. It is not proof that everything it does can be undone.
Git revert records changes that reverse earlier commits. It does not retract a notification, restore an external account, or repair data already consumed downstream.
For a consequential slice, I would name the recovery mechanism: code rollback, configuration disable, data restoration, compensating action, or forward repair. I would also identify the point after which an ordinary code rollback is insufficient.
Temporary compatibility layers need an exit condition. An expand-migrate-contract change may deliberately leave old and new paths in place. That is sensible only if someone owns the removal condition and review point. The cleanup can be a later approved slice; it should not be forced into the first change merely to make one PR look complete.
I would use one familiar repository, one real reversible regression, and one implementation stream. No infrastructure changes, data migration, or automatic production release. One task record should be enough.
Before starting, I would agree the implementation scope, acceptance oracle, human review budget, and stop conditions. Then I would reproduce the failure, make a meaningful red test green, run the focused suite and critical-boundary smoke, and ask for independent verification at the exact candidate.
Two failure exercises matter as much as the happy path: reject evidence from an old candidate, and resume safely after an interrupted read-only step. Any later test of uncertain writes should start with a controlled fixture, not a production experiment.
I would record human review time, elapsed time, rework, acceptance coverage, and external mutations. If there is no comparable historical baseline, the first task establishes one. A single successful run cannot demonstrate a lower defect rate or a reliable productivity improvement.
I would stop when the task crosses an unapproved security, data, or API boundary, when evidence cannot be reproduced, or when the agreed budget runs out. The next approval must come from the person who owns the decision, not from the agent’s confidence that another attempt will work.
The useful unit of progress is an accepted outcome with applicable evidence. More agent activity is easy to generate. Authority that survives code changes, recovery that survives lost responses, and evidence that answers the original question are harder. Those are the parts I would prove first.
]]>语音摘要：使用本地语音合成生成的中文简要讲解。
在之前的一人公司设计里，我把问题归结为协调：客户反馈、产品决策、代码、运行故障和下一步工作，最终都要回到同一个人的注意力里。
软件工厂是其中更具体的一段：一个已经值得做的问题，怎样变成可以检查、可以接手、必要时可以撤回的软件变更？
我的起点很实际。让一个 coding agent 改代码并不难。难的是让下一次执行知道上一次做到哪里，让审查者看到原始需求而不只是实现者的总结，让失败停在正确的位置，而不是靠更多提示词把流程往前推。
这篇文章整理的是我正在推敲的设计。它还不是一个经过端到端验证、可以无人值守运行的交付平台；文中的风险分级、返工上限和验收规则，都需要通过小规模试点检验。
Matt Pocock 的 Dictionary of AI Coding 用一个朴素的定义描述 software factory：通过触发事件自动启动 agent 会话。
按我的理解，一个定时任务，每次处理一种小型维护工作，留下一个等待人工审查的 PR，就已经符合这个定义。工厂不要求人退出所有决策。
这与 dark factory 有区别。按照同一份词典，后者指的是某个代码范围内，人不再阅读工厂生成的变更。依此理解，晚上无人看守地执行、早上由人审 PR，仍然是有人审查的软件工厂。
我目前想验证的是前一种模式。自动化负责准备和执行一部分工作，人保留产品选择、关键授权和发布责任。
设计里最容易混淆的是“任务”。它可能指一张工单，也可能指实现、审查这样的工作类型，还可能指某个 agent 正在进行的一次尝试。
下面是我对 Matt Pocock 软件工厂术语草案的改编，用于表达自己的交付约束，不是对原词表的逐项引用。
因此，一张工单可以经历调查、实现、审查和返工等多个 Run。它的持久状态不应该只存在于其中某次聊天里。
需要人回答问题时，也不必让一个 Run 一直等待。它应该记录问题和已有证据，结束执行，把工单停在“待澄清”或“待决策”。人工更新后，新的事件再启动下一次 Run。
这使恢复不必依赖某个会话仍然活着。
我不打算先建一个包办所有事情的新平台。
Git 已经能保存代码历史，CI 能记录检查结果，工单系统能承载需求和责任归属，可观测性系统能报告运行问题。重新复制这些数据，很容易得到几份互相矛盾的“最新状态”。
在这个设计里，Hermes 是协调和审批入口，连接已有系统：
1
2
3
4
5
6
7
8
9
10
11
12
13
问题来源 / 客户反馈 / 运行信号
↓
工单：请求、范围、验收条件
↓
控制平面：路由、权限、预算
↓
隔离执行 → 候选代码
↓
自动检查 → 独立审查
↓
人的决策与发布授权
↓
正常发布流程 → 结果验证
工单负责说明要解决什么；Git 负责说明代码是什么；CI 负责说明某个版本跑过哪些检查。Company OS 保存经过审阅的设计和决策，个人知识系统则保持自己的隐私边界。
控制平面记录这些对象的关联与执行历史，但不替它们宣布事实。一个 agent 说“测试通过了”，仍然需要对应的命令、退出状态和结果；说“已经发布”，也需要发布系统和运行环境的证据。
只有当现有系统之间确实缺少某项能力时，才值得增加新的组件。
我希望一张工单能串起下面这些问题：
1
2
3
4
5
6
7
request：为什么做？要达到什么结果？
scope：允许改什么？明确不做什么？
plan：怎样实现、检查和恢复？
decision：有哪些重要取舍，由谁决定？
change：实际改了什么，执行结果是什么？
audit：这个候选版本是否满足原请求？
debt：哪些剩余风险被明确接受？
这不意味着每次修改都要创建七个文件。小型文档变更可以把请求留在工单里，把变更和检查留在 PR 里。涉及接口、数据语义或重要业务规则时，才需要更完整的记录。
这些记录应该通过稳定工单 ID 和候选 commit SHA 关联，而不是把同一段描述复制到多个地方。
尤其是审查意见，必须说明它针对哪个版本。审查通过后又改了代码，旧意见就不能直接当作新候选的通行证。试点阶段，我倾向于每个新 SHA 都重新审查；将来是否允许对内容完全相同的 rebase 沿用结论，也应由明确的验证规则决定。
失败结果同样需要保留。最后一版测试通过，不应抹掉此前的失败、原因和修正过程。
我把三个决定分开处理：风险有多高、谁来执行、谁有权验收。
模型名称只能影响执行者选择。它不能扩大权限，也不能替代验收。
当前设计用四档风险组织检查：
工作类型提供默认清单，风险标记只能提高要求。“chore”不是低风险证明，“bug fix”也不能掩盖认证逻辑的变化。一般风险不明时至少按 L2 处理，数据分类不明或触及生产则按 L3 处理。
分级也不只发生在入口。实现中发现要扩大范围、增加依赖、修改测试预期、跳过测试、更新快照或放宽超时，都应该提出范围问题并重新评估，而不是由实现者为了完成任务自行决定。
工厂本身的控制规则也属于被保护的代码。修改触发器、审查规则或权限边界，至少按 L2 对待；削弱关键制动措施时按 L3 处理。
独立审查不能只是让模型再读一遍自己的总结。
我希望审查者拿到原始请求、验收条件、base SHA、candidate SHA、完整 diff 和实际检查结果。审查 Run 不参与实现，也不能一边修代码一边批准自己。
更换一个模型不自动产生独立性。审查者如果只看到实现者筛选过的描述，仍可能继承同一套错误假设。
同样，流程界面里存在 review 状态，并不证明审查是硬门槛。需要验证状态转换和权限控制，确认执行者不能绕过必需审查直接宣布完成。Hermes 的 Kanban 文档和 worker lanes 文档描述了卡片、运行记录、交接和完成契约等机制，但具体配置与执行语义仍需在试点版本上检查。
代码工作区的隔离也要说清楚。Git worktree 可以避免两个执行者把文件修改混在一起，却不限制进程读取宿主机文件、凭据或访问网络。需要安全沙箱时，必须另外设置操作系统层面的文件、网络、凭据和资源约束。
人的批准也必须是一个可以核验的事件，来自自动化不能冒充的身份。由 agent 自己填一个 approved: true 字段，不能算人工授权。试点中，L2/L3 变更全部由人审阅，较低风险变更仍做抽查；其中 L3 的恢复或回滚路径也需要实际测试。
一个流程失败后，最省事的实现往往是重新启动整个 agent。这样做容易反复调查已经知道的事实，也可能把权限故障误当成代码问题。
我希望每条反馈至少包含：问题类型、期望与实际、证据、严重程度、负责人和下一步。
例如，缺少验收条件应交回需求负责人；产品语义有冲突应交回决策人；测试失败应把最小复现、失败命令和结果交给实现者。外部服务不可用则暂停并记录恢复条件，不应靠继续改代码来消耗重试。
对于代码审查发现的问题，最短路径通常是：
1
2
3
4
5
实现 → 检查 → 审查
↓ 发现缺口
定向返工
↓
新候选 → 重新检查与审查
试点暂定每条触发链最多连续进行两次自动返工，然后交给人重新判断范围或方案。两次只是一个待测的运行约束，不是普遍最优值。基础设施重试计数，也不能默认等同于审查返工计数。
未满足的验收条件不能被改名为“技术债”后关闭。技术债只容纳经过明确接受、有负责人和复查条件的剩余风险。
每增加一个触发源，都要回答几个具体问题：
谁启动执行？它可以读写哪些系统？结束后触发什么？人在哪一步重新进入？重复事件如何避免创建重复工作？
例如，外部事件 ID 可以用于创建工单或卡片时的幂等键，但这不等于所有后续操作都已经幂等。创建分支、提交 PR、写入外部记录，都需要自己的去重和读回规则。
Runner 中断时，接手者至少要能从持久记录找到原始请求、当前候选版本、已经完成的外部操作、已有证据和阻塞原因。无法确认某次写入是否成功，就先查询目标状态，而不是直接再写一次。
这也是我把 Runner 看作可替换执行容量的原因。即使机器长期存在，恢复也不应依赖它的聊天上下文或唯一一份本地文件。
我不准备一开始就接入所有告警、所有仓库，再安排多个 agent 并行工作。对小团队而言，生成变更的速度很容易超过审查能力。
第一步应是手动选择一项非敏感、低风险、范围清楚的工作。在执行前固定验收条件，确认实现者和审查者的实际身份、权限与工作区，然后完整走一遍实现、检查、审查和人工决策。合并与部署仍是独立授权。
随后，在可丢弃环境中主动制造失败：加入一个已知缺陷，检查审查能否拦住；让必需检查失败，检查流程能否拒绝完成；在审查后修改候选，检查旧结论是否失效；重复发送事件，检查是否产生重复工作；让任务缺少必要信息，检查执行能否停止并准确提问。
这些测试通过，只能说明相应控制路径在测试条件下有效。它们不证明整个工厂可靠，更不证明它已经节省成本。
下一步才是少量真实任务，逐个执行，记录成功、失败、阻塞和放弃的案例。观察人工提示、审查、修复和维护的总投入，以及工具费用、漏检缺陷和最终结果。没有可比较的人工基线，就不声称节省了多少时间。
我希望这套设计最终减少的是上下文丢失、重复调查和无效返工。它是否值得继续建设，要看一个被接受的结果总共花了多少成本，以及它是否真的解决了值得解决的问题。
在得到这些证据以前，我宁愿先建设一个速度有限、停止行为经过测试的交付循环，也不急着把更多工作自动送进队列。
本文进一步展开了此前 Company OS 设计中的交付部分。风险层级、证据结构和试点约束是作者提出的工作方案，不代表上述项目自带的完整保证，也不是已验证的行业标准。
]]>Audio summary: a short English summary, generated with local text-to-speech.
In my earlier design for a one-person company, I kept coming back to coordination. Customer feedback, product decisions, code, operational failures and the next piece of work all compete for the same person’s attention.
The software factory covers a more specific part of that problem: once something is worth doing, how does it become a software change that someone can inspect, take over and, if necessary, roll back?
My starting point is practical. Getting a coding agent to change code is not especially difficult. The harder parts are letting the next execution pick up where the previous one stopped, giving reviewers the original request rather than just the implementer’s summary, and stopping failures in the right place instead of pushing the process forward with more prompts.
This article describes a design I am still working through. It is not an unattended delivery platform that I have validated end to end. The risk tiers, rework limits and acceptance rules below all need testing in a small pilot.
Matt Pocock’s Dictionary of AI Coding gives software factory a simple definition: automation that starts agent sessions in response to triggers.
My reading of that definition is that a scheduled job handling one kind of small maintenance task and leaving a PR for human review already qualifies. A factory does not require people to leave every decision to automation.
That differs from a dark factory. In the same dictionary, a dark factory is an area of a codebase where humans no longer read the changes the factory produces. On that reading, unattended execution overnight followed by human PR review in the morning is still a human-reviewed software factory.
The first model is the one I want to test. Automation prepares and carries out some of the work. People retain responsibility for product choices, consequential approvals and releases.
The word “task” is easy to overload. It can mean a ticket, a reusable kind of work such as implementation or review, or the particular attempt an agent is making right now.
The distinctions below are my adaptation of Matt Pocock’s draft software-factory vocabulary, not a verbatim glossary. I use them to express my own delivery constraints.
A single ticket can therefore go through several Runs for investigation, implementation, review and rework. Its durable state should not live only in one of those conversations.
When a human needs to answer a question, the Run does not have to wait indefinitely. It should record the question and available evidence, end execution, and leave the ticket in a state such as “needs clarification” or “needs decision”. A later event starts a fresh Run after the human updates it.
Recovery then does not depend on keeping a particular session alive.
I do not want to start by building a new platform that does everything.
Git already records code history. CI records check results. An issue tracker holds requests and ownership. Observability systems report operational problems. Copying all of that data into another system can easily produce several contradictory versions of the “latest state”.
In this design, Hermes is the coordination and approval entry point that connects the existing systems:
1
2
3
4
5
6
7
8
9
10
11
12
13
Problem sources / customer feedback / operational signals
↓
Ticket: request, scope, acceptance criteria
↓
Control plane: routing, permissions, budget
↓
Isolated execution → candidate code
↓
Automated checks → independent review
↓
Human decisions and release authorization
↓
Normal release process → outcome verification
The ticket explains what needs solving. Git identifies the code. CI records which checks ran against a particular version. Company OS holds reviewed designs and decisions, while the personal knowledge system keeps its own privacy boundary.
The control plane records the relationships between these objects and the execution history. It does not declare facts on their behalf. An agent saying “the tests passed” still needs the corresponding command, exit status and results. Saying “it has been deployed” needs evidence from the deployment system and the running environment.
A new component is worth adding only when a necessary capability is genuinely missing between the existing systems.
I want a ticket to connect the answers to these questions:
1
2
3
4
5
6
7
request: Why are we doing this? What outcome do we want?
scope: What may change? What is explicitly out of scope?
plan: How will we implement, check and recover?
decision: What consequential trade-offs exist, and who decides?
change: What actually changed? What did execution return?
audit: Does this candidate satisfy the original request?
debt: Which remaining risks have been explicitly accepted?
This does not mean creating seven files for every change. For a small documentation update, the request can stay in the ticket, with the change and checks in the PR. Interface changes, data semantics and important business rules call for a fuller record.
These records should link through a stable ticket ID and candidate commit SHA, rather than repeat the same description in several places.
Review findings, in particular, must identify the version they apply to. If code changes after approval, the old review cannot automatically authorize the new candidate. During the pilot, I would require a fresh review for every new SHA. Any later policy for carrying a review across a content-identical rebase would need explicit verification rules.
Failed results need to survive too. A passing final test run should not erase earlier failures, their causes or the changes made to address them.
I treat risk classification, executor selection and acceptance authority as separate decisions.
A model name can influence who performs the work. It cannot expand permissions or replace acceptance checks.
The current design organizes checks into four risk tiers:
The work type supplies a default checklist; risk flags can only raise the requirements. A “chore” label is not evidence of low risk, and calling something a “bug fix” does not make an authentication change harmless. Unknown general risk defaults to at least L2. Unknown data classification or any production touch defaults to L3.
Classification also continues during execution. Discovering a need to expand scope, add a dependency, change test expectations, skip tests, update snapshots or increase timeouts should raise a scope question and trigger reassessment. The implementer should not make those decisions unilaterally just to finish the job.
The factory’s own controls are protected code too. Changes to triggers, review rules or permission boundaries are at least L2. Weakening a critical stopping mechanism makes them L3.
Independent review cannot mean asking a model to reread its own summary.
I want the reviewer to receive the original request, acceptance criteria, base SHA, candidate SHA, complete diff and actual check results. The review Run does not implement the change and cannot approve its own edits while repairing the code.
Switching models does not automatically create independence. If a reviewer sees only the implementer’s selected account of the work, it can inherit the same faulty assumptions.
Likewise, a review state in a workflow interface does not prove that review is an enforced gate. State transitions and permission controls need testing to establish that an executor cannot bypass a required review and declare completion. Hermes’s Kanban documentation and worker lanes documentation describe mechanisms for cards, run history, handoffs and completion contracts, but their configuration and execution semantics still need checking in the version used for the pilot.
Workspace isolation needs a precise definition as well. A Git worktree can keep two executors’ file edits apart. It does not restrict a process from reading host files or credentials, or accessing the network. A security sandbox requires separate operating-system-level controls over files, network access, credentials and resources.
Human approval must also be a verifiable event from an identity that automation cannot impersonate. An agent filling in an approved: true field does not count as human authorization. During the pilot, humans review every L2/L3 change and spot-check lower-risk changes. L3 recovery or rollback paths also need actual testing.
After a workflow fails, the easiest implementation is often to restart the entire agent. That can repeat investigations into facts already known, or treat a permissions failure as a code problem.
I want each piece of feedback to include at least the problem type, expected and actual behavior, evidence, severity, owner and next action.
Missing acceptance criteria should go back to the request owner. Conflicting product semantics should go to the decision-maker. A test failure should return a minimal reproduction, the failing command and its results to the implementer. An unavailable external service calls for a pause and recorded recovery conditions, not more code changes to use up the retry budget.
For findings from a code review, the shortest route is usually:
1
2
3
4
5
Implementation → checks → review
↓ finding
Targeted rework
↓
New candidate → fresh checks and review
The pilot provisionally allows at most two consecutive automated rework Runs per trigger chain before a human reassesses the scope or approach. Two is an operating constraint to test, not a universally optimal number. Infrastructure retry counts should not be assumed to measure review rework either.
An unmet acceptance criterion cannot be renamed “technical debt” so the ticket can close. Debt covers only explicitly accepted remaining risks with an owner and conditions for revisiting them.
Every new trigger source needs answers to a few specific questions:
Who starts execution? Which systems may it read and write? What does completion trigger next? Where does a human re-enter? How do duplicate events avoid creating duplicate work?
An external event ID, for example, can be an idempotency key when creating a ticket or card. That does not make every subsequent operation idempotent. Branch creation, PR creation and writes to external records each need their own deduplication and read-back rules.
If a Runner stops, whoever takes over should at least be able to recover the original request, current candidate, completed external actions, available evidence and blocking reason from durable records. If it is unclear whether a write succeeded, query the target state before writing again.
That is why I treat a Runner as replaceable execution capacity. Even when the machine persists, recovery should not depend on its conversation context or the only local copy of a file.
I do not plan to begin by connecting every alert and repository, then running several agents in parallel. For a small team, generating changes can easily outpace the capacity to review them.
The first step should be one manually selected, non-sensitive, low-risk piece of work with a clear scope. Fix the acceptance criteria before execution. Confirm the implementer’s and reviewer’s actual identities, permissions and workspaces. Then walk through implementation, checks, review and a human decision. Merge and deployment remain separately authorized actions.
Next, deliberately introduce failures in a disposable environment. Add a known defect and see whether review catches it. Fail a required check and see whether the workflow refuses completion. Change the candidate after review and see whether the old approval becomes invalid. Deliver the same event twice and look for duplicate work. Omit necessary information and see whether execution stops and asks a precise question.
Passing those tests would show only that the corresponding control paths worked under the test conditions. It would not establish that the whole factory is reliable, much less that it has saved money.
Only then would I move to a small number of real tasks, one at a time, recording successes, failures, blockers and abandoned attempts. I would track the total human effort spent prompting, reviewing, repairing and maintaining the system, alongside tool costs, missed defects and final outcomes. Without a comparable manual baseline, I would not claim a particular time saving.
I hope this design will reduce lost context, repeated investigation and unproductive rework. Whether it is worth building further depends on the total cost of an accepted result, and whether that result solves a problem worth solving.
Until I have that evidence, I would rather work toward a delivery loop with limited throughput and tested stopping behavior than automatically send more work into the queue.
This article develops the delivery portion of my earlier Company OS design. The risk tiers, evidence structure and pilot constraints are proposed working rules. They are neither complete guarantees supplied by the projects above nor validated industry standards.
]]>Audio summary: a short spoken version of this post.
In an earlier post on the changing developer role, I described how AI shifts a developer’s work from writing code toward designing and validating systems. I still think that is right, but it leaves a harder question open: if a model can produce the code, sketch the architecture, and propose the workflow, what is the engineer responsible for?
My answer is not another skill for the list. It is responsibility for an outcome that matters: knowing which problem is worth solving, making the result trustworthy where it runs, and finding out whether it helped. Code is part of that work. It is not the result.
Take a hypothetical support team considering AI-generated replies. A model that writes fluent drafts is a promising component. It is not evidence that support got better.
Someone still has to decide what information a reply may rely on, which policies apply, when a human must review it, how odd cases are handled, and how staff correct mistakes. Then someone has to find out whether it helped. Replies may be more accurate. Or the team may be sending more of them, faster, with the same errors.
These are not cleanup tasks for after implementation. They cross code, data, operating rules, and people, and together they decide whether the feature becomes a system people can use and trust. Ordinary software is no different: automate a step without knowing why it exists and you get an ineffective process that runs faster. “Can we build this?” is the easy question. “What should improve, for whom, and how would we know?” is the one that matters.
It is tempting to say the human part is designing the workflow. I do not think that holds. Models can already propose steps, integrations, handoffs, and failure cases. Calling workflow design permanently beyond AI repeats the old mistake of equating engineering with typing code, one level up.
What a model cannot do is answer for the choice. A plausible design still has to be tested against a specific organization, its users, and the cost of being wrong. What can run automatically, and what must stay reviewable or reversible? Who approves the change, and who looks after it once the demo is over? Someone has to make those calls and change course when the evidence disagrees. That is responsibility, not design skill.
Trust in an AI-enabled system does not come from a capable model. It comes from clear limits on what the system may do, checks suited to the risk, a way for people to inspect and correct its work, and a sensible response when it is uncertain or fails.
For one task, the right limit is a draft a person approves. For another, the system can act alone within a narrow, reversible scope. The choice depends on what an error costs, how good the checks are, and who lives with the result. This is not paperwork around the real engineering. It is the engineering. A demo that works once but cannot be checked, corrected, or operated is not finished.
Evidence is the other half of trust. Lines generated, agent steps completed, and impressive screenshots describe activity, not value. The measure has to follow the goal: fewer errors, faster turnaround, less repetitive work, or whatever the people affected care about. If I cannot say what evidence would show a change helped, I am not ready to call it done.
“Own the outcome” can sound like asking one engineer to answer for everything. That is not what I mean. Engineers rarely control the policy, the staffing, or whether the organization adopts a change. Product owners, domain experts, operators, leaders, and users all shape whether it succeeds. Holding someone accountable for decisions they cannot make is not ownership. It is blame looking for a target.
So the line I draw is this. Within my authority, I own the quality of the system, the honesty of what we claim about it, and the evidence for whether it works. Beyond that authority, my job is to make risks and trade-offs visible to the people who can decide, and to say plainly when a decision is not mine to make. The organization owes the other half: explicit ownership, involvement from the people affected, and enough authority for whoever is accountable for a risk to address it. An engineer can make consequences legible. They cannot replace the organization around the system.
There is a fair objection here. If the job is outcomes and trust, does deep technical craft matter less? Is domain expertise just one input among many?
The objection gets something important right. Craft and domain knowledge are how anyone notices that generated output is plausible but wrong: code that passes the obvious tests and mishandles the case that matters, a shortcut that breaks a real constraint, a metric that improves while users get worse service. Without that expertise, owning the outcome is a promise nobody can keep, because the failures that matter look correct.
Where I differ is on what follows. Expertise does not matter less. It matters more when it is aimed at the whole outcome instead of measured by how much code it produces.
AI will change the mix of engineering work. Some implementation gets faster, some tasks disappear, new failure modes appear, and the balance will differ by team and domain. Under all of that is the question I want to be able to answer for any system I put my name to: did it make something that matters better, within limits people understand, and can I show it?
Getting a model to produce code does not answer that. It is where the question starts.
]]>Audio summary: a short spoken version of this post.
A Sidekiq queue called default is easy to start with. It is also an easy place to hide a growing set of contradictory requirements.
An email, a payment reconciliation, a search-index refresh, a nightly report, and a cache warm-up can all be valid background jobs. They do not have the same urgency. They do not have the same tolerance for delay. They do not have the same answer to the question: “What happens if this job is enqueued twice?”
The queue name is not the strategy. It is only the first label.
The useful design work is to classify each job before deciding where it runs:
1
2
3
4
5
urgency How quickly does it need to start?
tolerance How bad is a delay or a missed run?
duplicates Is a second copy harmless, wasteful, or dangerous?
retry Should failure wait, retry, or alert somebody?
concurrency Can several copies run safely at the same time?
Once those answers are written down, Sidekiq’s queue modes and uniqueness options become much easier to use without turning sidekiq.yml into a collection of guesses.
I find it useful to separate two properties that are often treated as one.
Urgency is about time to start. A job may be urgent because a user is waiting for its result, or because a delayed action will make another workflow incorrect.
Tolerance is about how much delay or loss the business can accept. A low-urgency job may still have low tolerance for being lost. A nightly export can wait for an hour, but it may still need an operator-visible failure if it has not completed by morning.
That gives a rough first classification:
| Class | Urgency | Delay tolerance | Typical jobs |
|---|---|---|---|
| Critical | high | low | payment capture, access changes, time-sensitive notifications |
| Interactive | high | short | user-requested exports, document generation, webhook follow-up |
| Important | medium | moderate | search indexing, CRM sync, progress aggregation |
| Bulk | low | high | historical backfills, analytics imports, cache warming |
| Periodic | deadline-based | depends on deadline | daily reports, cleanup, reconciliation |
This is not a universal taxonomy. It is a prompt to make the trade-off visible.
A job’s class should also answer what happens when the system is busy. If bulk work consumes every thread, should an access-change job wait behind it? If the answer is no, the queues need separate capacity, not only different names.
A single queue is not neutral. It says that all jobs compete for the same capacity and that the order in which they are checked is good enough.
That may be perfectly reasonable for a small application. It becomes a problem when a slow or high-volume producer can delay a different kind of work.
A simple separation might look like this:
1
2
3
4
critical payments, security, account changes
interactive user-facing work with a short response expectation
normal ordinary application jobs
bulk imports, backfills, expensive maintenance
The names are less important than the ownership rules. A queue should have a reason to exist, a rough workload profile, and a way to tell whether it is being starved.
Sidekiq supports strict queue ordering and weighted queues. Strict ordering means a process checks queues in the declared order. Weighted queues give a queue more chances to be selected, but they do not guarantee a fixed percentage of completed work. The Sidekiq advanced options documentation describes both modes and the limitation that a process cannot mix them.
For example, a weighted process could be configured like this:
1
2
3
4
5
:queues:
- [critical, 6]
- [interactive, 3]
- [normal, 2]
- [bulk, 1]
This is a useful starting point when the queues share a worker pool and the application can tolerate probabilistic selection. It is not a service-level agreement. A busy critical queue can still consume capacity, and a low-weight queue can still wait for a long time under sustained load.
If a queue must remain available even when another workload is unhealthy, give it separate process capacity.
For example:
1
2
3
process A: critical, interactive
process B: normal
process C: bulk
Or, if critical work really must be checked first within its process:
1
2
process A: critical -> interactive
process B: normal -> bulk
This costs more operationally, but it makes the boundary easier to reason about. A large import cannot fill every worker slot needed for a payment-related job.
Strict ordering has a catch: lower queues can starve while higher queues remain busy. Weighted queues have a different catch: they improve sharing but do not provide a hard deadline. The choice depends on which failure is worse for the application.
I would rather have two deliberately sized processes than six queues that all run through one overloaded pool with no clear capacity model.
Queue selection cannot repair a job that is unsafe to run twice.
Sidekiq retries failed jobs, and a worker can lose its connection after the remote operation has succeeded but before the process records success locally. From the application’s point of view, the job may be attempted again even though the first attempt partially completed.
That is why idempotency belongs in the job design, not only in the queue configuration.
A job that records a payment, sends a message, or applies an external change should have a business-level idempotency key where the downstream system supports one. A job that updates a local projection should usually write the same result again safely rather than assuming it will run exactly once.
1
2
3
4
5
6
7
8
9
10
class RebuildSearchDocumentJob
include Sidekiq::Job
sidekiq_options queue: :normal, retry: 8
def perform(document_id)
document = Document.find(document_id)
SearchIndex.upsert(document.id, document.search_payload)
end
end
The important property here is not the number 8. It is that upsert and the document payload make a retry safe. Retry counts should follow the failure mode and the business deadline, not a copied default.
A uniqueness lock answers a different question:
Should I allow another copy of this job to be enqueued or executed while this copy is in progress?
Idempotency answers:
If this operation happens again, will the result remain correct?
A unique lock can reduce duplicate work. It cannot prove that a remote API call happened only once, and it cannot replace a durable business record or an idempotency key.
There are two common ways to get uniqueness in Sidekiq applications:
sidekiq-unique-jobs gem.They have different APIs and operational details, so a team should choose one deliberately and read the version-specific documentation. The examples below use sidekiq-unique-jobs terminology.
The gem provides several lock types. The useful question is not “Which lock is strongest?” It is “At what point do duplicates become a problem?”
Suppose a record is edited several times in a short period and each edit queues a search refresh. The latest refresh may make earlier refreshes unnecessary.
1
2
3
4
5
6
7
8
9
10
11
12
class RefreshSearchJob
include Sidekiq::Job
sidekiq_options \
queue: :normal,
lock: :until_executing,
on_conflict: :log
def perform(record_id)
SearchIndex.refresh(record_id)
end
end
An :until_executing lock prevents multiple copies from accumulating before the worker starts. This is appropriate when dropping a duplicate is acceptable because another job will produce the same current state.
It is not appropriate for a job where every event must be processed. In that case, model the event or use an outbox rather than treating a duplicate as disposable.
For a rebuild or synchronization that should not overlap with another copy, :until_executed holds the lock through execution:
1
2
3
4
5
6
7
8
9
10
11
12
13
class SyncCustomerJob
include Sidekiq::Job
sidekiq_options \
queue: :important,
lock: :until_executed,
on_conflict: :log,
retry: 6
def perform(customer_id)
CustomerSync.call(customer_id)
end
end
This can stop a thundering herd when several triggers request the same synchronization. It also means that a failed job may keep the lock while it retries, depending on the lock and retry behavior. That can be correct, but it can also delay a fresh request longer than the caller expects.
Some jobs are safe to enqueue more than once but unsafe to run concurrently. A generated report might be overwritten by two writers, or a third-party endpoint might reject simultaneous updates.
For that case, :while_executing is a server-side concurrency lock. It does not stop duplicate jobs from entering Redis. It stops them from doing the protected work at the same time.
That distinction matters operationally. A queue can still grow even when only one copy is executing.
For periodic work, :until_expired can be useful when the business rule is “at most one request per interval” rather than “one copy until successful completion”.
Be careful with long lock periods. A lock can prevent a legitimate recovery or manual rerun. Locks also need an expiry or a cleanup plan, because a process can die while a lock exists.
The Sidekiq Enterprise documentation makes the same broader point about uniqueness: treat it as best effort, use a bounded period, and keep the lock short. The open source gem has its own reaper and configuration choices, but the operational lesson is the same.
When a duplicate is detected, the result should be intentional:
:log is reasonable when the duplicate is harmless and should be discarded.:reject is useful when an operator needs a visible record that a required job was not accepted.:reschedule can be appropriate when the work must happen later, but it changes queue pressure and retry statistics.:replace fits jobs where the newest request supersedes an older pending request.The correct choice depends on whether the duplicate represents the same desired state or a separate event.
For example, two requests to rebuild the current search document may collapse into one. Two invoice events should not. The first is state convergence; the second is event processing. Giving both jobs the same uniqueness policy is a data-model mistake disguised as a queue setting.
Before adding a new worker, I would write a small record like this:
| Question | Example answer for a search refresh |
|---|---|
| User waiting? | No |
| Start within | A few minutes |
| Maximum useful delay | 30 minutes |
| Duplicate meaning | Same desired state |
| Concurrent execution safe? | Yes, if the index uses upsert |
| Retry | Yes, with bounded backoff |
| Queue | normal |
| Uniqueness | :until_executing |
| Conflict | Log and drop |
| Alert | Queue age or repeated failures |
For a payment capture, the answers would be different. The job might need a dedicated high-priority process, a durable idempotency key, an explicit failure alert, and no silent duplicate dropping.
Writing the answers down also exposes missing requirements. If nobody knows the maximum useful delay, it is difficult to choose a queue weight responsibly.
A healthy Sidekiq process can still be serving an unhealthy queue policy. I would monitor at least:
Queue latency is often more useful than total job count. A bulk queue can contain many jobs without being in trouble. A critical queue with a small number of old jobs may already be failing its purpose.
Alerts should follow the class of work. A five-minute delay may be harmless for a backfill and unacceptable for an access change. One global queue-latency threshold usually hides that distinction.
For a Rails application with mixed workloads, I would start conservatively:
critical or interactive queue only when there is a measured reason.The goal is not to produce a complicated queue topology. It is to stop unrelated jobs from competing as if they had the same urgency and the same failure semantics.
Sidekiq gives us useful mechanics: queues, weights, retries, capsules, and uniqueness locks. The application still has to decide what a job means, how late is too late, and whether a duplicate is an error, a no-op, or a separate event. Those decisions belong in the design of the job, not hidden in a YAML file.
Audio summary: a short spoken version of this post.
I recently put Google sign-in in front of a self-hosted web dashboard.
The dashboard was already running behind HTTPS and a reverse proxy. The first version used a shared password at the proxy layer. That worked, but it made the dashboard feel like a private server rather than an application with a proper identity boundary.
The target architecture was straightforward:
1
2
3
4
Browser
-> HTTPS reverse proxy
-> OAuth2 proxy
-> local dashboard
The dashboard itself did not need to learn about Google. The authentication layer could handle the OAuth flow, create a session cookie, and forward an authenticated request upstream.
The first login attempt failed immediately:
1
Error 400: redirect_uri_mismatch
This error is less mysterious than it looks.
An OAuth login has a round trip:
The return address in step three is the redirect URI. It is not merely a hint. Google compares it with the list registered on the OAuth client. The comparison is effectively exact.
These are different redirect URIs:
1
2
3
4
https://dashboard.example.com/oauth2/callback
https://dashboard.example.com/auth/callback
https://dashboard.example.com/oauth2/callback/
http://dashboard.example.com/oauth2/callback
The differences are small to a person reading them. They are not small to the OAuth provider.
The scheme, hostname, path, port, and trailing slash all matter.
The reverse proxy had one public hostname. The authentication proxy had another idea about its callback path. Google had a third value registered in the OAuth client.
Each individual component looked reasonable:
But OAuth does not care that the pieces are individually reasonable. It cares that the exact same URI appears in both places.
The useful debugging step was to inspect the authorization request generated by the running service rather than infer it from a configuration file. In the browser’s developer tools, I checked the redirect to Google’s authorization endpoint and decoded only its redirect_uri parameter. I did not copy or share the full authorization URL: it can contain transient state and identifying configuration.
The value to compare was the decoded callback URI:
1
https://dashboard.example.com/oauth2/callback
That was the value Google needed to find in the OAuth client’s Authorized redirect URIs list. Google’s web-server OAuth documentation requires the URI in the request to match a registered URI for that client.
I added the exact URI to the Google OAuth client:
1
https://dashboard.example.com/oauth2/callback
The following details mattered:
https, not http;127.0.0.1;After saving the OAuth client, the login flow worked without changing the dashboard application itself.
The authentication proxy and dashboard both remained on private upstream addresses behind the reverse proxy. Only the reverse proxy listened on the public HTTPS interface.
The resulting shape was:
1
2
3
4
Public HTTPS
-> reverse proxy
-> authentication proxy
-> dashboard
That distinction is important. Google authentication is not a replacement for network boundaries, TLS, rate limiting, or careful proxy configuration.
I kept the existing network allowlist as an additional boundary. Google identity answers “who is this?” The network policy answers “from where may this service be reached?” Those are different questions, and it is useful to keep both.
A Google OAuth client can authenticate Google accounts, but authentication alone does not always mean authorization.
If a dashboard is intended only for an organisation, the proxy should enforce the allowed email domain or organisation policy after Google returns the identity. In a generic configuration that might look like:
1
email_domains = ["example.org"]
This is not a substitute for checking the actual claims and provider configuration. It is one layer in the access policy.
For a small private dashboard, I prefer an organisation-restricted Google Workspace application where possible, plus an explicit allowed-domain check in the authentication proxy.
The working setup contains values that are useful to the machine but not useful to readers:
A technical explanation does not need those values. The important information is the relationship between the components and the verification method.
I also avoid copying an authorization URL into a ticket or chat. It contains state and can contain identifying information even when the client secret is not present.
When Google reports redirect_uri_mismatch, I now check these in order:
redirect_uri query parameter.This process is faster than starting with the assumption that Google OAuth itself is broken.
OAuth failures often happen at system boundaries. The identity provider knows one URL, the reverse proxy advertises another, and the application logs a third. Looking at only one configuration file creates false confidence.
The reliable source of truth is the request that actually leaves the running system.
For redirect errors, inspect the live authorization URL, extract the redirect URI, and compare it exactly with the provider’s registered value. Once I treated the URI as a strict contract instead of a descriptive label, the error became a two-minute configuration fix.
The dashboard was not failing to authenticate Google. Google was correctly refusing to send a credential to an address the application had not explicitly registered.
]]>A green CI run can still leave me with an uncomfortable question:
Did we run the right tests?
The opposite problem is easier to recognise. A pull request runs every test, every time, even when the change is a README edit or a small isolated helper. The feedback loop gets slower, the bill gets larger, and eventually somebody starts looking for ways around it.
That is how a safety system becomes a delivery tax.
Recently I applied a risk-based CI gate to Turtle, following improvements I had already made in REX. The change was deliberately modest. It did not pretend that a script could understand the entire architecture. It classified changes into test levels, kept RuboCop independent, and used the full suite for high-risk paths and releases.
The more interesting part came afterwards: caching test-result context and asking whether a code graph or an AI agent could help us decide what deserves testing in the first place.
The usual CI conversation is presented as a binary choice:
1
2
3
4
5
run everything every time
-> safe but slow
run only what looks relevant
-> fast but risky
Neither is good enough.
Running everything is a reasonable baseline, especially for a small repository. But it hides the fact that not all changes have the same risk. A documentation-only change does not need the same RSpec scope as a database migration, dependency lockfile change, or production release.
Running a guessed subset is more dangerous. File paths are not the same thing as runtime behaviour. A change in a serializer may affect an API response. A configuration change may alter boot behaviour. A seemingly local model callback may be used by a background job, an admin screen, and a mobile endpoint.
The useful goal is not “run fewer tests.” It is:
Make the smallest test decision that is still justified by evidence, and make the decision visible.
The Turtle workflow now has a small ci_gate job before RSpec. It classifies the pull request and publishes the decision in the GitHub Actions summary.
The levels are intentionally understandable:
| Level | Category | Typical coverage |
|---|---|---|
| L0 | Static-only | RuboCop and other static checks; documentation-only work can skip RSpec |
| L1 | Affected | Fast library, model, and helper specs |
| L2 | Smoke | Fast specs plus request, controller, and view specs |
| L3 | Full | The complete parallel RSpec suite |
The classifier considers the changed paths, the branch name, and optional labels such as risk:bugfix or risk:feat. High-risk paths, including dependencies, database changes, Dockerfiles, workflow files, and test boot configuration, escalate the level. A label can raise the required level, but it cannot lower a level implied by the changed files.
That last rule matters. A label is a useful human signal. It is not permission to bypass evidence.
The release path always selects L3. Production is where optimism becomes an incident, so the test decision should be boring and conservative there.
RuboCop remains a separate required job. The test classifier does not decide whether Ruby style and static checks should run. This is important because a fast test path should not accidentally become a fast-everything path.
The first version is not a perfect test-impact analysis system.
For example, L1 currently runs a broad group of fast specs rather than calculating the exact affected examples. L2 adds request, controller, and view coverage. This is useful prioritisation, but it is still a policy-based approximation.
That is a feature, not an embarrassment.
A transparent approximation is easier to review than a mysterious “AI selected 14 tests” result. The workflow tells us which level was selected and why. If the policy is wrong, we can improve the policy without pretending that the existing result was mathematically precise.
A good first gate should make risk visible before it tries to automate every decision.
The second improvement in REX was to cache test-result context. The important distinction is that we do not treat an old green result as proof that a new commit is safe.
The cache key includes the things that can change what a test means:
1
2
3
4
5
6
repository
commit and ref
suite and test level
exact command
Ruby / Node / pnpm environment
lockfiles and test configuration
The key is derived from a canonical JSON description and a SHA-256 digest. If the environment or test command changes, the key changes too.
This is the shape of the context rather than just a directory called test-cache:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
{
"repository": "...",
"sha": "...",
"suite": "rspec",
"level": "full",
"command": "bundle exec rails parallel:spec",
"ruby": "...",
"node": "...",
"files": {
"Gemfile.lock": "...",
"pnpm-lock.yaml": "...",
"config/webpacker.yml": "..."
}
}
The exact commit is part of the identity. That means the cache is not pretending that a result for one source revision is a current result for another revision.
So what is it useful for?
One use is diagnostics. When a replacement run starts, it can find the previous failed RSpec result for the same branch and environment, then run the previously failed examples first. This gives fast feedback about a likely regression while the required suite continues to provide the real gate.
The ordering looks like this:
1
2
3
4
5
6
new commit
-> calculate exact test context
-> restore matching failure evidence
-> run previous failed examples first
-> run the required suite
-> upload current results
The failed-example lane is not a replacement for the required suite. It is a short feedback lane. A cached failure is evidence about where to look first, not a verdict about the new code.
That distinction is easy to lose when a CI optimisation is described as “test caching.” We should cache evidence and reproducible context, not confidence.
GitHub’s dependency caching documentation makes a related point: cache keys should change when the inputs that produce the cached output change, and workflows can use the cache-hit result to decide whether work can be skipped. For test results, the inputs need to be much richer than one lockfile hash.
The next question is harder:
How do we know which tests are connected to the changed code?
This is where code-intelligence and graph tools become interesting. A graph can represent relationships that a directory-based classifier cannot see:
1
2
3
4
5
6
7
changed controller
-> route
-> authorization policy
-> service object
-> model callback
-> background job
-> request and system specs
A useful system could use those relationships to propose a test set. It might also identify a path that crosses an application boundary, such as a Rails endpoint calling another service or emitting a job consumed elsewhere.
For my own setup, GitNexus is an interesting candidate because it exposes dependency, call-chain, execution-flow, and impact-analysis views through a local-first code knowledge graph. The important word is candidate. Its output still needs to be checked against the actual Rails conventions and test suite.
CodeQL offers a different model. It is designed for querying code as data and analysing data-flow paths. That is particularly valuable for security and source-to-sink questions, although it is not a drop-in RSpec test selector for a Rails application.
Commercial test-impact systems such as Launchable take another approach: combine repository changes with historical test behaviour and use predictive selection to choose tests. That can be powerful, but it introduces a new dependency on historical data quality. A test that has never failed may still be important. A flaky or under-specified test can distort the model.
I would separate the possible tools into three layers:
1
2
3
4
5
6
7
8
static graph facts
-> symbols, routes, calls, imports, ownership
historical evidence
-> test duration, failures, changed files, flaky examples
AI interpretation
-> explain the likely blast radius and recommend coverage
The first layer should be as deterministic as possible. The second should be measured rather than invented. The third can help us reason, but should show its evidence and uncertainty.
I would not begin with an agent that is allowed to skip tests. I would begin with an agent that produces a reviewable proposal:
1
2
3
4
5
6
7
8
9
Inputs:
diff, changed symbols, routes, graph paths, test history
Output:
selected test categories
likely affected tests
high-risk paths not covered
reasons and confidence
tests that still must run unconditionally
For example:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Changed: Api::CoursesController#update
Graph evidence:
route -> controller -> authorization -> Course#update
controller enqueues CourseIndexJob
Recommended:
request specs for courses
authorization specs
CourseIndexJob specs
one full smoke path
Mandatory:
RuboCop
dependency/security checks
release-level full suite
Confidence:
medium: dynamic dispatch and shared concerns are not fully resolved
The agent is useful because it gathers and explains relationships. It is not the authority that decides whether production is safe.
That authority should remain in explicit CI policy, source-backed tests, and human-reviewed changes to the policy itself.
There are several reasons not to overtrust this approach:
This is why I like the idea of evidence layers. The system should distinguish a parsed route from an inferred relationship, and an observed test failure from a model prediction.
The source code and the test results remain authoritative. The graph is an index. The AI explanation is a navigation and reasoning layer.
The next practical step is not a full autonomous test selector. It is a report-only mode.
For each pull request:
After enough runs, we can measure whether the recommendations are useful:
Only after that evidence exists would I consider allowing a tool to reduce required coverage for a narrow class of changes. Even then, releases and high-risk paths should remain conservative.
The best CI system is not the one with the cleverest test selector. It is the one that makes a safe decision quickly, explains why it made that decision, and leaves enough evidence to improve the next decision.
Risk-based gates give us a policy.
Exact-context caches give us reproducible evidence and faster failure feedback.
Code graphs give us a better map of what a change might affect.
AI can help interpret that map, especially when the architecture is difficult to hold in one person’s head. But it should not turn an uncertain guess into a green checkmark.
The useful progression is:
1
2
3
4
5
full suite baseline
-> explicit risk policy
-> exact-context evidence cache
-> graph-assisted recommendations
-> measured, reviewable automation
That is slower than promising magic and much faster than debugging a production regression caused by a test suite that quietly stopped looking in the right place.
]]>A production deployment can be green and still be broken.
We recently deployed a new version of a Rails learning platform. The image built. The Kubernetes deployments rolled out. The pods became ready. The workers started.
Then somebody opened the login page and got a 500.
That is the kind of incident that makes a team question every green checkmark it has been relying on.
The first request to the application redirected to the login page as expected. The login page itself returned 500.
The Rails error was an asset lookup failure. The page expected a CSS entry such as mypage.css, but the Webpacker manifest did not contain it.
The Kubernetes view of the world looked healthy:
1
2
3
4
rails ready
migration complete
scheduler ready
worker ready
The application view of the world was different:
1
2
GET /login -> 500
missing Webpacker asset -> mypage.css
Both statements were true. The pods were healthy enough to pass their readiness checks. The application was not healthy enough for a user to sign in.
A readiness probe answers a narrow question: should this pod receive traffic?
It does not answer:
Our rollout check was necessary, but it was not sufficient. We had verified process health, not application behaviour.
The missing check was small:
1
2
3
GET /status -> 200
GET /login -> 200
critical CSS/JS assets -> 200
A deployment that cannot pass those checks should not be reported as successful.
The application had recently moved its JavaScript dependencies from Yarn to pnpm. Webpacker 5 still carries assumptions about Yarn.
The production Docker build installed pnpm, but the old Webpacker Rake task still had a Yarn-oriented verification step. In addition, the build copied only part of the Rails configuration before compiling the assets. The pnpm-specific configuration was not reliably available at the point where Webpacker started.
There was another problem layered on top of that. The build command had been written in a form that allowed the asset compilation failure to be ignored:
1
bundle exec rake webpacker:compile || true
That transformed a useful build failure into a bad image that looked successful.
The manifest check we added later made the situation visible:
1
2
manifest = JSON.parse(File.read("public/assets/manifest.json"))
abort unless manifest.key?("mypage.css")
The right place to discover a missing asset is the builder, before the image is pushed or deployed. Not in a user’s browser after the rollout.
The first rollback path was also wrong for an incident.
Instead of switching Kubernetes back to an image that had already been built and verified, the rollback went through the production build process again. That meant waiting for dependency installation, native extensions, asset compilation, image creation, and rollout.
Rebuilding an old commit is useful when an old image no longer exists. It is a poor emergency rollback strategy when the known-good image is already in the registry.
The faster operation is an image switch guarded by the current version:
1
2
expected current image: bad-sha
rollback image: known-good-sha
If the cluster is no longer running the expected bad version, the rollback should stop rather than overwrite an unrelated change.
The rollback should then update the application, migration, scheduler, and worker deployments, wait for each rollout, and run the same application smoke checks used after deployment.
A rollback is not complete when kubectl accepts the command. It is complete when the service is serving the known-good image and the user-facing checks pass.
We made the build and release path more conservative in four ways.
The production image no longer ignores Webpacker errors. The build must compile the assets and produce a manifest containing the critical login stylesheet.
The pnpm integration is explicit. The builder passes a known Webpack executable path to Webpacker and invokes the compiler without the old Yarn-only verification task.
This means a broken manifest stops the build. That is exactly what we want.
After the migration and rollout, the release checks:
1
2
3
4
/status
/login
critical CSS asset
critical JavaScript asset
A pod can be ready while an application is broken. The smoke gate checks the path a user actually takes.
This is not a replacement for feature tests. It is a fast last line of defence between deployment and traffic.
The rollback procedure now targets an existing image identified by its full commit SHA. It checks the current image before changing anything and waits for all selected deployments to become ready.
It does not rebuild the old source during the incident.
That distinction matters. Build pipelines optimise for repeatability. Emergency rollback optimises for time and a known result.
We also changed the CI model. Not every change needs the same test scope.
A documentation-only change should not start a database and a long RSpec suite. A user-facing feature should run focused tests and a basic browser smoke. A database, dependency, CI, or shared test-support change should escalate to a broader suite.
The current levels are:
1
2
3
4
5
6
L0 changed files and static checks
L1 affected tests
L2 basic E2E smoke
L3 full RSpec and integration tests
L4 full E2E
L5 performance tests
The promotion path from main to production remains strict. Faster feedback on a feature branch must not become weaker release confidence.
The next part of this work is test-result reuse.
A failed test result is useful evidence. It can tell us which examples to rerun first. But a cached pass is not proof that a new commit is safe.
Our result context therefore includes more than a branch name:
A failed-example retry is allowed only when the context matches. The first attempt and retry remain separate in the CI report.
The rule is simple:
Use cached failures to find the problem faster. Never use cached passes to bypass the required tests.
When a deployment reports success, I want to see evidence from three layers:
/login, critical assets, and one meaningful user path work.If one layer is missing, the release is only partially verified.
I also want a rollback drill. A rollback procedure that has never been exercised is a paragraph in a document, not an operational capability. The team should know which image is known good, how to switch to it, how long the operation takes, and what evidence proves recovery.
The incident was not caused by one exotic failure. It came from ordinary assumptions lining up in the wrong direction:
Each assumption was understandable in isolation. Together they produced a deployment that was green in the pipeline and broken for users.
The fix is not to add more green badges. It is to make each badge answer a precise question, fail when that question has a bad answer, and keep a fast path back to a version that has already worked.
That is the standard I want from production engineering: not the appearance of safety, but evidence that the system can start, serve a real request, and recover when the next change goes wrong.
]]>By that point, resetting a password and deleting a suspicious plugin would not have been enough. I needed a reason to trust the code and database again.
I chose to rebuild from a reviewed pre-incident baseline, test a separate recovery candidate, and replace the affected installation through a controlled cutover. The recovery also uncovered ordinary application problems: missing plugin activation, broken images, and a theme widget displaying content it should not have displayed.
Those repairs were part of getting the site back, too.
This account omits identifying details, incident dates, infrastructure information, and exploit mechanics. It describes the recovery decisions and checks recorded at the time, rather than claiming anything about the site’s current security.
The investigation drew on access logs, a copy of the affected code, and database records. Each answered a different question.
The logs helped establish a sequence of administrative activity. The code showed malicious modifications. Database records showed unauthorised accounts and changes to application state. Together, they supported a much stronger conclusion than any isolated request or suspicious filename.
They did not establish the original entry point conclusively.
That distinction matters. Evidence of an attacker using administrative access does not, by itself, explain how they obtained it. A stolen password, an earlier vulnerability, and an existing backdoor are different explanations. I could not honestly choose one just because it made the story easier to tell.
I also could not use a successful recovery to conclude that no information had been accessed. Restoring an application and assessing the impact of a compromise are separate jobs.
Before destructive cleanup, I preserved copies of the code, database, and available logs. Backup verification and independently stored copies gave me something to return to during the investigation.
There is an important distinction between an evidence copy and a recovery copy. The affected installation was useful for understanding what happened. That did not make it suitable to put back online.
In a live incident, containment may need to happen immediately to protect visitors and prevent further changes. Preserving evidence should support that response, not become an excuse to leave a harmful site running.
A checksum has a similarly limited purpose. It can show that a copied archive matches the original archive. It cannot show that the original archive was free of malicious code.
Malicious changes were spread across the installation rather than confined to one removable component. That made an in-place cleanup difficult to verify. Deleting everything I had found would still leave the question of what I had missed.
A separate recovery candidate gave me a clearer basis for comparison. I could review the earlier baseline, check the expected users and application state, and test the candidate before switching the live site over.
An older backup is not automatically trustworthy. It may predate the visible symptoms without predating the compromise. It may also contain outdated software. A sound recovery process needs both a reviewed baseline and supported software from trusted sources; blindly restoring an old archive can restore old vulnerabilities along with the content.
The same caution applies to the database. Replacing application files does not remove an unauthorised account or repair altered settings. The recovery had to consider code and data together.
I prepared the candidate without immediately overwriting the affected installation. This preserved the evidence and allowed checks before cutover.
Code and database state were treated as one release. A tested code directory connected to the wrong database state would not have been the candidate I had verified.
Before destructive database cleanup, I made and verified another backup. This mattered because the recovery procedure changed once the old data was removed. Reversing a directory change would no longer be enough; restoring data would also be necessary.
For a security incident, a rollback plan must be explicit about where it leads. It should provide a safe maintenance state or a reviewed recovery version. Re-enabling a known-compromised installation is not a safe fallback simply because it is convenient.
After the code switch, some pages did not behave as expected.
Required plugins were present on disk but were not active in the restored application state. That left page components and a contact form displaying incorrectly. Image delivery also needed attention, and an empty theme widget still displayed built-in default content.
These were useful reminders of how much behaviour sits outside the files themselves. A plugin directory can exist without its functionality being available. A page can return a successful response while showing broken content.
The recorded follow-up checks confirmed that affected page components and the contact form rendered again, and that the checked images loaded correctly. That is a narrower claim than saying every workflow passed. A visible form does not prove submission works or that its email reaches the intended inbox.
For a recovery checklist, I would include representative pages, media, login and permissions, form submission, email delivery, and any external integrations the site depends on. Each needs its own observed result.
The recovery record showed that the replacement site served the checked pages, expected accounts remained, identified unauthorised accounts were absent, and known malicious components were no longer present in the candidate. Existing WordPress login sessions were invalidated as part of the response.
Those were useful results. They were not proof that every possible persistence mechanism had been eliminated or that all potentially exposed credentials had been replaced.
Likewise, a scan that finds none of the previously identified malicious patterns is evidence about those patterns. It is not a certificate that the whole environment is safe.
I want an incident report to say exactly what was checked and what passed. Recommendations should stay recommendations until someone implements and verifies them. Credential replacement, stronger authentication, least privilege, supported dependencies, and protected backups belong in a general recovery plan, but listing them is not evidence of completion.
The backup work after the recovery focused on making failures visible and copies verifiable. Archive readability, checksums, and copies held separately from the hosting account are all useful checks.
They still do not replace a restore test.
A backup can be readable while missing something the application needs. An incremental archive can be valid while depending on another archive that has been lost. A database dump can import successfully while the restored application fails to behave correctly.
A small restore fixture was exercised during the follow-up work. That helped test the mechanics, but it was not an end-to-end rehearsal of the entire live site. Keeping that distinction in the record makes the next piece of work clear.
A follow-up integrity check was introduced to detect unexpected changes to application files and important administrative state. Its job was to report changes for review, not delete files on its own.
That separation was deliberate. Legitimate updates change files too. A detector needs an understood baseline, reliable alert delivery, and a person who can distinguish an approved change from something that needs investigation.
The baseline itself must come from a reviewed state. Otherwise, monitoring can faithfully tell you that an already-compromised installation has not changed.
The most useful decision was to stop treating the task as a search for a few bad files. Once changes were spread across code and accounts, a separate, reviewed recovery candidate was easier to reason about than repeated edits to the affected installation.
The ordinary application checks mattered just as much to completing the work. Users need functioning pages and forms, not merely a server that responds.
I would use the same approach again: contain the incident, preserve what is needed for investigation, rebuild from reviewed sources, test the behaviour people depend on, and record the limits of the evidence. Recovery is easier to review when every claim has a check behind it and every unverified item remains visible.
I did not start with the idea of building an AI company. I started with a more ordinary problem: too many small operational loops were competing for the same limited attention.
I have been thinking about a slightly unusual company structure.
Not a startup with a large engineering team and an AI assistant on the side. A one-person company in which the founder can move between product, development, project management, customer conversations, and market research without losing the thread between them.
The attraction is obvious. A capable language model can help write code, investigate an error, turn an idea into a product brief, summarize competitors, and prepare the next decision. It can fill some gaps that would traditionally require several specialists.
But “AI can do many jobs” is not yet an operating model.
The harder question is how to connect those jobs safely. If Sentry sees an error, can an agent investigate it? If a market signal suggests a new feature, how does that become a real product decision? If an agent changes code, who verifies it? If a cloud worker disappears, what state is lost?
I am treating this as a design and learning project, not as a claim that one person can automatically replace an entire company. None of this is a finished product yet. It is a design hypothesis assembled from existing tools, small experiments, and a set of constraints I want to test.
A small company usually has several different loops running at once:
1
2
3
4
5
customer problem
↓
market signal → product decision → feature work → release
↑ ↓
customer feedback ← support ← observability
In a larger organization, these loops are distributed across people and systems. Product managers maintain the roadmap. Developers work from issues and pull requests. Sentry and other observability tools report production problems. Marketing and sales bring back market information.
In a one-person company, all of these loops eventually return to one person.
The bottleneck is not necessarily typing code. It is remembering what matters, deciding what to do next, switching contexts, and keeping the evidence connected to the decision.
That is where an AI-native operating system could help.
My current model has four related but distinct responsibilities.
Sentry should remain the source of truth for runtime errors and performance problems. An agent could:
The important word is propose.
A Sentry event is evidence that something happened. It is not permission to edit production code, deploy a change, or declare an incident resolved. An AI-generated patch still needs tests, diff inspection, review, and a rollback path.
The first useful experiment is therefore not “let the agent fix every error”. It is a small Sentry-to-pull-request workflow with a narrow repository scope and a measurable success criterion.
Features need a different system from errors. A bug report can start with a stack trace. A feature starts with a customer problem, a market observation, or a strategic choice.
Jira, Redmine, or another issue tracker should remain the source of truth for:
An agent can help turn a rough idea into a smaller set of tasks. It can identify missing acceptance criteria, find related work, draft release notes, and flag scope expansion. It should not quietly turn every interesting market signal into a feature.
For a very small company, Jira may be more process than is necessary. Redmine offers more control if I am willing to operate it. A simpler hosted issue tracker may be the better starting point. The correct choice depends less on the feature list than on whether I will actually maintain the workflow.
The AI project manager is not a virtual person who “owns delivery”. It is a coordinator that keeps project state visible.
A useful project-management agent could answer:
It could prepare a daily or weekly report from the issue tracker, Git, pull requests, CI, and Sentry. That is valuable because reporting is repetitive. The final decision about priority, trade-offs, and commitments still belongs to the founder.
A useful report might be deliberately boring:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Weekly project report
Progress:
- 2 tasks completed
- 1 pull request awaiting review
Blocked:
- Sentry issue has no reproducible fixture
Risk:
- The next milestone contains more work than the current weekly capacity
Recommendation:
- Split the export feature and defer the admin UI
The point is not to produce an impressive narrative. It is to compress project state into a decision that I can inspect.
Market tracking is a separate loop again. It should collect and compare:
A market agent should produce a small, cited digest rather than an endless stream of links. Every item needs a source, date, confidence, and an explanation of why it may matter.
A trend is not validation. A competitor announcement is not proof of demand. A model’s summary is not primary evidence.
The business test is equally important: the system is useful only if it improves validated outcomes, such as reaching a useful product decision sooner, reducing unproductive development, finding customer pain earlier, or delivering a first paid version with less coordination overhead. More summaries, tickets, and agent traces are not outcomes by themselves.
The current design looks like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
founder
│
decisions and approvals
│
Hermes
personal control plane and router
┌─────────────────┼─────────────────┐
│ │ │
Sentry issue tracker market sources
│ │ │
└────────────── agents ────────────┘
│
Git, CI, worktrees, reports
│
cloud or local workers
The systems should not all become one database.
This separation is important. An agent should retrieve the relevant context without silently becoming the owner of every piece of state.
I am considering four broad categories of tools.
AI Studio is useful for trying models, prompts, generated applications, and API ideas quickly. It is a good laboratory and prototyping surface.
I would not make it the company’s control plane. A prototype interface is not the same thing as durable workflow state, identity management, audit history, repository permissions, or release governance.
A shared ChatGPT workspace could be a useful human-facing layer for drafting product requirements, reviewing market notes, and asking cross-functional questions.
It should not become the authoritative system for code, incidents, releases, or customer records. Those need systems with explicit state, access controls, and history.
Hermes is the natural candidate for my primary personal control plane because it already connects memory, skills, document retrieval, scheduled workflows, model routing, messaging, and delegated coding work.
ZeroClaw is interesting as a smaller, more portable worker or experimental runtime. I do not want to create two unrestricted control planes that both believe they own the same memory, credentials, and automations.
The division should be explicit: one orchestrator, specialized workers, narrow permissions.
A Spot VM can provide inexpensive, interruptible capacity for jobs that can be retried or discarded. It is a reasonable place for an ephemeral coding worker, a batch analysis task, or a disposable test environment.
It is not a good place for the only copy of company state. Important state must live in durable storage, and a worker must be able to restart without guessing what happened before the interruption.
The previous estimates I saw for monthly Spot VM cost should be treated as planning assumptions, not promises. Region, machine type, disks, network traffic, quotas, and interruption behavior all affect the real cost.
My current default is therefore deliberately small: Hermes as the orchestrator, Sentry and GitHub as integration sources, an existing issue tracker for product state, and a Spot worker only for jobs that are demonstrably interruptible. I do not want to build the full platform before proving that one workflow saves time.
The obvious alternative is to buy a collection of SaaS products and avoid building anything. That is probably the right answer for much of the system.
Existing services are better at durable records, authentication, notifications, and collaboration than a one-person custom platform would be. The reason to add a control layer is narrower: connect the systems I already use, route work to the right model or worker, enforce approval boundaries, and produce a useful report without copying all state into a second database.
Self-hosting becomes worthwhile only when it solves a demonstrated problem: privacy, integration control, cost at a known workload, or a workflow that the hosted products cannot express. Otherwise, operating another dashboard is just another coordination loop.
Not every action should have the same approval requirement.
The model should not be asked to decide whether its own action is safe. The workflow should encode the boundary, and the human should remain at the sensitive gates.
I do not want to start by building a complete multi-tenant enterprise platform. That would be an impressive way to avoid learning whether the workflow is useful.
The first vertical slice should be:
1
2
3
4
5
6
7
8
Sentry event
→ sanitized issue context
→ bounded agent investigation
→ isolated worktree
→ proposed fix and test
→ independent verification
→ pull request
→ human review
I would measure:
If this loop does not work on a small, synthetic or low-risk fixture, adding more dashboards and more agents will not help.
This is the beginning of a series rather than a finished architecture. Possible follow-up articles include:
Each article should be based on an actual experiment, repository state, source-backed comparison, or failure. I would rather publish a small workflow that worked and explain its limits than publish a grand diagram that has never been exercised.
The goal is not to pretend that one person has become ten people.
The goal is to reduce the cost of moving between important roles without losing judgment. A developer should be able to investigate a market signal. A founder should be able to understand an operational incident. A project manager should be able to see what is actually blocked. An agent should be able to do the repetitive preparation work while leaving accountability visible.
That requires more than a strong model. It requires clear sources of truth, narrow permissions, durable state, independent verification, and a willingness to stop automation when the evidence is weak.
For now, this is a plan. The next step is not to automate the whole company. It is to prove one bounded loop, measure it honestly, and let the next article follow from what actually happened.