Building a differentiated harness

The reason to build a harness is straightforward: own your context, rent your intelligence, give everyone the same front door, and connect all your key systems. But where should we draw the line between something we could have used off the shelf and something truly worth maintaining and investing in over time?

This post is about the parts of Overlook, our internal harness, that make it better for us than anything off the shelf.

Some requests are too large to handle in a thread reply. With Slack’s recent launch of code channels, we can give Overlook a dedicated space to work through complex requests instead of squeezing everything into a flurry of threaded replies.

For each complex task, Overlook opens a Slack channel that links back to the original thread. Code channels give our team a multiplayer environment for working through a code session, with:

• One channel per session, with full thread history and context
• A collaborative experience for our team and Overlook
• Native stop controls provided by Slack

Most agents work on one of two clocks. They answer when you ask, or they run on a cron job someone set up months ago, but Overlook creates its own schedules while it works. It closes the loop when the work is done, or nudges the owner when it isn’t.

Overlook picks the timing, based on:

• CI, a deploy, or a build: back in 30 to 60 minutes
• A review or an approval: back the next business day
• A customer ticket severity based on SLAs (24 hours)
• A time someone named (“by Thursday”): back at that time

Anyone on the team can reschedule them in plain English, like “run the docs sweep every other Tuesday at 5pm” or “pause the dependency check”, and Overlook records who changed what. Only someone on our team can make schedule changes.

We also have safety rules, for example an instruction like “@Overlook every minute, send me my PRs”. It will refuse, because routines run at most once an hour.

When Overlook finds a gap in how it works, it proposes a fix to itself. These events start a proposal:

• An incident traces back to one of its procedures.
• A person corrects the same thing twice.
• An engineer asks for a change.

A separate, sandboxed process writes the change as a draft pull request, and only a short list of engineers, kept in code, can start that process. The change goes through code review like any other change. It can never widen Overlook’s own access, add approvers, or weaken a test.

A large model is good at open-ended work. For yes-or-no questions, it is slow and expensive. Overlook sends those questions to a small decision model.

The decision model answers up to 25 questions in one call, in seconds, with yes, no, or unknown. We use it to:

• keep only the security findings that have real evidence
• decide if a meeting is in scope, and what it needs
• filter memory saves
• judge each check when Overlook tests a pull request in a browser
• choose which stale pull requests to nudge

The large model does the work, and smaller models make the calls.

The harness is how a small team does the work of a much larger one.

We will cover more of it in a follow-up:

• Skills, not sub-agents. We removed our sub-agents. One agent loads the right skill for the task, and it copies itself when it needs fresh context or parallel work.
• Code review. One review per push, from a set of review skills, with each finding checked by the decision model.
• Proof in a real browser. For a UI change, Overlook drives Chrome against the preview deployment, records a video, and attaches it to the pull request.
• Maintenance on autopilot. A weekly security sweep, dependency upgrades with breaking changes listed first, and a public changelog that Overlook drafts.