Cortex

The problem

Any agent that has to operate a real website runs into the same wall twice. Hand-written selectors break the moment a site renames a class or reshuffles a layout. Handing the raw page to an LLM on every visit works, but it is slow, expensive, non-deterministic, and wastes the model's context on markup it cannot act on directly. Neither approach touches the harder cases at all: a Cloudflare or hCaptcha wall, a react-select widget that silently eats its own input on a stray backspace, a login flow that must never expose a password to the model doing the clicking. I needed a service an agent could drive by intent, that survives a redesign instead of breaking on one, and that keeps every credential out of the model's context by construction rather than by convention.

Constraints

• ·The consumer is an LLM agent over MCP, so every tool has to accept and return compact JSON, never a raw DOM dump the model has to parse itself.
• ·Real anti-bot exposure. Targets include LinkedIn, Cloudflare and hCaptcha protected job boards, and application flows with non-standard widgets, not a synthetic test page.
• ·Secrets cannot reach the model. A password or an OTP code can never appear in a tool argument or a tool result.
• ·The recorded graph doubles as a future training set, so a silently wrong recorded action is a permanent, invisible error, not a cache miss to be evicted.
• ·One engineer, continuous build, no team to split browser-stealth work from the agent-facing surface, so every hard call needed to be written down rather than discussed in a hallway.

Architecture

An MCP server (FastMCP) exposes 41 tools over streamable-HTTP or stdio, gated by a bearer token on every request except a separate token-gated /secret route. The tools are thin wrappers over Hands, the orchestrator that drives a real, stealth-configured Chrome session through Playwright: view and read return a compact structural document, never the DOM, and act tools such as fill_form, click, and combo_select drive the page by ref and verify every stateful action against live DOM state afterward. Site knowledge, structure fingerprints, learned widget recipes, and the mapped state graph, lives in an embedded DuckDB store where Hands is the sole writer. Two other producers, a batch scraper with per-site specialists and a structure-only mapper called genesis, feed that same graph one-directionally through a single ingest path, so an interactive agent visit and a bulk scrape both enrich one brain instead of three disconnected stores.

• ›MCP server (mcp_server.py): 41 tools over streamable-HTTP or stdio, a bearer-token gate on every request, a separate token-gated /secret UI mounted on the same app
• ›Hands (hands.py, orchestrator): view, read, mark, crawl_marked, fill_form, click, select_option, combo_select, traverse, scan, extract, login, request_secret and more
• ›Browser engine: Playwright driving a real installed Chrome (not bundled Chromium), fingerprint and hardware-emulation spoofing, a tab pool, human-like mouse and typing
• ›DuckDB store, the brain: sites, maps, nodes, edges, manifests, sessions, field_health, widget_patterns, embedded and single-writer, Hands holds the lock
• ›Structure fingerprinting: a Merkle tree over tag, role, and child fingerprints only, never id, class, or text, so drift-heal descends only into subtrees that actually changed
• ›Secret broker: a token-gated /secret route where a human types a password or OTP, the server injects it server-side and never returns the value to the agent
• ›Batch scraper (H1, main.py run): per-site specialists for Google Maps, LinkedIn jobs, and scholarships, feeding the brain through ingest_blueprint
• ›Mapper (H2, main.py map): genesis structure extraction, a pure function of the current DOM with no browser of its own, also feeding the brain through ingest_blueprint

The agent only ever talks to the MCP server and only ever sees compact JSON. Hands is the sole writer to the DuckDB graph. The batch scraper and the mapper are separate producers that enrich the same graph one-directionally, never writing to it directly.

How one request flows

1. 01The agent opens a session. open_session launches a real installed Chrome with stealth fingerprinting and a persistent profile, and returns a session id. Nothing about the page has been read yet.
2. 02The agent reads before it acts. view or read returns a compact structural document: interactive controls with a short ref, headings, reading-order text. The full DOM stays in process.
3. 03The agent acts by ref. click, type, select_option, combo_select, and fill_form take refs from the last read. Stateless fields batch together, stateful ones run one at a time, and every field returns a verified outcome, never a silent success.
4. 04Structure gets mapped when it matters. scan runs structure extraction into a blueprint, or the agent marks leaves with mark and follows them in one pass with crawl_marked. A known site can instead be flown over with graph_view and traverse, replaying stored actions with no re-scanning.
5. 05A login or a secret routes to a human. login or request_secret opens the token-gated /secret form. A human types the value there, the server injects it into the page, and only a status string returns to the agent.
6. 06Every verified action is written back. A recipe or a fingerprint is persisted to the DuckDB graph only after it is checked against the live DOM, because an unverified write would be a permanent, invisible error in what doubles as training data.
7. 07The session closes with state kept for reuse. close_session tears the browser down, but the encrypted storage state persists, so the next session reuses cookies instead of logging in cold, the single riskiest action the system can take.

Engineering decisions

What changed along the way

• →Structure extraction (genesis) originally launched its own browser and crawled sub-pages to decide where to look. That self-navigation was the most brittle part of the system, and it left entirely once an external agent took over navigation, leaving genesis a pure function of the current DOM.
• →The MCP transport was designed around a separate agent container reaching the service over a network, HTTP only, no stdio. That consumer never materialized. The real consumer became Claude Code spawning the server locally, so stdio transport was added afterward.
• →The plan called for stored recipes to be consulted before an action, a lookup that would save re-deriving a widget's behavior on a second visit. Only the write-back half shipped so far, every verified action still gets recorded, but nothing yet reads a recipe before acting, recorded openly as an unbuilt piece rather than left to be discovered.
• →Drift-heal shipped in two passes. The first compared only a flat set of leaf blocks. A later pass made the comparison hierarchical over the actual outline tree, so a change localizes to the section that moved instead of a scattered list of changed blocks.

Security and reliability

• ·Every MCP request needs a bearer token once the server binds anywhere but loopback. The server refuses to start on a public bind with no token configured, rather than starting unauthenticated.
• ·Passwords and OTP codes never pass through a tool argument or a tool result. The /secret UI is its own token-gated route with a constant-time comparison, and it fails closed with no token set.
• ·Stored browser session state is encrypted at rest with Fernet, keyed from an environment secret. A wrong or missing key returns nothing, and there is no silent plaintext fallback.
• ·The raw Playwright-eval debug tool is disabled unless explicitly enabled, so there is no anonymous code execution path by default.
• ·A detected final-submit click is refused without an explicit confirmation flag, enforced by the tool itself, not by agent discipline.
• ·Every stateful form action is checked against the live DOM after it runs, and a mismatch is recorded as a failure rather than reported as a success.
• ·Session reuse is preferred over cold automated login, the single most flagged, most ToS-risky action available, so re-authentication happens interactively only on expiry.

Metrics

What shipped

• ✓Drives a real, stealth-configured Chrome session by intent, over MCP, with the DOM never leaving the process.
• ✓Reads, fills, and verifies web forms, including react-select and chip widgets, with a per-field, non-silent outcome for every action.
• ✓Heals its own structural map of a site through Merkle-fingerprint diffing instead of re-scanning from zero on every visit.
• ✓Keeps every credential out of the driving agent's context through a separate, token-gated human channel.
• ✓Unifies three separate producers, an interactive agent, a batch scraper, and a structure-only mapper, into one DuckDB graph with a single, auditable writer.

Lessons learned

• ·Being an actual browser beat every layer of stealth injection tried on top of a fake one. The simplest fix was also the correct one.
• ·A macro breaks the moment the DOM re-renders. A declarative description of what a widget is survives it, and that distinction was worth rebuilding the recipe system around.
• ·Writing down not just what shipped, but what was planned and never wired in, made the system easier to trust, because the gaps were visible instead of implied.
• ·Text-blind structural fingerprints and text-based recurrence signals are two different jobs. Trying to make one signal do both was the wrong instinct, and splitting them was the right one.
• ·A guard that defaults to inactive protects nobody. Naming the real cost on both sides of a safety default, in writing, is what actually gets it flipped on.

Tech stack