diff --git a/AGENTS.md b/AGENTS.md index 19ec799..63c4739 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,12 +1,55 @@ # Repository Guidelines -## Scope +## Scope and current phase - Build a CDP-only network observability and debugging control plane for clark-browser/Chromium. - Collect and correlate CDP `Network`/`Fetch` HTTP(S), failure, cache, and WebSocket events. - Keep the MVP read-only. Any future request or response mutation must be explicit opt-in and default off. +- Work in M0 architecture/Spike: validate one browser instance, a local test page, CDP event coverage, stable schema, redaction/truncation, and disconnect behavior before M1. +- Optimize first for one or a few isolated browser instances; do not assume cluster-scale concurrency in M0/M1. - Prefer the smallest correct change and reuse existing patterns before adding dependencies or abstractions. +## Users and supported scenarios + +- Help automation developers diagnose request failures, ordering, retries, and page behavior. +- Let data-pipeline developers consume structured browser network events downstream. +- Let operations and test users inspect session-scoped HTTP and WebSocket failure chains. + +## Architecture and data flow + +- CDP access validates short-lived credentials, protects the WebSocket connection, discovers targets, binds sessions, maintains heartbeats, and reports disconnects. +- The collector enables CDP `Network`/`Fetch` as needed, correlates `requestId`, records HTTP and WebSocket lifecycles, and reads response bodies only on demand. +- The policy layer normalizes events, filters by URL/resource/status, and applies redaction, sampling, size, event, and retention limits. +- Start storage and query with JSONL containing session, target, request, timing, URL/method, safe header summaries, body availability/reference, and error fields. Do not persist full bodies by default. +- Keep query and redacted export session-isolated and paginated. Re-evaluate Postgres, object storage, and asynchronous streams only after MVP evidence. +- A later console may show sessions, timelines, filters, and errors, but never control the browser GUI. + +Data flow: `clark-browser (CDP enabled)` → protected CDP WebSocket → target/session binding → Network/Fetch events and on-demand body reads → normalize/filter/redact/limit → JSONL/event stream → session-scoped API/export. + +CDP observes browser-internal decrypted events. It is not a client/server proxy, does not generate certificates, and must not change the network path. + +## Priority + +- P0: single-instance CDP access; HTTP request/response/failure and WebSocket events; `requestId` correlation; default redaction; body/event limits; JSONL export; session isolation; disconnect observability. +- P1: multiple targets/sessions, paginated query, metrics/alerts, permission audit, asynchronous event streams, and tiered storage. +- P2: Fetch mutation, versioned rules, replay, browser-cluster access, and a minimal console. + +## MVP acceptance + +- A page load reconstructs request → response → completion/failure by `session_id`, with fixture-defined key fields 100% correct. +- WebSocket open/close and sent/received frames correlate to their connection; HTTPS and HTTP/2 remain observable without MITM. +- Sensitive values are redacted by default. Oversized bodies are not persisted and retain a truncation or unavailable reason. +- Query and export are paginated and session-isolated; unauthorized sessions cannot read or export another session. +- CDP disconnects, target closure, dropped events, and individual body-read failures are reported or recoverable without terminating the browser workload. +- Local end-to-end coverage includes connection, collection, correlation, redaction, storage, query, export, and disconnect, with no VNC, MITM, proxy certificate, or CA dependency. + +## Iteration plan + +- M0: confirm clark-browser CDP startup/connection, event coverage, stable schema, and one local page-load loop. +- M1: deliver single-instance collection, redaction/limits, JSONL query/export, disconnect recovery, and local fixtures. +- M2: add multiple targets/sessions, tiered storage, an asynchronous queue, metrics/alerts, and permission audit when evidence warrants them. +- M3: add explicitly authorized Fetch mutation, replay, versioned rules, and browser-cluster access only after earlier phases validate the need. + ## Non-goals - Do not add VNC/noVNC, desktop streaming, or GUI browser control. @@ -22,6 +65,14 @@ - Report disconnects, target closure, dropped events, and body-read failures without stopping the browser workload. - Validate all external input and preserve compatibility of persisted schemas and exported data. +## Constraints and validation + +- CDP does not cover non-browser processes, WebRTC media contents, or final rendered page state. +- Large bodies, high-frequency WebSocket frames, and multi-instance concurrency create memory, throughput, and privacy costs; prefer bounded, sampled, or discardable handling in the MVP. +- Advance beyond M0 only when local fixtures and a local test page demonstrate event/correlation completeness, zero sensitive-value leakage, body-limit behavior, disconnect recovery, and acceptable single-instance resource cost. +- Trigger ordinary HTTP, a failed request, HTTPS, HTTP/2 where available, and a WebSocket during validation; never require external production systems. +- Validate the product by reducing repeated debugging and collection glue code before building clusters, paid proxy integrations, or an operations console. + ## Development and tests - Keep commits focused; do not add speculative scaffolding, dependencies, CI, or an unconfirmed application stack.