AGENTS.md

@@ -1,12 +1,55 @@
 # Repository Guidelines
 
-## Scope
+## Scope and current phase
 
 - Build a CDP-only network observability and debugging control plane for clark-browser/Chromium.
 - Collect and correlate CDP `Network`/`Fetch` HTTP(S), failure, cache, and WebSocket events.
 - Keep the MVP read-only. Any future request or response mutation must be explicit opt-in and default off.
+- Work in M0 architecture/Spike: validate one browser instance, a local test page, CDP event coverage, stable schema, redaction/truncation, and disconnect behavior before M1.
+- Optimize first for one or a few isolated browser instances; do not assume cluster-scale concurrency in M0/M1.
 - Prefer the smallest correct change and reuse existing patterns before adding dependencies or abstractions.
 
+## Users and supported scenarios
+
+- Help automation developers diagnose request failures, ordering, retries, and page behavior.
+- Let data-pipeline developers consume structured browser network events downstream.
+- Let operations and test users inspect session-scoped HTTP and WebSocket failure chains.
+
+## Architecture and data flow
+
+- CDP access validates short-lived credentials, protects the WebSocket connection, discovers targets, binds sessions, maintains heartbeats, and reports disconnects.
+- The collector enables CDP `Network`/`Fetch` as needed, correlates `requestId`, records HTTP and WebSocket lifecycles, and reads response bodies only on demand.
+- The policy layer normalizes events, filters by URL/resource/status, and applies redaction, sampling, size, event, and retention limits.
+- Start storage and query with JSONL containing session, target, request, timing, URL/method, safe header summaries, body availability/reference, and error fields. Do not persist full bodies by default.
+- Keep query and redacted export session-isolated and paginated. Re-evaluate Postgres, object storage, and asynchronous streams only after MVP evidence.
+- A later console may show sessions, timelines, filters, and errors, but never control the browser GUI.
+
+Data flow: `clark-browser (CDP enabled)` → protected CDP WebSocket → target/session binding → Network/Fetch events and on-demand body reads → normalize/filter/redact/limit → JSONL/event stream → session-scoped API/export.
+
+CDP observes browser-internal decrypted events. It is not a client/server proxy, does not generate certificates, and must not change the network path.
+
+## Priority
+
+- P0: single-instance CDP access; HTTP request/response/failure and WebSocket events; `requestId` correlation; default redaction; body/event limits; JSONL export; session isolation; disconnect observability.
+- P1: multiple targets/sessions, paginated query, metrics/alerts, permission audit, asynchronous event streams, and tiered storage.
+- P2: Fetch mutation, versioned rules, replay, browser-cluster access, and a minimal console.
+
+## MVP acceptance
+
+- A page load reconstructs request → response → completion/failure by `session_id`, with fixture-defined key fields 100% correct.
+- WebSocket open/close and sent/received frames correlate to their connection; HTTPS and HTTP/2 remain observable without MITM.
+- Sensitive values are redacted by default. Oversized bodies are not persisted and retain a truncation or unavailable reason.
+- Query and export are paginated and session-isolated; unauthorized sessions cannot read or export another session.
+- CDP disconnects, target closure, dropped events, and individual body-read failures are reported or recoverable without terminating the browser workload.
+- Local end-to-end coverage includes connection, collection, correlation, redaction, storage, query, export, and disconnect, with no VNC, MITM, proxy certificate, or CA dependency.
+
+## Iteration plan
+
+- M0: confirm clark-browser CDP startup/connection, event coverage, stable schema, and one local page-load loop.
+- M1: deliver single-instance collection, redaction/limits, JSONL query/export, disconnect recovery, and local fixtures.
+- M2: add multiple targets/sessions, tiered storage, an asynchronous queue, metrics/alerts, and permission audit when evidence warrants them.
+- M3: add explicitly authorized Fetch mutation, replay, versioned rules, and browser-cluster access only after earlier phases validate the need.
+
 ## Non-goals
 
 - Do not add VNC/noVNC, desktop streaming, or GUI browser control.
@@ -16,12 +59,21 @@
 ## Security and reliability
 
 - Never commit or log CDP tokens, cookies, `Authorization`, `Set-Cookie`, credentials, full sensitive bodies, or production data.
-- Use short-lived CDP credentials over a protected internal or TLS connection, with session isolation and auditable access.
+- Never put CDP tokens, `Cookie`, `Authorization`, credentials, or unredacted sensitive values in any URL.
+- Treat CDP access as browser-high-privilege: use short-lived credentials, protected internal or TLS transport, least privilege, session isolation, and auditable access.
 - Redact sensitive headers and bodies before persistence or export; enforce body, event, frame, and retention limits.
 - Treat response bodies as optional: cache, target closure, protocol behavior, or lifecycle timing may make them unavailable.
 - Report disconnects, target closure, dropped events, and body-read failures without stopping the browser workload.
 - Validate all external input and preserve compatibility of persisted schemas and exported data.
 
+## Constraints and validation
+
+- CDP does not cover non-browser processes, WebRTC media contents, or final rendered page state.
+- Large bodies, high-frequency WebSocket frames, and multi-instance concurrency create memory, throughput, and privacy costs; prefer bounded, sampled, or discardable handling in the MVP.
+- Advance beyond M0 only when local fixtures and a local test page demonstrate event/correlation completeness, zero sensitive-value leakage, body-limit behavior, disconnect recovery, and acceptable single-instance resource cost.
+- Trigger ordinary HTTP, a failed request, HTTPS, HTTP/2 where available, and a WebSocket during validation; never require external production systems.
+- Validate the product by reducing repeated debugging and collection glue code before building clusters, paid proxy integrations, or an operations console.
+
 ## Development and tests
 
 - Keep commits focused; do not add speculative scaffolding, dependencies, CI, or an unconfirmed application stack.