lume-ctrl/AGENTS.md

Repository Guidelines

Scope and current phase

• Build a CDP-only network observability and debugging control plane for clark-browser/Chromium.
• Collect and correlate CDP Network/Fetch HTTP(S), failure, cache, and WebSocket events.
• Keep the MVP read-only. Any future request or response mutation must be explicit opt-in and default off.
• Work in M0 architecture/Spike: validate one browser instance, a local test page, CDP event coverage, stable schema, redaction/truncation, and disconnect behavior before M1.
• Optimize first for one or a few isolated browser instances; do not assume cluster-scale concurrency in M0/M1.
• Prefer the smallest correct change and reuse existing patterns before adding dependencies or abstractions.

Users and supported scenarios

• Help automation developers diagnose request failures, ordering, retries, and page behavior.
• Let data-pipeline developers consume structured browser network events downstream.
• Let operations and test users inspect session-scoped HTTP and WebSocket failure chains.

Architecture and data flow

• CDP access validates short-lived credentials, protects the WebSocket connection, discovers targets, binds sessions, maintains heartbeats, and reports disconnects.
• The collector enables CDP Network/Fetch as needed, correlates requestId, records HTTP and WebSocket lifecycles, and reads response bodies only on demand.
• The policy layer normalizes events, filters by URL/resource/status, and applies redaction, sampling, size, event, and retention limits.
• Start storage and query with JSONL containing session, target, request, timing, URL/method, safe header summaries, body availability/reference, and error fields. Do not persist full bodies by default.
• Keep query and redacted export session-isolated and paginated. Re-evaluate Postgres, object storage, and asynchronous streams only after MVP evidence.
• A later console may show sessions, timelines, filters, and errors, but never control the browser GUI.

Data flow: clark-browser (CDP enabled) → protected CDP WebSocket → target/session binding → Network/Fetch events and on-demand body reads → normalize/filter/redact/limit → JSONL/event stream → session-scoped API/export.

CDP observes browser-internal decrypted events. It is not a client/server proxy, does not generate certificates, and must not change the network path.

Priority

• P0: single-instance CDP access; HTTP request/response/failure and WebSocket events; requestId correlation; default redaction; body/event limits; JSONL export; session isolation; disconnect observability.
• P1: multiple targets/sessions, paginated query, metrics/alerts, permission audit, asynchronous event streams, and tiered storage.
• P2: Fetch mutation, versioned rules, replay, browser-cluster access, and a minimal console.

MVP acceptance

• A page load reconstructs request → response → completion/failure by session_id, with fixture-defined key fields 100% correct.
• WebSocket open/close and sent/received frames correlate to their connection; HTTPS and HTTP/2 remain observable without MITM.
• Sensitive values are redacted by default. Oversized bodies are not persisted and retain a truncation or unavailable reason.
• Query and export are paginated and session-isolated; unauthorized sessions cannot read or export another session.
• CDP disconnects, target closure, dropped events, and individual body-read failures are reported or recoverable without terminating the browser workload.
• Local end-to-end coverage includes connection, collection, correlation, redaction, storage, query, export, and disconnect, with no VNC, MITM, proxy certificate, or CA dependency.

Iteration plan

• M0: confirm clark-browser CDP startup/connection, event coverage, stable schema, and one local page-load loop.
• M1: deliver single-instance collection, redaction/limits, JSONL query/export, disconnect recovery, and local fixtures.
• M2: add multiple targets/sessions, tiered storage, an asynchronous queue, metrics/alerts, and permission audit when evidence warrants them.
• M3: add explicitly authorized Fetch mutation, replay, versioned rules, and browser-cluster access only after earlier phases validate the need.

Non-goals

• Do not add VNC/noVNC, desktop streaming, or GUI browser control.
• Do not add MITM, proxy certificates, CA injection, TLS interception, packet capture, or SSLKEYLOGFILE flows.
• Do not add proxy pools, IP rotation, fingerprint generation, account operations, DOM scraping, or media decoding.

Security and reliability

• Never commit or log CDP tokens, cookies, Authorization, Set-Cookie, credentials, full sensitive bodies, or production data.
• Use short-lived CDP credentials over a protected internal or TLS connection, with session isolation and auditable access.
• Redact sensitive headers and bodies before persistence or export; enforce body, event, frame, and retention limits.
• Treat response bodies as optional: cache, target closure, protocol behavior, or lifecycle timing may make them unavailable.
• Report disconnects, target closure, dropped events, and body-read failures without stopping the browser workload.
• Validate all external input and preserve compatibility of persisted schemas and exported data.

Constraints and validation

• CDP does not cover non-browser processes, WebRTC media contents, or final rendered page state.
• Large bodies, high-frequency WebSocket frames, and multi-instance concurrency create memory, throughput, and privacy costs; prefer bounded, sampled, or discardable handling in the MVP.
• Advance beyond M0 only when local fixtures and a local test page demonstrate event/correlation completeness, zero sensitive-value leakage, body-limit behavior, disconnect recovery, and acceptable single-instance resource cost.
• Trigger ordinary HTTP, a failed request, HTTPS, HTTP/2 where available, and a WebSocket during validation; never require external production systems.
• Validate the product by reducing repeated debugging and collection glue code before building clusters, paid proxy integrations, or an operations console.

Development and tests

• Keep commits focused; do not add speculative scaffolding, dependencies, CI, or an unconfirmed application stack.
• Use local fixtures only. Never use production credentials, send real external messages, deploy, publish, or perform destructive data operations.
• Add the smallest relevant test for non-trivial behavior.
• Cover request-to-response completion/failure correlation, WebSocket frames, redaction, truncation, session isolation, pagination, disconnect recovery, and unavailable response bodies where applicable.
• Run relevant tests and formatting checks before pushing.