83 lines
6.6 KiB
Markdown
83 lines
6.6 KiB
Markdown
# Repository Guidelines
|
|
|
|
## Scope and current phase
|
|
|
|
- Build a CDP-only network observability and debugging control plane for clark-browser/Chromium.
|
|
- Collect and correlate CDP `Network`/`Fetch` HTTP(S), failure, cache, and WebSocket events.
|
|
- Keep the MVP read-only. Any future request or response mutation must be explicit opt-in and default off.
|
|
- Work in M0 architecture/Spike: validate one browser instance, a local test page, CDP event coverage, stable schema, redaction/truncation, and disconnect behavior before M1.
|
|
- Optimize first for one or a few isolated browser instances; do not assume cluster-scale concurrency in M0/M1.
|
|
- Prefer the smallest correct change and reuse existing patterns before adding dependencies or abstractions.
|
|
|
|
## Users and supported scenarios
|
|
|
|
- Help automation developers diagnose request failures, ordering, retries, and page behavior.
|
|
- Let data-pipeline developers consume structured browser network events downstream.
|
|
- Let operations and test users inspect session-scoped HTTP and WebSocket failure chains.
|
|
|
|
## Architecture and data flow
|
|
|
|
- CDP access validates short-lived credentials, protects the WebSocket connection, discovers targets, binds sessions, maintains heartbeats, and reports disconnects.
|
|
- The collector enables CDP `Network`/`Fetch` as needed, correlates `requestId`, records HTTP and WebSocket lifecycles, and reads response bodies only on demand.
|
|
- The policy layer normalizes events, filters by URL/resource/status, and applies redaction, sampling, size, event, and retention limits.
|
|
- Start storage and query with JSONL containing session, target, request, timing, URL/method, safe header summaries, body availability/reference, and error fields. Do not persist full bodies by default.
|
|
- Keep query and redacted export session-isolated and paginated. Re-evaluate Postgres, object storage, and asynchronous streams only after MVP evidence.
|
|
- A later console may show sessions, timelines, filters, and errors, but never control the browser GUI.
|
|
|
|
Data flow: `clark-browser (CDP enabled)` → protected CDP WebSocket → target/session binding → Network/Fetch events and on-demand body reads → normalize/filter/redact/limit → JSONL/event stream → session-scoped API/export.
|
|
|
|
CDP observes browser-internal decrypted events. It is not a client/server proxy, does not generate certificates, and must not change the network path.
|
|
|
|
## Priority
|
|
|
|
- P0: single-instance CDP access; HTTP request/response/failure and WebSocket events; `requestId` correlation; default redaction; body/event limits; JSONL export; session isolation; disconnect observability.
|
|
- P1: multiple targets/sessions, paginated query, metrics/alerts, permission audit, asynchronous event streams, and tiered storage.
|
|
- P2: Fetch mutation, versioned rules, replay, browser-cluster access, and a minimal console.
|
|
|
|
## MVP acceptance
|
|
|
|
- A page load reconstructs request → response → completion/failure by `session_id`, with fixture-defined key fields 100% correct.
|
|
- WebSocket open/close and sent/received frames correlate to their connection; HTTPS and HTTP/2 remain observable without MITM.
|
|
- Sensitive values are redacted by default. Oversized bodies are not persisted and retain a truncation or unavailable reason.
|
|
- Query and export are paginated and session-isolated; unauthorized sessions cannot read or export another session.
|
|
- CDP disconnects, target closure, dropped events, and individual body-read failures are reported or recoverable without terminating the browser workload.
|
|
- Local end-to-end coverage includes connection, collection, correlation, redaction, storage, query, export, and disconnect, with no VNC, MITM, proxy certificate, or CA dependency.
|
|
|
|
## Iteration plan
|
|
|
|
- M0: confirm clark-browser CDP startup/connection, event coverage, stable schema, and one local page-load loop.
|
|
- M1: deliver single-instance collection, redaction/limits, JSONL query/export, disconnect recovery, and local fixtures.
|
|
- M2: add multiple targets/sessions, tiered storage, an asynchronous queue, metrics/alerts, and permission audit when evidence warrants them.
|
|
- M3: add explicitly authorized Fetch mutation, replay, versioned rules, and browser-cluster access only after earlier phases validate the need.
|
|
|
|
## Non-goals
|
|
|
|
- Do not add VNC/noVNC, desktop streaming, or GUI browser control.
|
|
- Do not add MITM, proxy certificates, CA injection, TLS interception, packet capture, or `SSLKEYLOGFILE` flows.
|
|
- Do not add proxy pools, IP rotation, fingerprint generation, account operations, DOM scraping, or media decoding.
|
|
|
|
## Security and reliability
|
|
|
|
- Never commit or log CDP tokens, cookies, `Authorization`, `Set-Cookie`, credentials, full sensitive bodies, or production data.
|
|
- Use short-lived CDP credentials over a protected internal or TLS connection, with session isolation and auditable access.
|
|
- Redact sensitive headers and bodies before persistence or export; enforce body, event, frame, and retention limits.
|
|
- Treat response bodies as optional: cache, target closure, protocol behavior, or lifecycle timing may make them unavailable.
|
|
- Report disconnects, target closure, dropped events, and body-read failures without stopping the browser workload.
|
|
- Validate all external input and preserve compatibility of persisted schemas and exported data.
|
|
|
|
## Constraints and validation
|
|
|
|
- CDP does not cover non-browser processes, WebRTC media contents, or final rendered page state.
|
|
- Large bodies, high-frequency WebSocket frames, and multi-instance concurrency create memory, throughput, and privacy costs; prefer bounded, sampled, or discardable handling in the MVP.
|
|
- Advance beyond M0 only when local fixtures and a local test page demonstrate event/correlation completeness, zero sensitive-value leakage, body-limit behavior, disconnect recovery, and acceptable single-instance resource cost.
|
|
- Trigger ordinary HTTP, a failed request, HTTPS, HTTP/2 where available, and a WebSocket during validation; never require external production systems.
|
|
- Validate the product by reducing repeated debugging and collection glue code before building clusters, paid proxy integrations, or an operations console.
|
|
|
|
## Development and tests
|
|
|
|
- Keep commits focused; do not add speculative scaffolding, dependencies, CI, or an unconfirmed application stack.
|
|
- Use local fixtures only. Never use production credentials, send real external messages, deploy, publish, or perform destructive data operations.
|
|
- Add the smallest relevant test for non-trivial behavior.
|
|
- Cover request-to-response completion/failure correlation, WebSocket frames, redaction, truncation, session isolation, pagination, disconnect recovery, and unavailable response bodies where applicable.
|
|
- Run relevant tests and formatting checks before pushing.
|