374 lines
17 KiB
Markdown
374 lines
17 KiB
Markdown
# W06 cloud mTLS Agent RPC smoke (2026-09-18)
|
|
|
|
## Scope and boundary
|
|
|
|
This is a disposable, mock-mode deployment check on the owner-authorized Debian
|
|
13 ECS host. It does not represent W06 completion, G0, P1, or production
|
|
mTLS authorization. The listener was bound to host loopback
|
|
`127.0.0.1:19090`; no public port or security-group rule was opened.
|
|
|
|
The test used:
|
|
|
|
- the deployed Agent release binary SHA-256
|
|
`6047a9a6a1f52db52e326167f1f50a17d8c46c34b34327294657d35c87d10cf9`;
|
|
- `SIP_GO_AGENT_MODE=mock`, Agent ID `agent-mtls`, Cell ID `cell-mtls`;
|
|
- TLS 1.3 with a disposable one-day CA, server certificate SAN
|
|
`agent.test`/`127.0.0.1`, and client certificate SAN `dispatcher.test`;
|
|
- generated Go stubs plus a temporary client outside the project production
|
|
binary; the client verified the server name and presented the client cert.
|
|
|
|
The temporary private keys were stored only under the remote disposable test
|
|
folder and removed after the run. No key or certificate contents are included
|
|
here.
|
|
|
|
## Procedure and output
|
|
|
|
The Agent was launched with the deployment TLS files and
|
|
`AGENT_GRPC_LISTEN=127.0.0.1:19090`. The temporary client connected with
|
|
TLS 1.3, called `ActivateAgent`, then called `GetAgentStatus` with the returned
|
|
session generation and binding tuple.
|
|
|
|
```text
|
|
agent_ready=1
|
|
activation_state=ACTIVATION_STATE_ACTIVE session_generation=1
|
|
status_agent=agent-mtls status_cell=cell-mtls mtls_authenticated=true session_active=true admission=ADMISSION_STATE_CLOSED
|
|
client_rc=0
|
|
```
|
|
|
|
Public test-artifact hashes, retained only to identify this run:
|
|
|
|
```text
|
|
ca.pem cdca06a27b8338a533b603fc6f27f4877973d9c87cb317052e1dac9839725536
|
|
server.pem a0eeaa5590e2e27657098a972bf563c694065ceae36313e7fe598e6c0e06cc79
|
|
client.pem 1ee5cdbfb9a347593319508f368fd9ff922a98be61fdd68f86fa155f65d98fce
|
|
client d0c567076b4e4d2c25979955e9698bb74546545eff15d0b00c627a1614b615d1
|
|
```
|
|
|
|
A follow-up negative transport check used the same server CA plus a client
|
|
certificate signed by a separate disposable rogue CA. The temporary client
|
|
was extended only to omit the certificate for this check; its hash was
|
|
`34e0b36417064be68410bbf6dbe5dbafe69f6a7471c6f80e2e959f1dab22a34a`.
|
|
|
|
```text
|
|
no client certificate: gRPC ActivateAgent failed with TLS alert "certificate required"; client_rc=2
|
|
rogue client certificate: gRPC ActivateAgent failed with a broken pipe after server certificate rejection; client_rc=2
|
|
```
|
|
|
|
The negative checks were run against the same loopback listener and the
|
|
remote disposable test directory was removed afterward. The rogue CA public
|
|
certificate hash was `9012fd9b725f0561161a1bcfd1ef1ea0f2c27c2f36ab3ab04beaa7cdbe1d4ce3`.
|
|
|
|
## Project RPC client boundary
|
|
|
|
A follow-up run used the project's production `internal/rpc.DialFromFiles`
|
|
wrapper (temporary command source removed after build), rather than a
|
|
standalone TLS client. It connected to the deployed Agent binary, completed
|
|
`ActivateAgent` and `GetAgentStatus`, and returned:
|
|
|
|
```text
|
|
agent_ready=1
|
|
client=project_rpc_dial activation=ACTIVATION_STATE_ACTIVE session_generation=1 mtls_authenticated=true session_active=true
|
|
client_rc=0
|
|
```
|
|
|
|
The temporary client binary SHA-256 was
|
|
`4a20234806b6df43aaf9a6397c9be2011fd82fa185bc25dc784517d5e73db19f`.
|
|
This validates the project client/TLS boundary against the deployed Agent; it
|
|
is not a Dispatcher process or cross-host integration test.
|
|
|
|
## Pre-activation status and Coordinator binding
|
|
|
|
The pre-activation R01 path was then exercised against a temporary binary built
|
|
from the current working tree (not installed over the release at `/opt`). The
|
|
binary SHA-256 was
|
|
`03d3f3d3acdab39c6f068f38586876b5f00c3845ed0dc29c6482cabbec1566ba`.
|
|
A temporary command using the project's `dispatcher.AgentCoordinator` and
|
|
`internal/rpc.DialFromFiles` performed `Probe`, used the returned boot ID for
|
|
`Activate`, and read the session-authorized status:
|
|
|
|
```text
|
|
agent_ready=1
|
|
probe_boot=agent-coordinator-client-1789728978523557105 probe_session_active=false activation_state=ACTIVATION_STATE_ACTIVE session_generation=1 active_session=true mtls_authenticated=true
|
|
client_rc=0
|
|
```
|
|
|
|
The temporary Coordinator client SHA-256 was
|
|
`b6f3f6aaf3808e72c3a7d2b8448bf74b2b513d89be8c084a3941ca5853212917`.
|
|
This proves the project R01 pre-activation probe and R02 session binding
|
|
against the current Agent implementation in mock mode. It does not prove the
|
|
Dispatcher Cobra process, cross-host endpoint inventory, or production
|
|
release deployment; the temporary binary, client, certificates, and spool
|
|
were removed afterward.
|
|
|
|
## Dispatcher Cobra endpoint-inventory smoke
|
|
|
|
The current working-tree binary was then run as both the mock Agent and the
|
|
actual `dispatcher` Cobra command. The Dispatcher used a strict,
|
|
deployment-owned endpoint inventory and its mTLS client certificate; no tenant
|
|
or command payload selected the endpoint. The temporary Agent/Dispatcher
|
|
binary SHA-256 was
|
|
`35510789e6797267010a57b365198b086ff794fed4615bddfe045d8229a492e6`, and the
|
|
inventory SHA-256 was
|
|
`d344dc4db123ea2583805bb7d20fa27cc0da98b7f99ce6f00334ce7cf88edd70`.
|
|
|
|
```text
|
|
agent_ready=1
|
|
dispatcher_rc=0
|
|
dispatcher_output:
|
|
{"agent_sessions":[{"agent_id":"agent-dispatcher-poc","boot_id":"agent-dispatcher-poc-1789731033229003007","cell_id":"cell-dispatcher-poc","session_generation":1}],"db":"/home/rogee/dispatcher-poc/dispatcher.db","mode":"mock","role":"dispatcher"}
|
|
```
|
|
|
|
The run used one loopback endpoint (`127.0.0.1:19090`) and was cleaned up
|
|
along with the temporary mTLS files and SQLite database. This validates the
|
|
first-version Dispatcher startup path: inventory load, mTLS dial, R01 probe,
|
|
boot binding, R02 activation, and JSON session reporting. It does not prove
|
|
two-Cell/cross-host operation, endpoint-role authorization matrix, task
|
|
execution through the scheduler, RabbitMQ application receipt, health/version
|
|
fault handling, or production release deployment.
|
|
|
|
## Two-Agent/two-Cell same-host inventory smoke
|
|
|
|
Using the same current-tree binaries and disposable mTLS files, the endpoint
|
|
inventory was expanded to two Agent/Cell bindings on two loopback listeners
|
|
(`127.0.0.1:19090` and `127.0.0.1:19091`). The binary hash remained
|
|
`35510789e6797267010a57b365198b086ff794fed4615bddfe045d8229a492e6`; the
|
|
two-entry inventory hash was
|
|
`917241cb50685002b4ba9666ea9a0096d733357b52cbb929f7e74980761a0803`.
|
|
|
|
```text
|
|
agents_ready=2
|
|
dispatcher_rc=0
|
|
dispatcher_output:
|
|
{"agent_sessions":[{"agent_id":"agent-cell-a","boot_id":"agent-cell-a-1789731290744754658","cell_id":"cell-a","session_generation":1},{"agent_id":"agent-cell-b","boot_id":"agent-cell-b-1789731290740322704","cell_id":"cell-b","session_generation":1}],"db":"/home/rogee/dispatcher-two-poc/dispatcher.db","mode":"mock","role":"dispatcher"}
|
|
```
|
|
|
|
Both Agent processes, the Dispatcher process, the temporary SQLite database,
|
|
and mTLS files were removed after the run. This is a same-host/two-loopback
|
|
smoke only; it does not prove two physical Cells, cross-host networking,
|
|
separate egress, endpoint-role authorization, task execution, health/failover,
|
|
or P1 capacity.
|
|
|
|
## Post-bind-authorization repeat
|
|
|
|
After adding server-side binding of activation/status metadata to the Agent's
|
|
configured `AgentStatus.AgentId` and `CellId`, the same two-loopback inventory
|
|
was repeated. The current-tree binary SHA-256 was
|
|
`bd74ade951eb542ec07ec488e1517530c4179f2bd81513d920ff7a81eeafbfbc` and the
|
|
inventory remained
|
|
`917241cb50685002b4ba9666ea9a0096d733357b52cbb929f7e74980761a0803`.
|
|
|
|
```text
|
|
agents_ready=2
|
|
dispatcher_rc=0
|
|
dispatcher_output:
|
|
{"agent_sessions":[{"agent_id":"agent-cell-a","boot_id":"agent-cell-a-1789731637923625989","cell_id":"cell-a","session_generation":1},{"agent_id":"agent-cell-b","boot_id":"agent-cell-b-1789731637925117185","cell_id":"cell-b","session_generation":1}],"db":"/home/rogee/dispatcher-two-poc/dispatcher.db","mode":"mock","role":"dispatcher"}
|
|
```
|
|
|
|
This repeat plus local `TestAgentIdentityIsBoundToConfiguredEndpoint` proves
|
|
that a configured Agent rejects an activation/status request carrying another
|
|
Agent/Cell identity. It is still not an endpoint-role certificate allowlist,
|
|
cross-host test, or production authorization matrix.
|
|
|
|
## Endpoint identity misbinding rejection
|
|
|
|
A negative endpoint-inventory run pointed `agent-cell-b`/`cell-b` at the
|
|
listener configured as `agent-cell-a`/`cell-a`. The current Dispatcher and
|
|
Agent binaries rejected the R01 probe before activation:
|
|
|
|
```text
|
|
agent_ready=1
|
|
dispatcher_rc=1
|
|
dispatcher_output:
|
|
2026/09/18 19:46:57 ERROR command failed error="probe Agent \"agent-cell-b\": rpc error: code = PermissionDenied desc = request Agent identity is not bound to this endpoint"
|
|
```
|
|
|
|
The wrong-inventory SHA-256 was
|
|
`267f25e990673066dd2f383f407a564c6558135be7d8eaf0081711469c68d330`. This
|
|
confirms that a valid mTLS peer cannot select another configured Agent/Cell by
|
|
changing deployment metadata. It does not replace a full certificate-role
|
|
allowlist or cross-host authorization matrix.
|
|
|
|
## Restart-based trust-root rotation smoke
|
|
|
|
A restart-based certificate rotation was exercised with the current-tree
|
|
binary. Phase one used CA1/server1/client1 and completed Dispatcher startup;
|
|
phase two restarted the Agent with CA2/server2. The old client certificate
|
|
was presented with a CA1+CA2 trust bundle so the new server certificate was
|
|
trusted while the old client chain remained testable. The new client used
|
|
CA2/client2.
|
|
|
|
Public artifact hashes:
|
|
|
|
```text
|
|
agent binary bd74ade951eb542ec07ec488e1517530c4179f2bd81513d920ff7a81eeafbfbc
|
|
ca2.pem e8b86883a446f8dce32080244c9f587f70618385516e6db7aee19a157fd91160
|
|
server2.pem a777f1c1131bd9292e12869b0b2e76529eb0c11295b386daa368308ebd994dba
|
|
client2.pem 1bd0df7e79b91504bcf1e17f03c10a4f785d9b82a584c7c1e7bb40bd2b6acbcf
|
|
old-client-bundle 7e5df4438028307e44ed0a497e438754a7c179b6dce85938994c22d9f56dd209
|
|
```
|
|
|
|
```text
|
|
phase_one_rc=0
|
|
old_client_rc=1
|
|
old_client_output: rpc error: code = Unavailable ... write: broken pipe
|
|
new_client_rc=0
|
|
new_client_output: {"agent_sessions":[{"agent_id":"agent-dispatcher-poc","cell_id":"cell-dispatcher-poc","session_generation":1}]...}
|
|
```
|
|
|
|
The old client was rejected after the Agent restarted with the replacement
|
|
trust root, while the replacement client completed R01/R02 successfully.
|
|
Temporary processes, keys, endpoint inventory, databases, and spool data were
|
|
removed. This proves restart-based trust-root replacement and old-client
|
|
rejection in mock mode; it does not prove live rotation without restart,
|
|
revocation distribution, shared-certificate fleet rotation, or production
|
|
health/failover.
|
|
|
|
## Dispatcher restart session-generation smoke
|
|
|
|
With one Agent left running, the current Dispatcher Cobra process was started
|
|
once with a fresh SQLite database and then started again with a different
|
|
Dispatcher ID/database. The startup path used generation `0`, allowing the
|
|
Agent's durable session journal to allocate the next generation rather than
|
|
self-fencing a restarted Dispatcher.
|
|
|
|
The current-tree binary SHA-256 was
|
|
`3db047d71e2d87cacde8fb645de9476b92662abcf093092150a4a71f747373ad`, and the
|
|
one-entry inventory remained
|
|
`d344dc4db123ea2583805bb7d20fa27cc0da98b7f99ce6f00334ce7cf88edd70`.
|
|
|
|
```text
|
|
agent_ready=1
|
|
first_rc=0
|
|
first_output:
|
|
{"agent_sessions":[{"agent_id":"agent-dispatcher-poc","boot_id":"agent-dispatcher-poc-1789732334849315059","cell_id":"cell-dispatcher-poc","session_generation":1}],"db":"/home/rogee/dispatcher-restart-poc/dispatcher-one.db","mode":"mock","role":"dispatcher"}
|
|
second_rc=0
|
|
second_output:
|
|
{"agent_sessions":[{"agent_id":"agent-dispatcher-poc","boot_id":"agent-dispatcher-poc-1789732334849315059","cell_id":"cell-dispatcher-poc","session_generation":2}],"db":"/home/rogee/dispatcher-restart-poc/dispatcher-two.db","mode":"mock","role":"dispatcher"}
|
|
```
|
|
|
|
This proves same-boot Dispatcher restart fencing/recovery in mock mode. It
|
|
does not prove crash-safe Dispatcher session ownership across machines, active
|
|
call migration, or production HA.
|
|
|
|
## Project Coordinator Execute over mTLS
|
|
|
|
A temporary project command using `internal/rpc.DialFromFiles` and
|
|
`dispatcher.AgentCoordinator.ExecuteRaw` exercised the full mock execution
|
|
boundary against the current Agent binary: R01 probe, R02 activation,
|
|
execution-permit issuance, Execute, and durable receipt. It used the approved
|
|
contract fixture only; no SIP leg or telephone call was attempted.
|
|
|
|
Artifact hashes:
|
|
|
|
```text
|
|
Agent binary 3db047d71e2d87cacde8fb645de9476b92662abcf093092150a4a71f747373ad
|
|
Coordinator client 2afa146c15cfe2fa792550e0a21a1eaba73e32cad5504e5442fa389c202140ed
|
|
call.execute fixture 0164664fd3503d72668b24bdecb623aa0414ce58191960b868abe8454988c742
|
|
```
|
|
|
|
```text
|
|
probe_boot=agent-execute-1789732783527268138 session_generation=1 permit=true receipt=true receipt_result=RESULT_CODE_ACCEPTED unknown=false state=EXECUTION_STATE_PERMIT_GRANTED
|
|
agent_ready=1
|
|
client_rc=0
|
|
```
|
|
|
|
This validates the project Dispatcher/Agent execution seam over loopback mTLS
|
|
in `mock` mode. It does not prove SIP originate, real AI/media, RabbitMQ
|
|
application receipt, or production call execution.
|
|
|
|
## Dispatcher SQLite reservation to Execute over mTLS
|
|
|
|
A second temporary project command exercised the full Dispatcher seam: it
|
|
opened SQLite, installed tenant/global/Cell quotas, ingested the contract
|
|
fixture, reserved the task, called `dispatcher.ExecuteReserved` through the
|
|
mTLS Agent Coordinator, and checked the persisted task status.
|
|
|
|
```text
|
|
Dispatcher client f3e540b0aea15596d621455f1df51a824509a983dca399e56b31972d71b199ad
|
|
Agent binary 3db047d71e2d87cacde8fb645de9476b92662abcf093092150a4a71f747373ad
|
|
call.execute fixture 0164664fd3503d72668b24bdecb623aa0414ce58191960b868abe8454988c742
|
|
|
|
probe_boot=agent-execute-1789733076935907928 permit=true receipt=true receipt_result=RESULT_CODE_ACCEPTED unknown=false task_status=running
|
|
agent_ready=1
|
|
client_rc=0
|
|
```
|
|
|
|
This is a mock loopback reservation-to-Agent execution smoke. It does not
|
|
prove RabbitMQ application receipt, SIP originate, real AI/media, call
|
|
completion, or production quota/fault-injection acceptance.
|
|
|
|
## Two-Agent SQLite reservation to Execute over mTLS
|
|
|
|
The two-loopback Agent inventory was also exercised through a temporary project
|
|
Dispatcher client that created two SQLite reservations with separate Cell
|
|
scopes and executed them against the two registered Agents. Both permits and
|
|
receipts were accepted and both task rows reached `running`.
|
|
|
|
```text
|
|
Dispatcher client 0d280c80880d1dafa46bd401704aea5c854a439d61e777f8e1b3cce039ac023d
|
|
Agent binary 3db047d71e2d87cacde8fb645de9476b92662abcf093092150a4a71f747373ad
|
|
call.execute fixture 0164664fd3503d72668b24bdecb623aa0414ce58191960b868abe8454988c742
|
|
|
|
session_a=1 session_b=1 permit_a=true receipt_a=true permit_b=true receipt_b=true result_a=RESULT_CODE_ACCEPTED result_b=RESULT_CODE_ACCEPTED task_a=running task_b=running
|
|
agents_ready=2
|
|
client_rc=0
|
|
```
|
|
|
|
This is same-host/two-loopback mock evidence for separate Cell reservations
|
|
and Agent execution. It does not prove physical Cell separation, final
|
|
cross-Cell last-originate barriers, RabbitMQ application receipt, SIP/media,
|
|
failover, or capacity.
|
|
|
|
## Dispatcher certificate fingerprint allowlist smoke
|
|
|
|
The Agent was restarted with `MTLS_PEER_CERT_FINGERPRINTS` containing only the
|
|
SHA-256 leaf fingerprint of the approved Dispatcher client certificate. A
|
|
second client certificate was signed by the same trusted CA but was not in the
|
|
allowlist.
|
|
|
|
```text
|
|
Agent binary 5abc8bb43076927c4eb1e04e2baf0e305715eaf019a7d2e87ccaa86ebaf6f6a5
|
|
allowed client leaf d7965d2e345d7f7496c2913f5a2b673c84cbddc72d93df47abb87c29d5f87b0c
|
|
rejected client leaf 6339698cf4979b26424203ac689ef4da9279aed2dd3222036b06f5a42ab0db4a
|
|
|
|
agent_ready=1
|
|
allowed_rc=0
|
|
rejected_rc=1
|
|
rejected_output: rpc error: code = PermissionDenied desc = mTLS certificate is not in the endpoint allowlist
|
|
```
|
|
|
|
Both certificates chained to the same CA, so the negative result exercises the
|
|
leaf allowlist rather than CA trust failure. This proves a deployment-owned
|
|
Dispatcher certificate allowlist in mock mode; it does not prove fleet-wide
|
|
role rotation, revocation distribution, or cross-host authorization.
|
|
|
|
## Draft service-unit verification
|
|
|
|
The two draft units under `deploy/systemd/` were copied to the same host's
|
|
`/tmp` directory and checked with:
|
|
|
|
```text
|
|
systemd-analyze verify /tmp/sip-go-agent-agent.service /tmp/sip-go-agent-dispatcher.service
|
|
exit=0
|
|
```
|
|
|
|
The command emitted only pre-existing warnings for the unrelated host
|
|
`cloudmonitor.service`; it emitted no warning or error for either project unit.
|
|
The units were not installed, enabled, or started by this check. This is a
|
|
syntax/executable-path check, not systemd runtime, restart, or production
|
|
hardening acceptance.
|
|
|
|
## Result
|
|
|
|
**Passed in isolation:** the deployed Agent accepted a verified client
|
|
certificate, completed the activation/session-generation exchange, and returned
|
|
`mtls_authenticated=true` and `session_active=true` over a TLS 1.3 Unary gRPC
|
|
call. The test also exercised the Agent's actual file-backed spool startup on
|
|
the cloud host.
|
|
|
|
**Not proven:** Dispatcher-to-Agent process interoperability, endpoint/role
|
|
allow-list authorization, certificate rotation/revocation, static artifact
|
|
activation, real/mixed mode, cross-host networking, health/version fault
|
|
handling, call execution, or two-Cell/P1 acceptance. The result is a cloud
|
|
mTLS smoke only.
|