Files
go-sip/docs/evidence/20260918-w06-mtls-cloud.md
T

17 KiB

W06 cloud mTLS Agent RPC smoke (2026-09-18)

Scope and boundary

This is a disposable, mock-mode deployment check on the owner-authorized Debian 13 ECS host. It does not represent W06 completion, G0, P1, or production mTLS authorization. The listener was bound to host loopback 127.0.0.1:19090; no public port or security-group rule was opened.

The test used:

  • the deployed Agent release binary SHA-256 6047a9a6a1f52db52e326167f1f50a17d8c46c34b34327294657d35c87d10cf9;
  • SIP_GO_AGENT_MODE=mock, Agent ID agent-mtls, Cell ID cell-mtls;
  • TLS 1.3 with a disposable one-day CA, server certificate SAN agent.test/127.0.0.1, and client certificate SAN dispatcher.test;
  • generated Go stubs plus a temporary client outside the project production binary; the client verified the server name and presented the client cert.

The temporary private keys were stored only under the remote disposable test folder and removed after the run. No key or certificate contents are included here.

Procedure and output

The Agent was launched with the deployment TLS files and AGENT_GRPC_LISTEN=127.0.0.1:19090. The temporary client connected with TLS 1.3, called ActivateAgent, then called GetAgentStatus with the returned session generation and binding tuple.

agent_ready=1
activation_state=ACTIVATION_STATE_ACTIVE session_generation=1
status_agent=agent-mtls status_cell=cell-mtls mtls_authenticated=true session_active=true admission=ADMISSION_STATE_CLOSED
client_rc=0

Public test-artifact hashes, retained only to identify this run:

ca.pem      cdca06a27b8338a533b603fc6f27f4877973d9c87cb317052e1dac9839725536
server.pem  a0eeaa5590e2e27657098a972bf563c694065ceae36313e7fe598e6c0e06cc79
client.pem  1ee5cdbfb9a347593319508f368fd9ff922a98be61fdd68f86fa155f65d98fce
client      d0c567076b4e4d2c25979955e9698bb74546545eff15d0b00c627a1614b615d1

A follow-up negative transport check used the same server CA plus a client certificate signed by a separate disposable rogue CA. The temporary client was extended only to omit the certificate for this check; its hash was 34e0b36417064be68410bbf6dbe5dbafe69f6a7471c6f80e2e959f1dab22a34a.

no client certificate: gRPC ActivateAgent failed with TLS alert "certificate required"; client_rc=2
rogue client certificate: gRPC ActivateAgent failed with a broken pipe after server certificate rejection; client_rc=2

The negative checks were run against the same loopback listener and the remote disposable test directory was removed afterward. The rogue CA public certificate hash was 9012fd9b725f0561161a1bcfd1ef1ea0f2c27c2f36ab3ab04beaa7cdbe1d4ce3.

Project RPC client boundary

A follow-up run used the project's production internal/rpc.DialFromFiles wrapper (temporary command source removed after build), rather than a standalone TLS client. It connected to the deployed Agent binary, completed ActivateAgent and GetAgentStatus, and returned:

agent_ready=1
client=project_rpc_dial activation=ACTIVATION_STATE_ACTIVE session_generation=1 mtls_authenticated=true session_active=true
client_rc=0

The temporary client binary SHA-256 was 4a20234806b6df43aaf9a6397c9be2011fd82fa185bc25dc784517d5e73db19f. This validates the project client/TLS boundary against the deployed Agent; it is not a Dispatcher process or cross-host integration test.

Pre-activation status and Coordinator binding

The pre-activation R01 path was then exercised against a temporary binary built from the current working tree (not installed over the release at /opt). The binary SHA-256 was 03d3f3d3acdab39c6f068f38586876b5f00c3845ed0dc29c6482cabbec1566ba. A temporary command using the project's dispatcher.AgentCoordinator and internal/rpc.DialFromFiles performed Probe, used the returned boot ID for Activate, and read the session-authorized status:

agent_ready=1
probe_boot=agent-coordinator-client-1789728978523557105 probe_session_active=false activation_state=ACTIVATION_STATE_ACTIVE session_generation=1 active_session=true mtls_authenticated=true
client_rc=0

The temporary Coordinator client SHA-256 was b6f3f6aaf3808e72c3a7d2b8448bf74b2b513d89be8c084a3941ca5853212917. This proves the project R01 pre-activation probe and R02 session binding against the current Agent implementation in mock mode. It does not prove the Dispatcher Cobra process, cross-host endpoint inventory, or production release deployment; the temporary binary, client, certificates, and spool were removed afterward.

Dispatcher Cobra endpoint-inventory smoke

The current working-tree binary was then run as both the mock Agent and the actual dispatcher Cobra command. The Dispatcher used a strict, deployment-owned endpoint inventory and its mTLS client certificate; no tenant or command payload selected the endpoint. The temporary Agent/Dispatcher binary SHA-256 was 35510789e6797267010a57b365198b086ff794fed4615bddfe045d8229a492e6, and the inventory SHA-256 was d344dc4db123ea2583805bb7d20fa27cc0da98b7f99ce6f00334ce7cf88edd70.

agent_ready=1
dispatcher_rc=0
dispatcher_output:
{"agent_sessions":[{"agent_id":"agent-dispatcher-poc","boot_id":"agent-dispatcher-poc-1789731033229003007","cell_id":"cell-dispatcher-poc","session_generation":1}],"db":"/home/rogee/dispatcher-poc/dispatcher.db","mode":"mock","role":"dispatcher"}

The run used one loopback endpoint (127.0.0.1:19090) and was cleaned up along with the temporary mTLS files and SQLite database. This validates the first-version Dispatcher startup path: inventory load, mTLS dial, R01 probe, boot binding, R02 activation, and JSON session reporting. It does not prove two-Cell/cross-host operation, endpoint-role authorization matrix, task execution through the scheduler, RabbitMQ application receipt, health/version fault handling, or production release deployment.

Two-Agent/two-Cell same-host inventory smoke

Using the same current-tree binaries and disposable mTLS files, the endpoint inventory was expanded to two Agent/Cell bindings on two loopback listeners (127.0.0.1:19090 and 127.0.0.1:19091). The binary hash remained 35510789e6797267010a57b365198b086ff794fed4615bddfe045d8229a492e6; the two-entry inventory hash was 917241cb50685002b4ba9666ea9a0096d733357b52cbb929f7e74980761a0803.

agents_ready=2
dispatcher_rc=0
dispatcher_output:
{"agent_sessions":[{"agent_id":"agent-cell-a","boot_id":"agent-cell-a-1789731290744754658","cell_id":"cell-a","session_generation":1},{"agent_id":"agent-cell-b","boot_id":"agent-cell-b-1789731290740322704","cell_id":"cell-b","session_generation":1}],"db":"/home/rogee/dispatcher-two-poc/dispatcher.db","mode":"mock","role":"dispatcher"}

Both Agent processes, the Dispatcher process, the temporary SQLite database, and mTLS files were removed after the run. This is a same-host/two-loopback smoke only; it does not prove two physical Cells, cross-host networking, separate egress, endpoint-role authorization, task execution, health/failover, or P1 capacity.

Post-bind-authorization repeat

After adding server-side binding of activation/status metadata to the Agent's configured AgentStatus.AgentId and CellId, the same two-loopback inventory was repeated. The current-tree binary SHA-256 was bd74ade951eb542ec07ec488e1517530c4179f2bd81513d920ff7a81eeafbfbc and the inventory remained 917241cb50685002b4ba9666ea9a0096d733357b52cbb929f7e74980761a0803.

agents_ready=2
dispatcher_rc=0
dispatcher_output:
{"agent_sessions":[{"agent_id":"agent-cell-a","boot_id":"agent-cell-a-1789731637923625989","cell_id":"cell-a","session_generation":1},{"agent_id":"agent-cell-b","boot_id":"agent-cell-b-1789731637925117185","cell_id":"cell-b","session_generation":1}],"db":"/home/rogee/dispatcher-two-poc/dispatcher.db","mode":"mock","role":"dispatcher"}

This repeat plus local TestAgentIdentityIsBoundToConfiguredEndpoint proves that a configured Agent rejects an activation/status request carrying another Agent/Cell identity. It is still not an endpoint-role certificate allowlist, cross-host test, or production authorization matrix.

Endpoint identity misbinding rejection

A negative endpoint-inventory run pointed agent-cell-b/cell-b at the listener configured as agent-cell-a/cell-a. The current Dispatcher and Agent binaries rejected the R01 probe before activation:

agent_ready=1
dispatcher_rc=1
dispatcher_output:
2026/09/18 19:46:57 ERROR command failed error="probe Agent \"agent-cell-b\": rpc error: code = PermissionDenied desc = request Agent identity is not bound to this endpoint"

The wrong-inventory SHA-256 was 267f25e990673066dd2f383f407a564c6558135be7d8eaf0081711469c68d330. This confirms that a valid mTLS peer cannot select another configured Agent/Cell by changing deployment metadata. It does not replace a full certificate-role allowlist or cross-host authorization matrix.

Restart-based trust-root rotation smoke

A restart-based certificate rotation was exercised with the current-tree binary. Phase one used CA1/server1/client1 and completed Dispatcher startup; phase two restarted the Agent with CA2/server2. The old client certificate was presented with a CA1+CA2 trust bundle so the new server certificate was trusted while the old client chain remained testable. The new client used CA2/client2.

Public artifact hashes:

agent binary       bd74ade951eb542ec07ec488e1517530c4179f2bd81513d920ff7a81eeafbfbc
ca2.pem            e8b86883a446f8dce32080244c9f587f70618385516e6db7aee19a157fd91160
server2.pem        a777f1c1131bd9292e12869b0b2e76529eb0c11295b386daa368308ebd994dba
client2.pem        1bd0df7e79b91504bcf1e17f03c10a4f785d9b82a584c7c1e7bb40bd2b6acbcf
old-client-bundle  7e5df4438028307e44ed0a497e438754a7c179b6dce85938994c22d9f56dd209
phase_one_rc=0
old_client_rc=1
old_client_output: rpc error: code = Unavailable ... write: broken pipe
new_client_rc=0
new_client_output: {"agent_sessions":[{"agent_id":"agent-dispatcher-poc","cell_id":"cell-dispatcher-poc","session_generation":1}]...}

The old client was rejected after the Agent restarted with the replacement trust root, while the replacement client completed R01/R02 successfully. Temporary processes, keys, endpoint inventory, databases, and spool data were removed. This proves restart-based trust-root replacement and old-client rejection in mock mode; it does not prove live rotation without restart, revocation distribution, shared-certificate fleet rotation, or production health/failover.

Dispatcher restart session-generation smoke

With one Agent left running, the current Dispatcher Cobra process was started once with a fresh SQLite database and then started again with a different Dispatcher ID/database. The startup path used generation 0, allowing the Agent's durable session journal to allocate the next generation rather than self-fencing a restarted Dispatcher.

The current-tree binary SHA-256 was 3db047d71e2d87cacde8fb645de9476b92662abcf093092150a4a71f747373ad, and the one-entry inventory remained d344dc4db123ea2583805bb7d20fa27cc0da98b7f99ce6f00334ce7cf88edd70.

agent_ready=1
first_rc=0
first_output:
{"agent_sessions":[{"agent_id":"agent-dispatcher-poc","boot_id":"agent-dispatcher-poc-1789732334849315059","cell_id":"cell-dispatcher-poc","session_generation":1}],"db":"/home/rogee/dispatcher-restart-poc/dispatcher-one.db","mode":"mock","role":"dispatcher"}
second_rc=0
second_output:
{"agent_sessions":[{"agent_id":"agent-dispatcher-poc","boot_id":"agent-dispatcher-poc-1789732334849315059","cell_id":"cell-dispatcher-poc","session_generation":2}],"db":"/home/rogee/dispatcher-restart-poc/dispatcher-two.db","mode":"mock","role":"dispatcher"}

This proves same-boot Dispatcher restart fencing/recovery in mock mode. It does not prove crash-safe Dispatcher session ownership across machines, active call migration, or production HA.

Project Coordinator Execute over mTLS

A temporary project command using internal/rpc.DialFromFiles and dispatcher.AgentCoordinator.ExecuteRaw exercised the full mock execution boundary against the current Agent binary: R01 probe, R02 activation, execution-permit issuance, Execute, and durable receipt. It used the approved contract fixture only; no SIP leg or telephone call was attempted.

Artifact hashes:

Agent binary          3db047d71e2d87cacde8fb645de9476b92662abcf093092150a4a71f747373ad
Coordinator client    2afa146c15cfe2fa792550e0a21a1eaba73e32cad5504e5442fa389c202140ed
call.execute fixture  0164664fd3503d72668b24bdecb623aa0414ce58191960b868abe8454988c742
probe_boot=agent-execute-1789732783527268138 session_generation=1 permit=true receipt=true receipt_result=RESULT_CODE_ACCEPTED unknown=false state=EXECUTION_STATE_PERMIT_GRANTED
agent_ready=1
client_rc=0

This validates the project Dispatcher/Agent execution seam over loopback mTLS in mock mode. It does not prove SIP originate, real AI/media, RabbitMQ application receipt, or production call execution.

Dispatcher SQLite reservation to Execute over mTLS

A second temporary project command exercised the full Dispatcher seam: it opened SQLite, installed tenant/global/Cell quotas, ingested the contract fixture, reserved the task, called dispatcher.ExecuteReserved through the mTLS Agent Coordinator, and checked the persisted task status.

Dispatcher client     f3e540b0aea15596d621455f1df51a824509a983dca399e56b31972d71b199ad
Agent binary          3db047d71e2d87cacde8fb645de9476b92662abcf093092150a4a71f747373ad
call.execute fixture  0164664fd3503d72668b24bdecb623aa0414ce58191960b868abe8454988c742

probe_boot=agent-execute-1789733076935907928 permit=true receipt=true receipt_result=RESULT_CODE_ACCEPTED unknown=false task_status=running
agent_ready=1
client_rc=0

This is a mock loopback reservation-to-Agent execution smoke. It does not prove RabbitMQ application receipt, SIP originate, real AI/media, call completion, or production quota/fault-injection acceptance.

Two-Agent SQLite reservation to Execute over mTLS

The two-loopback Agent inventory was also exercised through a temporary project Dispatcher client that created two SQLite reservations with separate Cell scopes and executed them against the two registered Agents. Both permits and receipts were accepted and both task rows reached running.

Dispatcher client    0d280c80880d1dafa46bd401704aea5c854a439d61e777f8e1b3cce039ac023d
Agent binary         3db047d71e2d87cacde8fb645de9476b92662abcf093092150a4a71f747373ad
call.execute fixture 0164664fd3503d72668b24bdecb623aa0414ce58191960b868abe8454988c742

session_a=1 session_b=1 permit_a=true receipt_a=true permit_b=true receipt_b=true result_a=RESULT_CODE_ACCEPTED result_b=RESULT_CODE_ACCEPTED task_a=running task_b=running
agents_ready=2
client_rc=0

This is same-host/two-loopback mock evidence for separate Cell reservations and Agent execution. It does not prove physical Cell separation, final cross-Cell last-originate barriers, RabbitMQ application receipt, SIP/media, failover, or capacity.

Dispatcher certificate fingerprint allowlist smoke

The Agent was restarted with MTLS_PEER_CERT_FINGERPRINTS containing only the SHA-256 leaf fingerprint of the approved Dispatcher client certificate. A second client certificate was signed by the same trusted CA but was not in the allowlist.

Agent binary             5abc8bb43076927c4eb1e04e2baf0e305715eaf019a7d2e87ccaa86ebaf6f6a5
allowed client leaf      d7965d2e345d7f7496c2913f5a2b673c84cbddc72d93df47abb87c29d5f87b0c
rejected client leaf     6339698cf4979b26424203ac689ef4da9279aed2dd3222036b06f5a42ab0db4a

agent_ready=1
allowed_rc=0
rejected_rc=1
rejected_output: rpc error: code = PermissionDenied desc = mTLS certificate is not in the endpoint allowlist

Both certificates chained to the same CA, so the negative result exercises the leaf allowlist rather than CA trust failure. This proves a deployment-owned Dispatcher certificate allowlist in mock mode; it does not prove fleet-wide role rotation, revocation distribution, or cross-host authorization.

Draft service-unit verification

The two draft units under deploy/systemd/ were copied to the same host's /tmp directory and checked with:

systemd-analyze verify /tmp/sip-go-agent-agent.service /tmp/sip-go-agent-dispatcher.service
exit=0

The command emitted only pre-existing warnings for the unrelated host cloudmonitor.service; it emitted no warning or error for either project unit. The units were not installed, enabled, or started by this check. This is a syntax/executable-path check, not systemd runtime, restart, or production hardening acceptance.

Result

Passed in isolation: the deployed Agent accepted a verified client certificate, completed the activation/session-generation exchange, and returned mtls_authenticated=true and session_active=true over a TLS 1.3 Unary gRPC call. The test also exercised the Agent's actual file-backed spool startup on the cloud host.

Not proven: Dispatcher-to-Agent process interoperability, endpoint/role allow-list authorization, certificate rotation/revocation, static artifact activation, real/mixed mode, cross-host networking, health/version fault handling, call execution, or two-Cell/P1 acceptance. The result is a cloud mTLS smoke only.