S3ntinel
Personal project · 2025
A multi-tenant observability platform that streams CPU, memory, network, and disk metrics from EC2 fleets over WebSockets, with sub-second alert latency and automated CloudWatch Agent rollout via AWS SSM.
10,000+
metrics / minute
<1s
alert latency
99.9%
data accuracy
Problem
Teams running EC2 fleets usually monitor them through the CloudWatch console: slow to navigate, scoped to one account, and not built for watching many instances at once. Getting a live, fleet-wide view — and getting alerted within seconds when something degrades — means stitching together dashboards, alarms, and per-instance agent setup by hand.
S3ntinel solves this as a single multi-tenant platform: organizations connect their AWS accounts, agents are deployed automatically, and metrics stream into a live dashboard in real time instead of on a polling delay.
Architecture
EC2 fleet
CloudWatch Agent on each instance
AWS SSM
automated agent deployment
Ingestion service
Go · AWS SDK · per-tenant collectors
Time-series database
retention + downsampling
Alert evaluator
threshold rules, sub-second
Live dashboard
Next.js · streaming charts per instance
Technical Decisions
WebSockets instead of polling
Dashboards that poll every 30–60 seconds are fine for capacity planning but useless for incident response. Persistent WebSocket connections let the server push each metric batch the moment it lands, which is what makes sub-second alert delivery possible. The trade-off is connection lifecycle management — heartbeats, reconnection with backoff, and resubscribing to the right tenant streams — which I built into the protocol from the start.
A time-series database, not Postgres
At 10,000+ data points per minute, a row-per-point relational schema bloats fast and aggregate queries (p95 CPU over 6 hours across 40 instances) get expensive. A purpose-built time-series store gives cheap appends, native downsampling, and time-bucketed queries — the access pattern is always 'metric × instance × time range', so the storage engine should be built around exactly that.
AWS SSM for agent rollout
Asking users to SSH into every instance and install the CloudWatch Agent kills onboarding. SSM Run Command lets the platform push agent installation and configuration to an entire fleet without inbound network access or key management. Onboarding a new organization becomes: connect the account, pick instances, done.
Multi-tenant from day one
Tenancy is enforced at the ingestion boundary — every metric is tagged with its organization at write time, and WebSocket subscriptions are scoped server-side so a client can only ever receive streams for its own tenant. Retrofitting isolation onto a single-tenant design later is far harder than building it in.
Challenges
Fan-out under sustained load
Pushing 10,000+ points per minute to many concurrent dashboard connections naively means every write triggers N socket sends. I batched outbound messages per connection on a short flush interval and moved serialization out of the hot path, so a burst of metrics becomes one frame per client per tick instead of hundreds. Slow clients get bounded buffers — if a consumer can't keep up, it drops to a snapshot-and-resume rather than back-pressuring the whole pipeline.
Keeping alerts under one second
Alert evaluation initially sat behind the database write, which put storage latency in the critical path. Splitting the pipeline so the evaluator consumes the ingest stream directly — in parallel with persistence — cut alert latency to sub-second consistently, because an alert no longer waits for a disk write to complete.
Agent configuration drift
With agents deployed across many instances, configs drift: an instance gets re-imaged, an agent dies silently, a region gets missed. Treating SSM as the reconciliation mechanism — periodically asserting desired agent state rather than installing once and hoping — is what got data accuracy to 99.9% instead of slowly decaying as fleets changed.