Goals
Q2 2026 objectives
Isolate ingestion Kafka topics
Kafka infrastructure is shared across ingestion and other teams, making it much harder to maintain ingestion reliability and scale independently.
- Separate Kafka infrastructure for ingestion from other teams
- Evaluate WarpStream as a full MSK replacement
- Fall back to multi-MSK cluster setup if WarpStream doesn't pan out
PersonHog
Person processing should be stable and fully handled by PersonHog.
- Route all reads and writes through PersonHog — no direct DB calls from other services
- Define and guarantee SLAs for read/write latency and availability
- Ship the read/write split architecture to production
- Roll out safely with strong validation of data
Redesign the capture contract
Capture lacks the structure needed for strong rate limiting and efficient downstream processing.
- Define a new capture contract with a strict schema for all endpoints
- Include critical metadata in headers so downstream consumers don't need to parse payloads
- Standardize response and error codes for SDKs
- Establish the SDK harness as the canonical way to stay on contract
- Migrate all SDKs to the new contract
- Enable rate limiting
Define and commit to ingestion SLAs
The primary ingestion lane needs consistently low latency, not just high availability.
- Publish metrics and SLAs for all pipelines, especially the general lane
- Automate monitoring to protect 99.99% of traffic from the 0.01% causing lag
Improve per-partition throughput
Current per-partition processing speed won't support 10x growth.
- Benchmark and identify per-partition bottlenecks
- Evaluate alternative languages or architectures
- Reduce the number of running ingestion pods
Make ingestion pipelines easy to manage
The ingestion team will own more pipelines over time and needs better tooling to manage them efficiently.
- Make Helm charts easy to configure for deploying new pipelines
- Provision metrics and alerting automatically with each pipeline
- Build MCP tooling for interactive pipeline support
Handbook
Features
- Client data capture
- Session recording capture
- Identity resolution
- Property definitions