Overview
Design a distributed telemetry monitoring infrastructure like Prometheus or Datadog capable of ingesting millions of metrics per second and evaluating alerting rules in real time.
Requirements
- 1Ingest 10M+ time-series metric data points per second from thousands of host servers and microservices
- 2Support fast interactive query aggregations (e.g. p50, p95, p99 latency, CPU averages over time windows)
- 3Real-time alerting engine evaluating threshold breach rules every 10-30 seconds
- 4Efficient data retention tiering: raw 1s data for 7 days, downsampled 1m data for 90 days, 1h data for 1 year
- 5High write throughput with zero data loss during traffic spikes
Architecture Hints (4)
Target Concepts
#TSDB#Time Series#Kafka#Flink#Alerting Engine