Overview
Design monitoring in the style of Prometheus or Datadog that ingests millions of metrics a second and evaluates alert rules in real time.
Requirements
- 01Ingest 10M+ metric data points a second from thousands of hosts and services
- 02Answer queries like p50, p95, and p99 latency and average CPU over a time window
- 03Evaluate alert rules every 10 to 30 seconds
- 04Tier retention: raw data for 7 days, one-minute rollups for 90 days, hourly rollups for a year
- 05Write at high throughput and lose nothing during a spike
Concepts in play
TSDBTime SeriesKafkaFlinkAlerting Engine