When engineering groups at Slack, Reddit, Canva, Airbnb, Shopify, and Uber have to ship code with confidence, they depend on Buildkite. As a CI/CD platform, Buildkite orchestrates advanced construct, check, and deployment pipelines for a few of the most demanding engineering organizations on the planet. It handles all the pieces from routine code commits to synthetic intelligence (AI) model-training workloads, processing over 50 billion requests per thirty days.
On the coronary heart of Buildkite’s check orchestration portfolio is Check Engine, a specialised analytics product designed to assist engineering groups perceive and optimize their check suites at scale. Check Engine aggregates outcomes throughout hundreds of builds, flags flaky exams, runs parallel check execution throughout machine fleets, and delivers interactive analytics on check execution knowledge. It helps arbitrary metadata tagging for dimensions like occasion kind, structure, language model, cloud supplier, and have flags.
The problem? Delivering all of this in actual time, throughout a number of enterprise tenants, at a quantity that will stress even essentially the most strong knowledge infrastructure. On this publish, we discover how Buildkite makes use of Amazon Managed Streaming for Apache Kafka (Amazon MSK) and Amazon Managed Service for Apache Flink to energy Check Engine’s streaming-first analytics structure at scale.
The issue: When scale breaks conventional architectures
Buildkite’s Check Engine should ingest and serve analytics on check telemetry from hundreds of distributed pipelines concurrently, for a number of enterprise clients. The size is unforgiving: 50 billion check executions per thirty days, 500K occasions per second at peak ingestion, and webhook payloads reaching 21 MB.
The architectural evolution and its limits
The unique Rails and PostgreSQL stack couldn’t maintain this progress. In 2024, the workforce re-architected round a distributed streaming layer, a stateful stream processor for pre-aggregations, and a number of specialised shops: a key-value retailer for quick lookups, a relational database for pre-computed aggregates, and an open desk format (Iceberg) with a distributed question engine (Trino) for versatile querying.
But the core rigidity remained unsolved. Enterprise clients demanded interactive, arbitrary slicing of billions of data throughout high-cardinality dimensions, not canned stories. The stream processor couldn’t deal with advert hoc aggregations at question time. The important thing-value retailer was blind to analytical queries. The distributed question engine supplied flexibility however was too gradual for interactive use.
The end result was a system that was costly and operationally advanced. It included 9 relational database clusters, sprawling ETL pipelines, and 24/7 pre-aggregation jobs operating no matter demand. It nonetheless couldn’t ship the one factor clients wanted most: quick, versatile, interactive analytics at scale.
Structure and implementation: MSK and Amazon Managed Service for Apache Flink because the streaming spine
The answer Buildkite arrived at facilities on Amazon Managed Streaming for Apache Kafka (Amazon MSK) and Amazon Managed Service for Apache Flink because the real-time knowledge streaming and processing layers, decoupling high-throughput ingestion from downstream analytics.
The information pipeline
The next diagram exhibits the end-to-end knowledge circulate from CI/CD brokers via Amazon MSK and Amazon Managed Service for Apache Flink to the analytics layer.
Amazon MSK sits on the essential junction between knowledge producers (the distributed CI/CD brokers and check collectors operating throughout buyer infrastructure) and the downstream processing and analytics layers. Amazon Managed Service for Apache Flink then transforms these uncooked occasion streams into enriched, queryable knowledge earlier than it reaches the analytics retailer.
Excessive-throughput ingestion from CI/CD pipelines
Amazon MSK’s function begins at ingestion. Check collectors embedded in CI/CD pipelines publish check execution occasions on to Kafka subjects. The prevailing Amazon MSK cluster handles between 5 MB/sec and 100 MB/sec of inbound knowledge beneath regular working circumstances. The structure is designed to soak up the numerous variance inherent in CI/CD workloads, the place pipeline exercise is bursty and correlated with engineering workforce working hours throughout world time zones.
When the Buildkite mission was initiated, MSK Specific Brokers weren’t but accessible, main the workforce to undertake MSK Tiered Storage as the first mechanism for scaling and restoration. With MSK Specific Brokers now typically accessible, the workforce is evaluating a migration of its most crucial log ingestion workload, which sustains as much as 1 GB/s at peak ingestion. MSK Specific Brokers carry automated storage scaling with zero storage administration overhead, as much as 20x sooner scaling and 90% sooner dealer restoration, 3x increased per-broker throughput, 5x extra partitions per dealer, and built-in Clever Rebalancing.
Actual-time stream processing with Amazon Managed Service for Apache Flink
Sitting between Amazon MSK and the analytics layer, Amazon Managed Service for Apache Flink acts because the stateful stream processing engine that transforms uncooked occasion streams earlier than they attain downstream programs. Buildkite chosen Flink for its exactly-once processing, mature stateful computation mannequin, and deep Kafka integration. Dealing with sustained peaks of over 25,000 occasions per second, Amazon Managed Service for Apache Flink eliminates the operational overhead of cluster provisioning, model upgrades, checkpointing, and job restoration. This frees engineering groups to deal with utility logic.
Amazon Managed Service for Apache Flink powers key stateful processing duties, together with flaky check detection via time-windowed sample matching, enriching execution occasions with pipeline and buyer metadata, and routing processed knowledge to downstream programs akin to ClickHouse for analytics, PostgreSQL for operational workloads, and Amazon Easy Storage Service (Amazon S3) for long-term archival.
Reliability and fault tolerance
Amazon MSK’s three-replica configuration ensures that no single dealer failure may cause knowledge loss or ingestion interruption. Mixed with versatile knowledge retention, the structure supplies a significant replay window. If a downstream shopper (Amazon Managed Service for Apache Flink, ClickHouse, or one other service) experiences an outage, it may resume processing from its final dedicated offset with out knowledge loss.
In the course of the migration to the present structure, Buildkite employed a dual-write technique: concurrently writing to each the prevailing PostgreSQL pipeline and the brand new Amazon MSK/ClickHouse path. This strategy allowed the workforce to validate knowledge consistency and step by step shift site visitors with out risking customer-facing disruption. This sample speaks to the operational maturity Amazon MSK supplies.
Operational effectivity good points
The shift to a streaming-first structure, mixed with the downstream simplification of the analytics engine, produced important operational enhancements:
- Flink workloads lowered by 60%+: Eliminating pre-aggregation jobs that ran repeatedly no matter demand.
- Key/worth retailer fully retired: Amazon MSK’s buffering functionality, mixed with ClickHouse’s question efficiency, eradicated the necessity for a separate fast-lookup retailer.
- PostgreSQL capability reduce in half: 9 separate database clusters consolidated and right-sized.
- 1000’s of strains of utility code deleted: Less complicated structure means much less ETL code, fewer failure modes, and sooner onboarding for brand spanking new engineers.
Platform efficiency at a look
| Metric | Worth |
| Month-to-month check executions (for check engine platform) | 50 billion (4x progress from 3B) |
| Sustained peak ingestion | 500K occasions/second |
| Whole data in analytics retailer | 200 billion |
| Log ingestion requests | 70,000+ per second |
| Peak webhook throughput | 1.7 GB/second |
| MSK inbound throughput vary | 5 MB/sec – 100 MB/sec |
Enterprise and developer impression
The technical structure in the end exists to serve one function: serving to builders ship higher software program sooner. The streaming-first structure constructed on Amazon MSK and Amazon Managed Service for Apache Flink delivers on that promise throughout 4 dimensions.
On-demand analytics changed pre-computed stories. Prospects can now interactively slice and cube 70 billion data throughout arbitrary metadata dimensions. They get solutions to queries like “Present me P50 check durations by occasion kind and structure for the final 30 days” in seconds, not hours. Actual-time log streaming via the “stay tail” function means builders not watch for a construct to finish earlier than diagnosing failures. At 25,000 occasions per second, this expertise scales throughout hundreds of concurrent enterprise pipelines with out degradation.
Smarter check intelligence comes from Amazon Managed Service for Apache Flink’s stateful flaky check detection: when a check begins exhibiting intermittent failure patterns, Amazon Managed Service for Apache Flink identifies it because it occurs, not after the very fact. That is what separates a proactive analytics platform from a reactive one. It requires publishing knowledge to Kafka, processing with Flink, and letting ClickHouse deal with the advanced learn requests.
Conclusion: Streaming as a strategic basis
Buildkite’s journey from a Rails/Postgres monolith to a streaming-first analytics platform displays a sample more and more frequent amongst enterprise SaaS corporations: a dependable, high-throughput streaming and processing layer just isn’t an optimization. It’s a prerequisite for working at scale.
Amazon MSK and Amazon Managed Service for Apache Flink type the spine that helps Buildkite ingest 50 billion check executions per thirty days, serve real-time interactive analytics to enterprise clients, and accomplish that at decrease value than the extra advanced structure it changed. Amazon MSK handles sturdy, elastic occasion buffering. Amazon Managed Service for Apache Flink transforms uncooked streams into enriched, queryable knowledge. Collectively they take up the operational complexity that will in any other case eat engineering capability.
For platform engineers evaluating streaming infrastructure for multi-tenant SaaS workloads, the sign is obvious: put money into the streaming spine early, and let managed companies deal with the operational complexity.
To be taught extra about Amazon MSK and Amazon Managed Service for Apache Flink, go to aws.amazon.com/msk and aws.amazon.com/managed-service-apache-flink.
Concerning the authors
