How Slack achieved operational excellence for Spark on Amazon EMR utilizing generative AI

At Slack, our knowledge platform processes terabytes of information every day utilizing Apache Spark on Amazon EMR on Amazon Elastic Compute Cloud (Amazon EC2), powering the insights that drive strategic decision-making throughout the group.

As our knowledge quantity expanded, so did our efficiency challenges. With conventional monitoring instruments, we couldn’t successfully handle our techniques when Spark jobs slowed down or prices spiraled uncontrolled. We had been caught looking out via cryptic logs, making educated guesses about useful resource allocation, and watching our engineering groups spend hours on handbook tuning that ought to have been automated. That’s why we constructed one thing higher: an in depth metrics framework designed particularly for Spark’s distinctive challenges. This can be a visibility system that offers us granular insights into software habits, useful resource utilization, and job-level efficiency patterns we by no means had earlier than. We’ve achieved 30–50% price reductions and 40–60% sooner job completion instances. That is actual operational effectivity that straight interprets to higher service for our customers and vital financial savings for our infrastructure funds. On this submit, we stroll you thru precisely how we constructed this framework, the important thing metrics that made the distinction, and the way your workforce can implement comparable monitoring to remodel your individual Spark operations.

Why complete Spark monitoring issues

In enterprise environments, poorly optimized Spark jobs can waste 1000’s of {dollars} in cloud compute prices, block crucial knowledge pipelines affecting downstream enterprise processes, create cascading failures throughout interconnected knowledge workflows, and impression service stage settlement (SLA) compliance for time-sensitive analytics.

The monitoring framework we’re analyzing captures over 40 distinct metrics throughout 5 key classes, offering the granular insights wanted to forestall these points.

How we ingest, course of, and act on Spark metrics

To handle the challenges of managing Spark at scale, we developed a customized monitoring and optimization pipeline—from metric assortment to AI-assisted tuning. It begins with our in-house Spark listener framework, which captures over 40 metrics in actual time throughout Spark purposes, jobs, phases, and duties whereas pulling crucial operational context from instruments similar to Apache Airflow and Apache Hadoop YARN.

An Apache Airflow-orchestrated Spark SQL pipeline transforms this knowledge into actionable insights, surfacing efficiency bottlenecks and failure factors. To combine these metrics into the developer tuning workflow, we expose a metrics software and a customized immediate via our inside analytics mannequin context protocol (MCP) server. This permits seamless integration with AI-assisted coding instruments similar to Cursor or Claude Code.

The next is the checklist of instruments used for our Spark monitoring resolution, which incorporates metric assortment to AI-assisted tuning:

The result’s quick, dependable, deterministic Spark tuning with out the guesswork. Builders get environment-aware suggestions, automated configuration updates, and ready-to-review pull requests.

Deep dive into Spark metrics assortment

On the middle of our real-time monitoring resolution lies a customized Spark listener framework that captures thorough telemetry throughout the Spark lifecycle. Spark’s built-in metrics are sometimes coarse, quick‑lived, and scattered throughout the consumer interface (UI) and logs, which leaves 4 crucial gaps:

1. Constant historic document
2. Weak linkage from purposes to jobs to phases to duties
3. Restricted context (consumer, cluster, workforce)
4. Poor visibility into patterns similar to skew, spill, and retries

Our expanded listener framework closes these gaps by unifying and enriching telemetry with atmosphere and configuration tags, constructing a sturdy, queryable historical past, and correlating occasions throughout the execution graph. It explains why duties fail, pinpoints the place reminiscence or CPU stress happens, compares meant configurations to precise utilization, and produces clear, repeatable tuning suggestions so groups can baseline habits, reduce waste, and resolve points sooner. The next structure diagram illustrates the circulation of the Spark metrics assortment pipeline.

Spark listener

Our listener framework captures Spark metrics at 4 distinct ranges:

1. Utility metrics: Total software success/failure charges, whole runtime, and useful resource allocation
2. Job-level metrics: Particular person job period and standing monitoring inside an software
3. Stage-level metrics: Stage execution particulars, shuffle operations, and reminiscence utilization per stage
4. Activity-level metrics: Particular person activity efficiency for deep debugging eventualities

The next Scala instance code reveals the SparkTaskListener extends the category SparkListener to seize detailed task-level metrics:

class SparkTaskListener(conf: SparkConf) extends SparkListener {
 val taskToStageId = new mutable.HashMap[Long, Int]()
 val stageToJobID = new mutable.HashMap[Int, Int]()
 personal val emitter: Emitter = getEmitter(conf)
  override def onTaskStart(taskStart: SparkListenerTaskStart): Unit = {
   taskToStageId += taskStart.taskInfo.taskId -> taskStart.stageId 
 }
 override def onTaskEnd(taskEnd: SparkListenerTaskEnd): Unit = {
   val taskInfo = taskEnd.taskInfo
   val taskMetrics = taskEnd.taskMetrics
   val jobId = stageToJobID.apply(taskToStageId.apply(taskInfo.taskId))
   val metrics = Map[String, Any](
     "event_type" -> "task_metric",
     "job_id" -> jobId,
     "task_id" -> taskInfo.taskId,
     "period" -> taskInfo.period,
     "executor_run_time" -> taskMetrics.executorRunTime,
     "memory_bytes_spilled" -> taskMetrics.memoryBytesSpilled,
     "bytes_read" -> taskMetrics.inputMetrics.bytesRead,
     "records_read" -> taskMetrics.inputMetrics.recordsRead
     // further metrics.....
   )
   emitter.report(convertToJson(metrics))
 }
}

Actual-time streaming to Kafka

These metrics are streamed in actual time to Kafka as JSON-formatted telemetry utilizing a versatile emitter system:

class KafkaEmitter(conf: SparkConf) extends Emitter {
     personal val dealer = conf.get("spark.customized.listener.kafkaBroker", "")
     personal val matter = conf.get("spark.customized.listener.kafkaTopic", "")
     personal var producer: Producer[String, Array[Byte]] = _
     override def report(str: String): Unit = {
         val message = str.getBytes(StandardCharsets.UTF_8)
         producer.ship(new ProducerRecord[String, Array[Byte]](matter, message))
     }
}

From Kafka, a downstream pipeline ingests these information into an Apache Iceberg desk.

Context-rich observability

Past commonplace Spark metrics, our framework captures important operational context:

• Airflow integration: DAG metadata, activity IDs, and execution timestamps
• Useful resource monitoring: Configurable executor metrics (heap utilization, execution reminiscence)
• Surroundings context: Cluster identification, consumer monitoring, and Spark configurations
• Failure evaluation: Detailed error messages and activity failure root causes

The mix of thorough metrics assortment and real-time streaming has redefined Spark monitoring at scale, laying the groundwork for highly effective insights.

Deep dive into Spark metrics processing

When uncooked metrics—typically containing thousands and thousands of information—are ingested from varied sources, a Spark SQL pipeline transforms this high-volume knowledge into actionable insights. It aggregates the information right into a single row per software ID, considerably decreasing complexity whereas preserving key efficiency indicators.

For consistency in how groups interpret and act on this knowledge, we apply the 5 Pillars of Spark Monitoring, a structured framework that turns uncooked telemetry into clear diagnostics and repeatable optimization methods, as proven within the following desk.

AI-powered Spark tuning

The next structure diagram illustrates the usage of agentic AI instruments to investigate the aggregated Spark metrics.

To combine these metrics right into a developer’s tuning workflow, we construct a customized Spark metrics software and a customized immediate that any agent can use. We use our current analytics service, a homegrown net software that customers can question our knowledge warehouse with, construct dashboards, and share insights. The backend is written in Python utilizing FastAPI, and we expose an MCP server from the identical service through the use of FastMCP. By exposing the Spark metrics software and customized immediate via the MCP server, we make it potential for builders to attach their most well-liked assisted coding instruments (Cursor, Claude Code, and extra) and use knowledge to information their tuning.

As a result of the information uncovered by the analytics MCP server is perhaps delicate, we use Amazon Bedrock in our Amazon Net Providers (AWS) account to offer the muse fashions to our MCP purchasers. This retains our knowledge safer and facilitates compliance as a result of it by no means leaves our AWS atmosphere.

Customized immediate

To create our customized immediate for AI-driven Spark tuning, we design a structured, rule-based format that encourages extra deterministic and standardized output. The immediate defines the required sections (software overview, present Spark configuration, job well being abstract, useful resource suggestions, and abstract) for consistency throughout analyses. We embrace detailed formatting guidelines, similar to wrapping values in backticks, avoiding line breaks, and implementing strict desk buildings to keep up readability and machine readability. The immediate additionally embeds specific steering for deciphering Spark metrics and mapping them to advisable tuning actions primarily based on finest practices, with clear standards for standing flags and impression explanations. The immediate implies that the AI’s suggestions will be traced, reproduced, and actioned primarily based on the supplied knowledge by tightly controlling the input-output circulation and trying to forestall hallucinations.

Remaining outcomes

The screenshots on this part present how our software carried out the evaluation and supplied suggestions. The next is a efficiency evaluation for an current software.

The next is a advice to cut back useful resource waste.

The impression

Our AI-powered framework has basically modified how Spark is monitored and managed at Slack. We’ve remodeled Spark tuning from a high-expertise, trial-and-error course of into an automatic, data-backed commonplace by transferring past conventional log-diving and embracing a structured, AI-driven method. The outcomes communicate for themselves, as proven within the following desk.

Conclusion

At Slack, our expertise with Spark monitoring reveals that you simply don’t should be a efficiency knowledgeable to attain distinctive outcomes. We’ve shifted from reacting to efficiency points to stopping them by systematically making use of 5 key metric classes.

The numbers communicate for themselves: 30–50% price reductions and 40–60% sooner job completion instances characterize operational effectivity that straight impacts our skill to serve thousands and thousands of customers worldwide. These enhancements compound over time as groups construct confidence of their knowledge infrastructure and might give attention to innovation moderately than troubleshooting.

Your group can obtain comparable outcomes. Begin with the fundamentals: implement complete monitoring, set up baseline metrics, and decide to steady optimization. Spark efficiency doesn’t require experience in each parameter, nevertheless it does require a powerful monitoring basis and a disciplined method to evaluation.

Acknowledgments

We need to give our because of all of the individuals who have contributed to this unimaginable journey: Johnny Cao, Nav Shergill, Yi Chen, Lakshmi Mohan, Apun Hiran, and Ricardo Bion.

Concerning the authors