Amazon EMR on EKS now supports Apache Spark troubleshooting agent
AWS adds an AI-powered troubleshooting agent for Apache Spark jobs running on Amazon EMR on EKS, enabling automated root-cause analysis and code fixes via natural language.
Amazon Web Services has introduced a troubleshooting agent for Apache Spark workloads running on Amazon EMR on EKS, allowing data engineers to diagnose job failures using natural language queries. The agent automates root-cause analysis and provides PySpark code recommendations without requiring manual inspection of distributed logs or the Spark History Server. This reduces the time engineers spend troubleshooting failures in Kubernetes-based Spark deployments.
The agent examines Spark History Server data, executor logs, and cluster configurations to identify common issues such as memory errors, data skew, resource contention, and connectivity failures. It now supports all EMR deployment models, including EMR on EC2, EMR Serverless, and EMR on EKS. Users can access the agent directly from the EMR on EKS console via a 'Troubleshoot with AI' option on failed jobs, streamlining the debugging process.
In addition to console access, the agent is available through the Model Context Protocol (MCP), enabling integration with compatible AI coding agents like Kiro, Claude Code, and Cursor. All interactions are read-only, authenticated via IAM roles, and logged in AWS CloudTrail for auditability and security compliance.
The Spark troubleshooting agent is available in all AWS Regions where SageMaker Unified Studio is supported. To begin using the agent, users can access it through the EMR on EKS console or deploy the MCP server in their preferred AI coding environment. Detailed setup instructions are provided in the EMR troubleshooting agent documentation.