AWS Glue announces automatic version upgrades for Apache Spark jobs

AWS Glue has introduced automatic version upgrades for Apache Spark jobs, eliminating the need for manual coordination to keep runtimes current. This feature ensures jobs receive the latest security patches, performance enhancements, and support for evolving table formats like Apache Iceberg. The process includes compatibility checks, user consent, and validation against established output profiles to maintain stability. Jobs previously requiring weeks of planning can now upgrade automatically by setting the version to "Auto."

Platform teams benefit by standardizing runtime versions across large job fleets without tracking individual owners, while business units without dedicated upgrade teams can maintain current jobs effortlessly. Each job undergoes validation against its output schema, data volume, latency, and format before upgrading, ensuring reliability. Jobs requiring code changes remain on their current version until fixes are applied.

AWS Glue provides an upgrade analysis with specific fixes for jobs needing adjustments, which can be applied directly or via the Apache Spark upgrade agent. This analysis helps teams address compatibility issues proactively, reducing downtime and manual effort. The feature is designed to minimize disruption while ensuring jobs stay current with minimal oversight.

Automatic upgrades are available in all AWS Commercial Regions and can be enabled by selecting "Auto" as the version when creating or updating jobs through AWS Glue Studio, AWS CLI, API, or SDKs. Alternatively, the AWS Glue skill allows bulk updates for multiple jobs, simplifying management across large environments.