AWS has announced the introduction of Spark Connect on Amazon EMR Serverless, starting with EMR release 7.13 (Apache Spark 3.5.6) and subsequent versions. This new feature enables developers to build and debug Spark applications directly from their preferred local environments while leveraging the full capabilities of Spark operations on EMR Serverless.
Previously, discrepancies between local and production environments often led to issues, requiring a cumbersome deploy-and-check cycle to identify problems. With Spark Connect, developers can use local tools such as IDEs (like VS Code or PyCharm), Jupyter notebooks, and Amazon SageMaker Unified Studio Data Notebooks to develop Spark code without the need for cluster provisioning or code repackaging. This streamlines the development process by allowing local Python sessions to remain local while Spark operations are executed remotely.
Key Features of Spark Connect
- Session Management: Each Spark Connect session is assigned a unique AWS resource with its own ARN, allowing for tailored AWS IAM permissions and cost allocation.
- Real-Time Monitoring: Developers can monitor their sessions through the Spark UI, which provides persistent session history and management capabilities.
- Client-Server Architecture: The lightweight PySpark library on the client sends operations to a Spark Connect server on EMR Serverless, which executes the code and returns results.
- Cost Efficiency: Users only pay for compute resources while their sessions are active, with automatic scaling based on workload demands.
Getting Started with Spark Connect
To begin using Spark Connect, developers need to follow these steps:
- Create an EMR Serverless application.
- Start a Spark Connect session.
- Connect from a local IDE or notebook.
During setup, an IAM role must be provided to grant the session access to necessary data sources, such as Amazon S3. The session endpoint includes a secure URL and an authentication token, ensuring encrypted communication.
Development Flexibility
Spark Connect supports a variety of development workflows, making it suitable for use in different environments, including IDEs and Jupyter notebooks. This flexibility allows teams to integrate Spark analytics into applications seamlessly, treating Spark as a database driver rather than a separate system.
Conclusion
With the launch of Spark Connect on EMR Serverless, AWS has bridged the gap between local development and production-scale execution. This feature allows developers to interactively build and debug PySpark applications while benefiting from the scalability and management capabilities of EMR Serverless. The service is available now with no additional charges beyond the standard EMR Serverless compute pricing.
