The migration to Apache Flink 2.2 on Amazon Managed Service introduces significant enhancements, including support for Java 17, improved checkpointing and recovery with RocksDB 8.10.0, and SQL-native AI/ML inference capabilities. Users currently on Flink 1.x may face challenges with the outdated Java 11 runtime and slower performance, making this upgrade essential.
Apache Flink serves as an open-source distributed processing engine for both stream and batch data, offering robust support for stateful processing and event-time semantics. The Amazon Managed Service simplifies the operational complexities of running Flink, allowing users to focus on their application code while the service manages infrastructure tasks.
Key Changes in Flink 2.2
Before initiating an upgrade, it's crucial to understand that Flink 2.2 has deprecated the DataSet API, discontinued Java 11 support, and replaced legacy connector interfaces. Users are advised to consult the Upgrading to Flink 2.2: Complete Guide and the State Compatibility Guide for Flink 2.2 Upgrades prior to upgrading production applications.
Upgrade Process Overview
To upgrade a running Flink 1.x application to 2.2, users can utilize the UpdateApplication API, AWS Management Console, AWS CloudFormation, AWS SDK, or Terraform Modules. The upgrade process retains application configurations, logs, metrics, and tags, provided that state and binaries are compatible.
Automatic Rollback Feature
With the auto-rollback feature enabled, any detected binary incompatibilities during job startup will trigger an automatic revert to the previous Flink version. Should state incompatibilities arise, users can invoke the Rollback API to restore the previous version and state.
Migration Steps
Before starting the migration, ensure the following:
- Update Flink dependencies to version 2.2.0.
- Replace deprecated APIs.
- Assess application state compatibility with Flink 2.2.
Specific steps include:
Update Dependencies
Modify your pom.xml file and replace legacy Kinesis connectors. The FlinkKinesisConsumer and FlinkKinesisProducer classes are no longer available in Flink 2.2.
Connector Updates
Ensure that AWS connectors are updated to Flink 2.x-compatible versions. Some connectors, like JDBC, OpenSearch, and Prometheus, may not yet have compatible releases.
Compatibility Checks
Check application logs for Kryo serialization fallbacks, indicating potential incompatibilities. Enable automatic rollback and verify that automatic snapshots are active before proceeding.
Executing the Upgrade
Run the upgrade while the application is in a RUNNING or READY state. The upgrade can be executed through:
- AWS CLI
- AWS Management Console
- AWS CloudFormation
- Terraform
After the upgrade, monitor the application status to ensure it transitions correctly. Key CloudWatch metrics should be observed to confirm operational health.
Rollback Procedures
If the application experiences issues post-upgrade, users can initiate a rollback via the Rollback API, reverting to the previous version and application code from the snapshot taken prior to the upgrade.
Conclusion
The upgrade to Apache Flink 2.2 on Amazon Managed Service offers a modern runtime environment and enhanced performance features. By following the outlined steps and conducting thorough testing, users can successfully transition to the latest version and take advantage of its capabilities.
