Amazon Managed Streaming for Apache Kafka (Amazon MSK) is increasingly adopting AWS Identity and Access Management (IAM) authentication, allowing for streamlined access control to Kafka resources. This guide focuses on configuring a custom domain name for Amazon MSK clusters utilizing IAM authentication.
Understanding the Architecture
To implement a custom domain name, the Kafka broker must present a server certificate that corresponds with this domain. The solution employs Network Load Balancers (NLBs) alongside Amazon Certificate Manager to manage the custom certificate for the MSK brokers. Additionally, an Amazon Route 53 Private Hosted Zone is used for DNS management of the custom domain.
Certificate Management
For TLS communication between clients and the MSK cluster, a certificate from AWS Certificate Manager (ACM) signed by a Private Certificate Authority (PCA) is necessary. This certificate must have the custom domain name as the Common Name (CN) and include Subject Alternative Names (SANs) for all broker DNS names.
Network Load Balancer Configuration
The NLB facilitates a TLS listener, enabling secure communication between clients and the NLB, which then negotiates TLS with the MSK brokers. This architecture also supports AWS PrivateLink, allowing secure access to the MSK cluster from external VPCs.
DNS Resolution
To resolve DNS queries for the custom domain, an Amazon Route 53 private hosted zone is utilized. This setup ensures that clients can resolve DNS from the Route 53 VPC resolver. For publicly accessible MSK clusters, a public NLB and Route 53 public hosted zone can be implemented.
Broker Configuration
Each broker's advertised listeners configuration must be updated to reflect the custom domain name and NLB ports. The advertised listeners setting is crucial for Kafka clients to connect to the brokers correctly, and each broker requires a unique DNS name.
Deployment Steps
- Use the provided CloudFormation template to deploy necessary resources including VPC, NLB, PCA, ACM certificate, and MSK cluster.
- Run the included script to update the advertised listeners with the custom domain name.
- Delete the CloudFormation stack to stop incurring costs when the setup is no longer needed.
Common Questions
- Can Amazon MSK Replicator be used with custom domain names? Yes, it supports both active-passive and active-active setups.
- Is it possible to use a global bootstrap DNS name for multi-region clusters? No, the correct AWS Region must be specified for IAM authentication.
- What if a broker fails? Amazon MSK automatically replaces the failed broker without changing the advertised listener configuration.
This guide outlines the steps to utilize an NLB, Route 53, and the advertised listener configuration to support custom domain names for Amazon MSK clusters when using IAM authentication. This approach allows for maintaining existing Kafka bootstrap DNS names while adhering to organizational naming conventions.
