Lambda triggered by MSK (Kafka) from another AWS account

Serverless Architectures
1

I have an ultimate goal of making Lambda triggered from Kafka topics while Kafka is an MSK cluster running on another AWS account. I started with SLS framework, but ended up trying things directly in AWS, so I guess it is more of a conceptual problem.

Setup
Lambda and MSK cluster live on different AWS accounts, i.e. A_L (lambda) and A_K (kafka or MSK cluster). Each is connected to its own VPC, i.e. VPC_L and VPC_K, more specifically to private subnet SN_L and SN_K that has connectivity to the Internet via publicly connected NAT gateway.

Immediate problem
I cannot use “MSK trigger” for lambda because it requires specification of the ARN for MSK cluster but that is not possible since that MSK cluster is living on another account and cannot be referenced in the context of lambda trigger.

Problem that I’m trying to solve
I am trying to use “Kafka trigger” which needs specification of bootstrap server (I have it), topic name (I have it), batch size, and starting position (not a problem). The problem is with the second group of options that allows for authentication of lambda trigger in front of MSK cluster. It can either be a 1) network-based setup in form of a combination of VPC/Subnet/SecurityGroup of MSK cluster, 2) secret-based setup in form of SASL/xxx configuration.

The former option, i.e. network-based setup, cannot be used because it requires VPC and other params from Kafka cluster, i.e. VPC_K, SN_K (subnet), and SG_K (security group) that are not available on the account where Lambda is being configured.

The latter requires specification of one of the SASL methods, i.e. PLAIN, SCRAM512, and SCRAM256 paired with a reference to SecretManager record storing a combination of username/password. I chose this method as the only theoretically possible for cross-account communication.

What I did

  1. On Lambda account I created a record in SecretManager with 2 keys: username and password and some specific values
  2. On Kafka account I enabled SASL/SCRAM authentication for MSK cluster. It required setting up a record in SecretManager on Kafka account which I did. I used the same credentials as in the secret on Lambda account. After that I can see 3 options for bootstrap server: <hostname>:9092 (plain), <hostname>:9094 (TCL), and <hostname>:9096 (SASL/SCRAM).
  3. I used bootstrap server specs (the one with 9096 port) in kafka trigger setup for my lambda function.

Outcomes
No matter what I try in terms of different combination of bootstrap server and SASL method I am getting this error on Lambda trigger side: PROBLEM: Connection error. Please check your event source connection configuration

I cannot find any way of debugging this error scenario as no other details are provided. I tried enabling CloudWatch on MSK cluster but it didn’t collect any helpful info.

Tags: SASL/SCRAM, cross-account, Kafka, MSK, AWS, Lambda

2

MSK is just Kafka. You would just use the module/driver of your choice for the language of your choice and publish messages to Kafka directly.
9Apps Showbox Tutuapp