eventSourceMapping kinesis->lambda

Serverless Framework
1

From the docs it should be possible to deploy an eventSourceMapping from kinesis to a lambda function.

this is my definition:

functions:
  myFunction:
    handler: org.example.MyJavaClass

resources:
  Resources:
    myKinesisStream:
      Type: AWS::Kinesis::Stream
      Properties:
        Name: myStreamInput
        ShardCount: 1
    myKinesisLambdaMapping:
      Type: AWS::Lambda::EventSourceMapping
      Properties:
        EventSourceArn:
          Fn::GetAtt:
            - "myKinesisStream"
            - "Arn"
        FunctionName:
          Fn::GetAtt:
            - "myFunction"
            - "Arn"
        StartingPosition: "TRIM_HORIZON"
        BatchSize: 10

the lambda function itself is ok and creates, the kinesis steam as well. just when I add the mapping it doesn’t work, as if the definition of the lambda function wasn’t known to the mapping factory, this error message is printed out:

# sls deploy
Serverless: Creating Stack...

  Validation Error ---------------------------------------

     Template error: instance of Fn::GetAtt references undefined
     resource myFunction

     For debugging logs, run again after setting SLS_DEBUG env var.

  Stack Trace --------------------------------------------

ValidationError: Template error: instance of Fn::GetAtt references undefined resource myFunction
    at Request.extractError (/home/oso/serverless/node_modules/aws-sdk/lib/protocol/query.js:40:29)
    at Request.callListeners (/home/oso/serverless/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
    at Request.emit (/home/oso/serverless/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/home/oso/serverless/node_modules/aws-sdk/lib/request.js:615:14)
    at Request.transition (/home/oso/serverless/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/home/oso/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /home/oso/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/home/oso/serverless/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/home/oso/serverless/node_modules/aws-sdk/lib/request.js:617:12)
    at Request.callListeners (/home/oso/serverless/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
    at Request.emit (/home/oso/serverless/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/home/oso/serverless/node_modules/aws-sdk/lib/request.js:615:14)
    at Request.transition (/home/oso/serverless/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/home/oso/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /home/oso/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/home/oso/serverless/node_modules/aws-sdk/lib/request.js:38:9)

I am pretty sure that I am missing just a small detail. Someone able to spot it?

The definition of the FunctionName seems to be the problematic part. AFAIK, the definition is created according to the documentation: SLS about event source mapping and AWS about event source mapping

2

This can probably be fixed with a DependsOn setting: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-dependson.html

Otherwise the kinesis stream or the lambda function might not have been created, thus the mapping doesn’t work. Please also check the function logical name as it isn’t necessarily just myFunction

1 Like
3

Can you please show an example configuration that works? I am trying to accomplish the exact same scenario and am stuck on the function name too – I also tried to use the full name of the lambda function as it is deployed on AWS but still no luck.

4

OK I figured it out…there needs to be DependsOn and IAM role

EDIT: I was trying to setup another stream, which was failing on permissions for an indiscernible reason. It appears to be because I needed to include IamPolicyLambda in DependsOn

service: kinesis
provider:
  name: aws
  runtime: nodejs4.3
  iamRoleStatements:
      -  Effect: "Allow"
         Action:
           -  "kinesis:DescribeStream"
           -  "kinesis:GetRecords"
           -  "kinesis:GetShardIterator"
           -  "kinesis:ListStreams"
           -  "logs:CreateLogGroup"
           -  "logs:CreateLogStream"
           -  "logs:PutLogEvents"
         Resource: "*"

functions:
  hello:
    handler: handler.hello

resources:
  Resources:
    KinesisStream:
      Type: AWS::Kinesis::Stream
      Properties:
        Name: another-test-stream
        ShardCount: 1
    KinesisTestMap:
      Type: AWS::Lambda::EventSourceMapping
      DependsOn : 
        - "hello"
        - "IamPolicyLambda
      Properties:
        BatchSize: 10
        EventSourceArn:
          Fn::GetAtt:
            - "KinesisStream"
            - "Arn"
        FunctionName:
          Fn::GetAtt:
            - "hello"
            - "Arn"
        StartingPosition: "TRIM_HORIZON"
1 Like
5

This solution does not work for me. Still get the same error as @mikkelspring. Any suggestions?

6

Hey, @yeehaa123 sorry didn’t see this because I left the job I was at. It is still working for me: but I have noticed that in general things can fail until you have created the function separately. I tend to comment out all my resources, deploy and then uncomment them/deploy again.

Also, the variables have changed with Serverless RC1, so now I have

service: kinesis-test

provider:
  name: aws
  runtime: nodejs4.3
  stage: dev
  iamRoleStatements:
    -  Effect: "Allow"
       Action:
         -  "kinesis:DescribeStream"
         -  "kinesis:GetRecords"
         -  "kinesis:GetShardIterator"
         -  "kinesis:ListStreams"
         -  "logs:CreateLogGroup"
         -  "logs:CreateLogStream"
         -  "logs:PutLogEvents"
       Resource: "*"


custom:
  stage: ${opt:stage, self:provider.stage}

functions:
  test:
    handler: index.test
    memorySize: 128

resources:
  Resources:
    KinesisStream:
      Type: AWS::Kinesis::Stream
      Properties:
        Name: test-stream
        ShardCount: 1
    KinesisTestMap:
      Type: AWS::Lambda::EventSourceMapping
      DependsOn : 
        - "TestLambdaFunction"
        - "IamPolicyLambdaExecution"
      Properties:
        BatchSize: 10
        EventSourceArn:
          Fn::GetAtt:
            - "KinesisStream"
            - "Arn"
        FunctionName:
          Fn::GetAtt:
            - "TestLambdaFunction"
            - "Arn"
        StartingPosition: "TRIM_HORIZON"
7

@yeeha123 If you are deploying with v1.0.0-rc1 or later it might be because the way the lambda functions are named in the auto-generated CloudFormation file have changed, so instead of every instance of hello in the resources section posted by @mikkelspring, you would need to replace it with HelloLambdaFunction as this is the name that Serverless gives the lambda hello in CloudFormation.

Cheers!

1 Like