DynamoDB streams creation

Serverless Framework
1

Hi,
How can I create a dynamoDB stream and attach a lambda function to it?
In the documentation (https://serverless.com/framework/docs/providers/aws/events/streams/#dynamodb--kinesis-streams) it is said that streams are not created.
Also I find confusing the fact that the first example (pasted below) has to “stream” fields and I don’t understand what does it mean.

functions:
  compute:
    handler: handler.compute
    events:
      - stream: arn:aws:dynamodb:region:XXXXXX:table/foo/stream/1901T00:00:00.000
      - stream:
          arn:
            Fn::GetAtt:
              - MyKinesisStream
              - Arn
1 Like
2

First make sure your resource includes a StreamSpecifcation so that is setup up the stream

resources:
  Resources:
    MyTable:
      Type: AWS::DynamoDB::Table
      Properties:
        TableName: my_table
        AttributeDefinitions:
          - AttributeName: id
            AttributeType: S
        KeySchema:
          - AttributeName: id
            KeyType: HASH
        ProvisionedThroughput:
          ReadCapacityUnits: 5
          WriteCapacityUnits: 5
        StreamSpecification:
          StreamViewType: NEW_AND_OLD_IMAGES

See http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_StreamSpecification.html for more information about values.

You also need to make sure your iamRoleStatement has the correct permissions.

provider:
  iamRoleStatements:
    - Effect: Allow
      Action:
        - dynamodb:DescribeStream
        - dynamodb:GetRecords
        - dynamodb:GetShardIterator
        - dynamodb:ListStreams
      Resource: arn:aws:dynamodb:*:*:table/my_table

I also have dynamodb:GetItem and dynamodb:PutItem because my Lambda needs to put and get items so you may or may not need them too.

Deploy now so that it creates the stream then go to the AWS console and get the Stream ARN for your table.

Finally configure your function with a stream event using the Stream ARN from the AWS console.

  streamFunction:
    handler: streamFunction.handler
    events:
      - stream: arn:aws:dynamodb:REGION:ACCOUNT_ID:table/my_table/stream/2016-11-20T22:55:25.566

Deploy again and your Lambda function should be getting called when records change.

The Serverless manual includes a new format:

- stream:
    arn:
      Fn::GetAtt:
        - MyTable
        - Arn

In Serverless 1.1 there is a confirmed bug that prevents this from working. The recommended solution is to use the arn format.

10 Likes
3

Perfect explanation @buggy! Thank you so much! Was struggling with this. The docs are not so helpful in this specific topic.

1 Like
4

I also worked around the known bug and defined the cloudformation mapping directly in the serverless.yml:

StreamFunctionEventSourceMappingDynamodbMyTable:
  Type: AWS::Lambda::EventSourceMapping
  DependsOn: IamPolicyLambdaExecution
  Properties:
    BatchSize: 10
    EventSourceArn:
      Fn::GetAtt:
        - MyTable
        - StreamArn
    FunctionName:
      Fn::GetAtt:
        - streamFunctionLambdaFunction
        - Arn
    StartingPosition: TRIM_HORIZON
    Enabled: True

This workaround has the advantage that you don’t have to know the ARN of the lambda function.

4 Likes
5

Hi @chgerkens! Can you please explain this? Where are you placing that YML definition? In the Resources section? Thanks in advance.

6

Right, under

 resources:
      Resources:

For instance below the DynamoDB table resource that includes the StreamSpecification.

8

Can you share the full serverless.yml file? It’s not clear from the documentation how to accomplish this.

1 Like
9

https://github.com/serverless/serverless/pull/3111 contains a complete and working serverless.yml example from @pmuens. I cannot deep-link to the post, so search for “FYI here’s the serverless.yml I’ve used (this time with the correct indentation).” on the pull request page I mentioned.

2 Likes
10

I also have a working DynamoDB Streams example as part of my GraphQL API example.
You can find it here: https://github.com/boazdejong/serverless-graphql-api#dynamodb-streams

5 Likes
11

Does anyone know how to get this working with serverless-offline? The above example @buggy posted doesn’t appear to work (at least for me it doesn’t)

12

@tgfischer The example is for a real DynamoDB. I don’t think serverless-offline supports DynamoDB streams.

14

beautiful. Worked for me.

16

I searched the whole document for this. I couldn’t find anything about creating stream using serverless. How did you get this solution?

17

This property definition:

FunctionName:
      Fn::GetAtt:
        - streamFunctionLambdaFunction
        - Arn

doesn’t really work. How can I reference the lambda function that I want the dynamo stream to trigger on update using the lambda function’s name?

18

There is no need for the Event Source Map you can just define all of the mappings required in the Lambda declaration itself