[Solved] Using "exists: true" for S3 bucket event reports deploy failure because bucket already exists

Serverless Framework
1

edit: So as embarrassed as I am to admit this, it turns out that the only thing I was doing wrong was using a version of serverless that was too old to support the (relatively new) existing: true option. I was running 1.45.x, and apparently this feature didn’t make it in until 1.47.x or 1.48.x. Either way, upgrading serverless to use the latest version solved my issue!

Hello! I’m trying to follow some better practices (best practices might be debatable!) when it comes to splitting up my serverless projects, so when I want to redeploy lambda functions / api gateway configurations I have fewer tendrils with things like S3 and DynamoDB (basically so data that’s persisted and can’t be safely deleted is in a different service). I’ve been following https://serverless-stack.com/chapters/s3-as-a-serverless-service.html and it’s been super useful so far. The basic goal is to be able create an event configuration in one service that listens for events on a bucket setup in a different service.

The problem is when I deploy the service with the event / lambda / function configuration, I get an error:

An error occurred: S3BucketTeststoreprojectorderimportdev - test-store-test-order-import-dev already exists in stack arn:aws:cloudformation:us-west-2:558299803503:stack/test-store-s3-DEV/b3a3c580-a9ff-11e9-a091-0ae846f1e916.

This is saying that test-store-test-order-import-dev already exists, which is true, it does, but as you can see (in the truncated yml) below I am using the existing: true option with the understanding that it shouldn’t try to re-create the bucket:

#first serverless.yml with my lambda functions / events...

provider:
  name: aws
  ...
  ...
  ...   
  iamRoleStatements:
    - Effect: Allow
      Action:
        - 's3:*'      
      Resource:
        - 'Fn::ImportValue': ${self:custom.alias}-ProjectOrderProcessingBucketArn    
        
         
functions:
  test-order-import:    
    name: store-test-order-import
    handler: src/functions/project/test-order-import.handle
    environment:
        PROJECT_STORE_CODE: Test
    events:      
      - s3:
          bucket: orders-${self:custom.lower-case-suffix.${opt:alias,'DEV'}}
          event: s3:ObjectCreated:*
          rules:
            - prefix: import/
            - suffix: .csv
          existing: true

In my (much simpler) s3 / bucket serverless service project, the relevant bits are here:

#second serverless.yml with config for the S3 bucket(s)

resources:    
  Resources:
    ProjectOrderProcessingBucket:
      Type: AWS::S3::Bucket
      DeletionPolicy: Retain
      Properties:
          BucketName: ${self:custom.bucket-name-test-order-import}${self:custom.lower-case-suffix.${opt:alias,'DEV'}}

  Outputs:
    ProjectOrderProcessingBucketArn:
      Value:
         Fn::GetAtt:
          - ProjectOrderProcessingBucket
          - Arn
      Export:
        Name: ${self:custom.alias}-ProjectOrderProcessingBucketArn

    ProjectOrderProcessingBucketName:
      Value:
        Ref: ProjectOrderProcessingBucket
      Export:
        Name: ${self:custom.alias}-ProjectOrderProcessingBucket

Now it may be worth noting that I get this same error regardless of whether or not I include the iamRoleStatement:

  iamRoleStatements:
    - Effect: Allow
      Action:
        - 's3:*'      
      Resource:
        - 'Fn::ImportValue': ${self:custom.alias}-ProjectOrderProcessingBucketArn

Now besides this, I’m not doing anything too “crazy”, my service configuration is pretty simple actually.

I’d greatly appreciate any thoughts / input / ideas or even steps I can take to debug this. Maybe this is a bug? Looking at https://serverless.com/framework/docs/providers/aws/events/s3#using-existing-buckets it seems like this should “just work” but clearly I’m doing something wrong (or, well, there is a bug!)

2

So as embarrassed as I am to admit this, it turns out that the only thing I was doing wrong was using a version of serverless that was too old to support the (relatively new) existing: true option.

I was running 1.45.x, and apparently this feature didn’t make it in until 1.47.x or 1.48.x. Either way, upgrading serverless to use the latest version solved my issue!

1 Like
3

Hi Kevin,

I am doing the similar thing of adding a bucket and adding event trigger in lambda on object creation. The initial deployment works fine and it add the associated policies roles, trigger to lambda and addition of new s3 bucket.

When I redeploy add existing: true for same bucket which was created earlier. It gives me error like “Failed to create resource. Configuration is ambiguously defined. Cannot have overlapping suffixes in two rules if the prefixes are overlapping for the same event type. See details in CloudWatch Log”

I know that we cant have same trigger event defined in s3 for same lambda however it should overwrite it or skip doing the same.

Can you suggest what I am doing wrong here?

Here is the sample to the same.

# Lambda functions
functions:
  ExperimentPlayVer04:
    name : experiment-play-ver04 # actual name in AWS
    handler: lambdaMain.lambda_handler
    role: ExperimentPlayLambdaRoleVer04
    events:
      - s3:
          bucket: ${self:custom.bucketRef.${self:provider.stage}}
          event: s3:ObjectCreated:*
          rules:
            - prefix: message_center-export/monitoring/
            - suffix: .csv

# resources
resources:
  Resources:
    ExperimentPlayLambdaCloudwatchVer03:
      Properties:
        ManagedPolicyName: experiment-play-ver04-lambda-cloudwatch
        PolicyDocument:
          Statement:
            - Action:
                - logs:CreateLogStream
                - logs:PutLogEvents
              Effect: Allow
              Resource:
                - arn:aws:logs:*:*:log-group:*
            - Action:
                - logs:PutLogEvents
              Effect: Allow
              Resource:
                - arn:aws:logs:*:*:log-group:*:*:*
            - Action:
                - logs:CreateLogGroup
                - cloudwatch:PutMetricData
              Effect: Allow
              Resource:
                - '*'
          Version: '2012-10-17'
      Type: AWS::IAM::ManagedPolicy
    ExperimentPlayS3AccessVer04:
      Properties:
        ManagedPolicyName: experiment-play-ver04-s3-access
        PolicyDocument:
          Statement:
            - Action:
                - s3:PutAccountPublicAccessBlock
                - s3:GetAccountPublicAccessBlock
                - s3:ListAllMyBuckets
                - s3:ListJobs
                - s3:CreateJob
                - s3:HeadBucket
                - s3:GetObject
                - s3:GetObjectAcl
                - s3:PutObject
              Effect: Allow
              Resource:
                - '*'
            - Action:
                - s3:*
              Effect: Allow
              Resource:
                - { "Fn::Join" : ["", ["arn:aws:s3:::", { "Ref" : "ServerlessDeploymentBucket" }]]}
                - { "Fn::Join" : ["", ["arn:aws:s3:::${self:custom.bucketRef.${self:provider.stage}}"]]}
          Version: '2012-10-17'
      Type: AWS::IAM::ManagedPolicy
    ExperimentPlayLambdaRoleVer04:
          Type: AWS::IAM::Role
          Properties:
            Path: /service-role/
            RoleName: experimentPlayLambdaRoleVer04
            AssumeRolePolicyDocument:
              Statement:
                - Effect: Allow
                  Principal:
                    Service:
                      - lambda.amazonaws.com
                      - events.amazonaws.com
                  Action: sts:AssumeRole
            ManagedPolicyArns:
              - 'Fn::Join':
                - ':'
                -
                  - 'arn:aws:iam:'
                  - Ref: 'AWS::AccountId'
                  - 'policy/experiment-play-ver04-lambda-cloudwatch'
              - 'Fn::Join':
                - ':'
                -
                  - 'arn:aws:iam:'
                  - Ref: 'AWS::AccountId'
                  - 'policy/experiment-play-ver04-s3-access'