UPDATE_FAILED: IamRoleLambdaExecution (Error Code: MalformedPolicyDocument)

Serverless Framework
,
1

Hello!

I have been receiving the error in the title of this post repeatedly, despite having scoured my serverless.yml file for hours, spending all night tweaking things, researching, tweaking again, consulting documentation, etc, etc. I cannot, for the life of me, figure out what I’ve got wrong and why this isn’t working.

The Stack

My stack is the photos backend for a mobile app I am developing. Initially, I had it set up to accept requests through an AWS HTTP 2.0 API Gateway and forward them to a one of three Lambda functions, one to return the result of a createPresignedPost request, one for deleting a single object, and one for deleting multiple objects. I also implemented a custom authorizer as a Lambda function, which receives a JSON web token generated when my users authenticate and passes it in each of the above three requests as an Authorization header, which is decoded using a secret stored in AWS secrets in order to authenticate the above CRUD-style requests. Users upload files using the createPresignedPost API call, which returns a URL and some headers, which are used to construct a POST request to that url with the image file attached, and, volia! the image is in my AWS bucket.

All of this was working wonderfully.

I then moved on to upgrade the system, first by adding an s3 event trigger on the aforementioned bucket, for the s3:ObjectCreated:* action, which runs a Lambda function that compresses uploaded images as JPGs. Initially, I followed the guidance to set up the event entirely under the the lambda function in my serverless.yml, and the first problem I ran into was I was getting the ‘bucket already exists’ error, because my event trigger was creating the bucket I had already specified in my resources section. First I marked it with existing: true, but this did nothing, and my assumption is this only works for buckets that exist already in the stack on AWS, not potentially in the serverless deployment. I had to remove the duplicate bucket from my resources section, and because I need additional configuration options, such as LifecycleConfiguration, CorsConfiguration, and PublicAccessBlockConfiguration, I followed guidance in the docs to move the s3 bucket and event trigger to the provider section of serverless.yml. I was still having problems and realized I had to define my s3 event trigger under the providers.s3 definition, as part of notificationConfiguration.LambdaConfigurations. This solved that error. Now I am stuck on the error in the title, something seems to be wrong with my iAmRoleStatements section, but I can’t figure out what it is.

The compression Lambda function gets files from the original bucket, sourceBucket, and after compressing them, places them into targetBucket. I created a resource for the targetBucket, but not the sourceBucket, because that is created in the definition under provider. I also have BucketPolicy resources for both buckets and I’ve given them minimum needed iAmRoleStatements permissions under provider. I also added a third bucket and corresponding BucketPolicy and iAMRoleStatement for a future publicBucket. Basically, users select or take photos in my app and they are immediately added to sourceBucket, then compressed and copied to targetBucket. Only after a user either finishes creating a profile or presses ‘Save Changes’ on their ‘edit profile’ page, will those photos in targetBucket be moved to publicBucket, using a batch action to limit API calls, where they can be consumed by the public in the app as profile photos. The photos in sourceBucket and targetBucket act as temporary files and have an ExpirationInDays LifecycleConfiguration rule that periodically cleans out any images left behind after users either successfully create/edit their profiles, or abandon the process.

serverless.yml

Below is my serverless.yml in full:

org: *redacted*
app: *redacted*
service: *redacted*

frameworkVersion: "3"

custom:
  region: us-east-1
  sourceBucketName: *redacted*
  targetBucketName: *redacted*
  publicBucketName: *redacted*
  jpgCompressionQuality: 90
  apiSecret: ${ssm:/aws/reference/secretsmanager/*redacted*}

provider:
  region: ${self:custom.region}
  name: aws
  runtime: nodejs14.x
  httpApi:
    cors: true
    authorizers:
      customAuthorizer:
        type: request
        functionName: tokenAuthorizer
        payloadVersion: '2.0'
        identitySource:
          - $request.header.Authorization
  s3:
    sourceBucket:
      name: ${self:custom.sourceBucketName}
      accessControl: Private
      lifecycleConfiguration:
        Rules:
          - Id: TempDeletionRule
            Status: Enabled
            ExpirationInDays: 7
      corsConfiguration:
        CorsRules:
        - AllowedMethods:
          - GET
          - POST
          AllowedOrigins:
          - "*"
          AllowedHeaders:
          - "*"
      publicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      notificationConfiguration:
        LambdaConfigurations:
          - Event: "s3:ObjectCreated:*"
            Function:
              "Fn::GetAtt":
                - CompressImagesHandlerLambdaFunction
                - Arn
  iamRoleStatements:
    - Effect: "Allow"
      Action:
        - "s3:GetObject"
        - "s3:GetObjectAcl"
        - "s3:PutObject"
        - "s3:PutObjectAcl"
        - "s3:DeleteObject"
        - "s3:PutAccelerateConfiguration"
      Resource:
        - arn:aws:s3:::${self:custom.sourceBucketName}
        - arn:aws:s3:::${self:custom.publicBucketName}
    - Effect: "Allow"
      Action:
        - "s3:GetObject"
        - "s3:ListBucket"
        - "s3:GetBucketLocation"
        - "s3:GetObjectVersion"
        - "s3:PutObject"
        - "s3:PutObjectAcl"
        - "s3:GetLifecycleConfiguration"
        - "s3:PutLifecycleConfiguration"
        - "s3:DeleteObject"
        - "s3:PutAccelerateConfiguration"
      Resource: arn:aws:s3:::${self:custom.targetBucketName}
    - Effect: "Allow"
      Action:
        - "lambda:InvokeFunction"
      Resource:
        - arn:aws:s3:::${self:custom.sourceBucketName}
    - Effect: "Allow"
      Action:
        - "ssm:GetParameter*"
        - "kms:Decrypt"
      Resource:
        - arn:aws:secretsmanager:${self:custom.region}:${AWS::AccountId}:secret:*
        - arn:aws:ssm:${self:custom.region}:${AWS::AccountId}:parameter/*

package:
  individually: true
  exclude:
    - ".serverless/**"
    - "*.*"

functions:
  tokenAuthorizer:
    handler: src/auth.handler
    environment:
      API_SECRET: ${self:custom.apiSecret}
  createPresignedPostHandler:
    handler: src/api.createPresignedPost
    events:
      - httpApi:
          method: POST
          path: /createUpload
          authorizer:
            name: customAuthorizer
            type: request
    environment:
      S3_BUCKET_NAME: ${self:custom.sourceBucketName}
  deleteObjectHandler:
    handler: src/api.deleteObject
    events:
      - httpApi:
          method: POST
          path: /deleteUpload
          authorizer:
            name: customAuthorizer
            type: request
    environment:
      S3_BUCKET_NAME: ${self:custom.sourceBucketName}
  deleteObjectsHandler:
    handler: src/api.deleteObjects
    events:
      - httpApi:
          method: POST
          path: /deleteUploads
          authorizer:
            name: customAuthorizer
            type: request
    environment:
      S3_BUCKET_NAME: ${self:custom.sourceBucketName}
  compressImagesHandler:
    handler: src/transform.compress
    events:
      - s3: sourceBucket
    environment:
      S3_TARGET_BUCKET_NAME: ${self:custom.targetBucketName}
      JPG_COMPRESSION_QUALITY: ${self:custom.jpgCompressionQuality}

resources:
  Resources:
    TargetBucket:
      Type: AWS::S3::Bucket
      Properties:
        BucketName: ${self:custom.targetBucketName}
        AccessControl: Private
        LifecycleConfiguration:
          Rules:
            - Id: TempCompressDeletionRule
              Status: Enabled
              ExpirationInDays: 7
        CorsConfiguration:
          CorsRules:
          - AllowedMethods:
            - GET
            - POST
            AllowedOrigins:
            - "*"
            AllowedHeaders:
            - "*"
        PublicAccessBlockConfiguration:
          BlockPublicAcls: true
          BlockPublicPolicy: true
          IgnorePublicAcls: true
          RestrictPublicBuckets: true
    PublicBucket:
      Type: AWS::S3::Bucket
      Properties:
        BucketName: ${self:custom.publicBucketName}
        AccessControl: Private
        CorsConfiguration:
          CorsRules:
          - AllowedMethods:
            - GET
            - POST
            AllowedOrigins:
            - "*"
            AllowedHeaders:
            - "*"
    SourceBucketPolicy:
      Type: AWS::S3::BucketPolicy
      Properties:
        PolicyDocument:
          Statement:
            - Sid: AdminActions
              Effect: Allow
              Principal:
                AWS: arn:aws:iam::${AWS::AccountId}:root
              Action:
                - "s3:GetObject"
                - "s3:GetObjectAcl"
                - "s3:PutObject"
                - "s3:PutObjectAcl"
                - "s3:DeleteObject"
              Resource: arn:aws:s3:::${self:custom.sourceBucketName}/*
        Bucket: ${self:custom.sourceBucketName}
    TargetBucketPolicy:
      Type: AWS::S3::BucketPolicy
      Properties:
        PolicyDocument:
          Statement:
            - Sid: AdminActions
              Effect: Allow
              Principal:
                AWS: arn:aws:iam::${AWS::AccountId}:root
              Action:
                - "s3:GetObject"
                - "s3:GetObjectAcl"
                - "s3:PutObject"
                - "s3:PutObjectAcl"
                - "s3:DeleteObject"
              Resource: arn:aws:s3:::${self:custom.targetBucketName}/*
        Bucket:
          Ref: TargetBucket
    PublicBucketPolicy:
      Type: AWS::S3::BucketPolicy
      Properties:
        PolicyDocument:
          Statement:
            - Sid: PublicObjectRead
              Effect: Allow
              Principal: "*"
              Action:
                - "s3:GetObject"
              Resource: arn:aws:s3:::${self:custom.publicBucketName}/*
            - Sid: AdminActions
              Effect: Allow
              Principal:
                AWS: arn:aws:iam::${AWS::AccountId}:root
              Action:
                - "s3:GetObject"
                - "s3:GetObjectAcl"
                - "s3:PutObject"
                - "s3:PutObjectAcl"
                - "s3:DeleteObject"
              Resource: arn:aws:s3:::${self:custom.publicBucketName}/*
        Bucket:
          Ref: PublicBucket
    CompressImagesHandlerLambdaPermissionSourceBucketS3:
      Type: "AWS::Lambda::Permission"
      Properties:
        FunctionName:
          "Fn::GetAtt":
            - CompressImagesHandlerLambdaFunction
            - Arn
        Principal: "s3.amazonaws.com"
        Action: "lambda:InvokeFunction"
        SourceAccount:
          Ref: AWS::AccountId
        SourceArn: arn:aws:s3:::${self:custom.sourceBucketName}

Error

The full error I am getting is:

Error:
UPDATE_FAILED: IamRoleLambdaExecution (AWS::IAM::Role)
The policy failed legacy parsing (Service: AmazonIdentityManagement; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: *redacted*; Proxy: null)

If anyone can identify why I am receiving this error, I would really, really appreciate the help. I have been banging my head against the wall for way too long on this with no progress, and I am up against the wall with a deadline. My backup plan is to revert to what I had working before and try to install the compression event trigger manually on the AWS web console, but I would really prefer the whole thing be written out using serverless/IaC. Hopefully it’s something stupid that my tired eyes just can’t see anymore.

Thanks,
Avana

Update:

For some reason this forum’s spam bot hid my reply to this post, so I’m adding it here.

I solved the malformed policy issue. For reference, I had to use ${aws:accountId} in string replacement contexts but AWS::AccountId in Ref: contexts. I was switching between either format for all in my attempts,

A new problem rears its ugly head

Anyway, now I’m getting a new error that has me stumped.

Error:
CREATE_FAILED: S3BucketSourceBucket (AWS::S3::Bucket)
*redacted* already exists in stack arn:aws:cloudformation:us-east-1:*redacted*:stack/*redacted*/*redacted*

Somehow cloud formation is trying to create my sourceBucket twice, even though I specifically removed it from the resources section because it’s defined with the event trigger under provider. I also tried removing the associated BucketPolicy to no avail.

I followed the instructions here to a tee, for configuring an event trigger to work with a custom bucket configuration.

What’s weird is in my attempts to troubleshoot this earlier, I removed all of the event trigger stuff, and just tried to have the original code that was working, plus create another two buckets as resources and I was getting this same error. As soon as I removed the other two buckets it worked again. Yet there was no naming confusion or mixed up variables.

So I am thinking either there is something going on with the fact that I’m trying to crate multiple buckets, or the guidance for referencing custom buckets in the serverless docs is wrong.

Can anyone help?

2

Update:

So I finally figured out that in string replacement of my aws acct ID I have to use ${aws:accountId}, but when using the account ID in a Ref: statement, I have to use AWS::AccountId. I was flipping between either or based on dodgy information from google searches.

A new problem rears its ugly head

Anyway, I got rid of the malformed policy error, but now it’s telling me that my sourceBucket already exists. I don’t understand this one because I removed it from the resources section specifically for this reason, and I tried removing its associated BucketPolicy.

Error:
CREATE_FAILED: S3BucketSourceBucket (AWS::S3::Bucket)
*redacted* already exists in stack arn:aws:cloudformation:us-east-1:*redacted*:stack/*redacted*/*redacted*

I followed the instructions here for using a custom bucket configuration with event triggers, but somehow it’s trying to create the bucket twice.

The weird thing is, when I was troubleshooting the original issue, I noticed this happened when I got rid of all of the event trigger stuff, and just tried to create 3 different buckets, with nothing else happening. As soon as I removed the two other buckets it would go back to working.

So my feeling is it either has something to do with trying to create multiple buckets, or the instructions in the serverless docs are wrong, and there is another way I should be referencing my s3 bucket defined in the provider section.

Can anyone help?

3

Update 2

I was able to get this stack deployed, but not with the event trigger, which no matter how I structured it or referenced my handler function, kept giving me this error:

Error:
UPDATE_FAILED: CompressImagesHandlerLambdaFunction (AWS::Lambda::Function)
Resource handler returned message: "Lambda function *redacted*-compressImagesHandler could not be found" (RequestToken: *redacted*, HandlerErrorCode: NotFound)

So, in the end I ended up just abandoning the event trigger, using serverless to scaffold my HTTP API Lambdas, and S3 Buckets, as well as all their IAM config, and then I just logged into the AWS console and added the event trigger manually, which is super simple.

I still wish I could have done this all in serverless using IaC, so if anyone has any ideas, I’d appreciate it if you shared!

4

I ran into a similar problem today, the solution for me was defining the “Resource” a bit differently.

From

arn:aws:secretsmanager:${self:custom.region}:${AWS::AccountId}:secret:*

To

arn:aws:secretsmanager:${self:custom.region}:${self:custom.accountId}:secret:*

Or defining the entire thing as an env var established in a preflow