AWS creds not available to Container functions?

Serverless Framework
,
1

While I have deployed hundreds of NodeJS Lambda functions via serverless this is my first time working with container functions. As always, it works flawlessly locally but now I am trying to get over the first time deploy to AWS hump.

  • I am deploying a containerised Node function derived from public.ecr.aws/lambda/nodejs:14.
  • The function copies some files to S3
  • Serverless has created an IAM role that is attached to the Lambda
  • The role includes s3:PutObject and s3:PutObjectAcl on the bucket and bucket paths
  • CloudWatch logs CredentialsProviderError: Could not load credentials from any providers
  • I started with the default const s3 = new S3Client() which I have always used in the past in native Node Lambda functions without issue. But that triggers the error.
  • I have tried forcing the region const s3 = new S3Client({region: "us-east-1"}) so that the constructor property is not empty. That too resulted in the same error.
  • When I echo out the environment it includes an AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID that do not match any of my known credentials. I believe these are autogenerated by AWS and unique to the IAM role.
  • If I edit my S3 client constructor to include the access key and secret found in the environment the function still does not work and still reports the same error.
  • I can pass user credentials, as found in my ~/.aws/credentials file and that does work. But I should not have to create a specific user when an IAM role is being created automatically.

I am wondering if it has something to do with the runtime interface client and emulator?

2

Ok, looks like a typical red-herring sort of AWS error message and was due to a misconfigured S3 ARN in serverless.yml.

If I replace my environment variables in the snippet below with the hardcoded ARN everything works. My deploy script that calls serverless deploy is setting the environment variables correctly so now I need to figure out why the variable is not being expanded properly in serverless.yml.

iam:
  role:
    statements:
      - Effect: "Allow"
        Action:
          - "s3:PutObject"
          - "s3:PutObjectAcl"
        Resource:
          # S3_ORIGIN_ARN is dynamically set by deploy.sh
          - ${env:S3_ORIGIN_ARN}/*
          - ${env:S3_ORIGIN_ARN}
3

It looks like environment variables have to be declared before they can be used? The only way I could use the environment variable in the resources section was to add a higher level environment section. This is both redundant and counter-intuitive. But if I don’t include the environment: section below I get the CredentialsProviderError: Could not load credentials from any providers error.

  environment:
    # Required so that IAM role resources can be properly expanded
    S3_ORIGIN_ARN: ${env:S3_ORIGIN_ARN}
    CF_DRAFT_ARN: ${env:CF_DRAFT_ARN}
    CF_PUBLISHED_ARN: ${env:CF_PUBLISHED_ARN}
  iam:
    role:
      statements:
        - Effect: "Allow"
          Action:
            - "s3:PutObject"
            - "s3:PutObjectAcl"
          Resource:
            # S3_ORIGIN is dynamically set by deploy.sh
            - ${env:S3_ORIGIN_ARN}/*
            - ${env:S3_ORIGIN_ARN}
        - Effect: "Allow"
          Action:
            - "cloudfront:CreateInvalidation"
          Resource:
            # CF_* is dynamically set by deploy.sh
            - ${env:CF_DRAFT_ARN}
            - ${env:CF_PUBLISHED_ARN}