Resource lookup using incorrect aws profile

Serverless Framework
,
1

Greetings,

After a recent upgrade to 1.30.0 I am noticing an interesting behavior. I’ve got a serverless.yml file that creates a lambda function:

functions:  detectNudity:
    handler: functions/detect_nudity.lambda_handler
    timeout: 5

I am invoking serverless using an aws profile for my staging environment:

$ serverless deploy --stage=staging --aws-profile=serverless-staging -v

And as I would expect, I can see the lambda function being created with an ARN that references the AWS Account ID associated with the given profile:

Service Information
service: MyService
stage: staging
region: us-east-1
stack: MyService-staging
api keys:
  None
endpoints:
  None
functions:
  detectNudity: MyService-staging-detectNudity

Stack Outputs
DetectNudityLambdaFunctionQualifiedArn: arn:aws:lambda:us-east-1:<EXPECTED_ACCOUNT_ID>:function:MyService-staging-detectNudity:40

However, as the build is completing, there appears to be an attempt to look up things that have been created, and when this step is hit, the ARN for the function is not correct. It contains the account id associated with my default aws profile credentials:

  Resource Not Found Exception ---------------------------

  Function not found: arn:aws:lambda:us-east-1:<DEFAULT_ACCOUNT_ID>:function:MyService-staging-detectNudity

There is a traceback associated with this error:

ResourceNotFoundException: Function not found: arn:aws:lambda:us-east-1:972099457318:function:ContentModeration-staging-profileDecision
    at Object.extractError (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/protocol/json.js:48:27)
    at Request.extractError (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/protocol/rest_json.js:52:8)
    at Request.callListeners (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
    at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:685:12)
    at Request.callListeners (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
    at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:685:12)
    at Request.callListeners (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
    at callNextListener (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:95:12)
    at IncomingMessage.onEnd (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/event_listeners.js:294:13)
    at emitNone (events.js:111:20)
    at IncomingMessage.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickDomainCallback (internal/process/next_tick.js:218:9)
From previous event:
    at PluginManager.invoke (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:390:22)
    at PluginManager.run (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:421:17)
    at variables.populateService.then.then (/usr/local/lib/node_modules/serverless/lib/Serverless.js:157:33)
    at runCallback (timers.js:810:20)
    at tryOnImmediate (timers.js:768:5)
    at processImmediate [as _immediateCallback] (timers.js:745:5)
From previous event:
    at Serverless.run (/usr/local/lib/node_modules/serverless/lib/Serverless.js:144:8)
    at serverless.init.then (/usr/local/lib/node_modules/serverless/bin/serverless:43:50)
    at <anonymous>

I’ve tried removing the [default] profile and credentials from my ~/.aws/credentials file, leaving only named profiles, but doing so results in a different error:

  Credentials Error --------------------------------------

  Missing credentials in config

     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Stack Trace --------------------------------------------

CredentialsError: Missing credentials in config
    at Object._errnoException (util.js:992:11)
    at _exceptionWithHostPort (util.js:1014:20)
    at internalConnect (net.js:960:16)
    at defaultTriggerAsyncIdScope (internal/async_hooks.js:284:19)
    at net.js:1058:9
    at _combinedTickCallback (internal/process/next_tick.js:131:7)
    at process._tickDomainCallback (internal/process/next_tick.js:218:9)
From previous event:
    at PluginManager.invoke (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:390:22)
    at PluginManager.run (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:421:17)
    at variables.populateService.then.then (/usr/local/lib/node_modules/serverless/lib/Serverless.js:157:33)
    at runCallback (timers.js:810:20)
    at tryOnImmediate (timers.js:768:5)
    at processImmediate [as _immediateCallback] (timers.js:745:5)
From previous event:
    at Serverless.run (/usr/local/lib/node_modules/serverless/lib/Serverless.js:144:8)
    at serverless.init.then (/usr/local/lib/node_modules/serverless/bin/serverless:43:50)
    at <anonymous>

I have also changed my ~/.aws/credentials file so that the named profile associated with my staging environment is the default profile, and this does work. However, it’s not a great solution as I’d really like to be able to use the same credentials file while deploying to different environments.

It seems clear from the observed behavior that there is some step in the serverless build process that is not passing the explicit profile name into an AWS SDK call, as the default behavior for the AWS SDK is to fall back to the default profile, but I have no idea how to figure out what step it is, or how to determine if this is a bug or a problem in my usage pattern.

I’d deeply appreciate any help anyone might offer in determining how best to fix this issue.

Yours

c

2

As usual, I’ve found a workaround minutes after giving up and posting here.

In my case, the most viable solution appears to be to have a ~/.aws/credentials file that contains only named profiles:

[prod]
aws_access_key_id = <SOME VALUE>
aws_secret_access_key = <SOME VALUE>

[serverless-staging]
aws_access_key_id = <SOME VALUE>
aws_secret_access_key = <SOME VALUE>

And then to set the AWS_PROFILE environment variable to ensure that the fallback profile matches the profile I have passed on the command line via --aws-profile. This solution works for my use-case, and unblocks me from moving forward.

However, it is definitely more evidence that something serverless is doing is falling back on the default AWS profile, and I still think it’d be nice to figure out what that is.

c

1 Like