I wonder if I hit a bug or if I’m doing anything wrong. Please consider that I don’t have any node experience.
I’m trying to use Localstack with Serverless. I start Localstack with docker-compose, I can run successfully basic things at the command line, like create a mock S3 bucket or store mock secrets in SecretsManager.
I have a serverless.yml config that works without issues when I deploy to AWS stages other than local (dev, uat, prod).
But when I try to deploy on local, I get the error you see in the subject, after 4 attempts to recover.
$ serverless deploy --region eu-west-1 --stage local --profile local
Serverless: Load command interactiveCli
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command config:tabcompletion
Serverless: Load command config:tabcompletion:install
Serverless: Load command config:tabcompletion:uninstall
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command upgrade
Serverless: Load command uninstall
Serverless: Load command requirements
Serverless: Load command requirements:clean
Serverless: Load command requirements:install
Serverless: Load command requirements:cleanCache
Serverless: config.options_stage: local
Serverless: serverless.service.custom.stage: undefined
Serverless: serverless.service.provider.stage: ${opt:stage, self:custom.defaultStage}
Serverless: config.stage: local
Serverless: Using serverless-localstack
Serverless: Load command deploy
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command generate-event
Serverless: Load command test
Serverless: Load command dashboard
Serverless: Load command output
Serverless: Load command output:get
Serverless: Load command output:list
Serverless: Load command param
Serverless: Load command param:get
Serverless: Load command param:list
Serverless: Load command studio
Serverless: Invoke deploy
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: config.options_stage: local
Serverless: serverless.service.custom.stage: undefined
Serverless: serverless.service.provider.stage: local
Serverless: config.stage: local
Serverless: Using serverless-localstack
Serverless: Reconfiguring service apigateway to use http://localhost:4566
Serverless: Reconfiguring service cloudformation to use http://localhost:4566
Serverless: Reconfiguring service cloudwatch to use http://localhost:4566
Serverless: Reconfiguring service lambda to use http://localhost:4566
Serverless: Reconfiguring service dynamodb to use http://localhost:4566
Serverless: Reconfiguring service kinesis to use http://localhost:4566
Serverless: Reconfiguring service route53 to use http://localhost:4566
Serverless: Reconfiguring service firehose to use http://localhost:4566
Serverless: Reconfiguring service stepfunctions to use http://localhost:4566
Serverless: Reconfiguring service es to use http://localhost:4566
Serverless: Reconfiguring service s3 to use http://localhost:4566
Serverless: Reconfiguring service ses to use http://localhost:4566
Serverless: Reconfiguring service sns to use http://localhost:4566
Serverless: Reconfiguring service sqs to use http://localhost:4566
Serverless: Reconfiguring service sts to use http://localhost:4566
Serverless: Reconfiguring service iam to use http://localhost:4566
Serverless: Reconfiguring service ssm to use http://localhost:4566
Serverless: Reconfiguring service rds to use http://localhost:4566
Serverless: Reconfiguring service ec2 to use http://localhost:4566
Serverless: Reconfiguring service elasticache to use http://localhost:4566
Serverless: Reconfiguring service kms to use http://localhost:4566
Serverless: Reconfiguring service secretsmanager to use http://localhost:4566
Serverless: Reconfiguring service logs to use http://localhost:4566
Serverless: Reconfiguring service cloudwatchlogs to use http://localhost:4566
Serverless: Reconfiguring service iot to use http://localhost:4566
Serverless: Reconfiguring service cognito-idp to use http://localhost:4566
Serverless: Reconfiguring service cognito-identity to use http://localhost:4566
Serverless: Reconfiguring service ecs to use http://localhost:4566
Serverless: Reconfiguring service eks to use http://localhost:4566
Serverless: Reconfiguring service xray to use http://localhost:4566
Serverless: Reconfiguring service appsync to use http://localhost:4566
Serverless: Reconfiguring service cloudfront to use http://localhost:4566
Serverless: Reconfiguring service athena to use http://localhost:4566
Serverless: Warning: Unable to find plugin named: TypeScriptPlugin
Serverless: config.options_stage: local
Serverless: serverless.service.custom.stage: undefined
Serverless: serverless.service.provider.stage: local
Serverless: config.stage: local
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Generated requirements from /Users/vw393/Desktop/infrastructure/serverless/sls-alarms/es-alarms/requirements.txt in /Users/vw393/Desktop/infrastructure/serverless/sls-alarms/es-alarms/.serverless/requirements.txt...
Serverless: Using static cache of requirements found at /Users/vw393/Library/Caches/serverless-python-requirements/6569aae901279c09889b9a7ab82ce395fe0f5239c56ea9fa6891deb3cf0399d6_slspyc ...
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Injecting required Python packages to package...
Serverless: Invoke aws:package:finalize
Serverless: Invoke aws:common:moveArtifactsToPackage
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:deploy:deploy
Serverless: [AWS cloudformation 200 0.523s 3 retries] describeStacks({ StackName: 'es-alarms-local' })
Serverless: Recoverable error occurred (Non-whitespace before first tag.
Line: 0
Column: 1
Char: {), sleeping for ~6 seconds. Try 1 of 4
Serverless: [AWS cloudformation 200 0.324s 3 retries] describeStacks({ StackName: 'es-alarms-local' })
Serverless: Recoverable error occurred (Non-whitespace before first tag.
Line: 0
Column: 1
Char: {), sleeping for ~4 seconds. Try 2 of 4
Serverless: [AWS cloudformation 200 0.21s 3 retries] describeStacks({ StackName: 'es-alarms-local' })
Serverless: Recoverable error occurred (Non-whitespace before first tag.
Line: 0
Column: 1
Char: {), sleeping for ~4 seconds. Try 3 of 4
Serverless: [AWS cloudformation 200 0.321s 3 retries] describeStacks({ StackName: 'es-alarms-local' })
Serverless: Recoverable error occurred (Non-whitespace before first tag.
Line: 0
Column: 1
Char: {), sleeping for ~6 seconds. Try 4 of 4
Serverless: [AWS cloudformation 200 0.388s 3 retries] describeStacks({ StackName: 'es-alarms-local' })
Serverless Error ---------------------------------------
ServerlessError: Non-whitespace before first tag.
Line: 0
Column: 1
Char: {
at /usr/local/lib/node_modules/serverless/lib/plugins/aws/provider/awsProvider.js:621:27
at processTicksAndRejections (internal/process/task_queues.js:93:5)
Get Support --------------------------------------------
Docs: docs.serverless.com
Bugs: github.com/serverless/serverless/issues
Issues: forum.serverless.com
Your Environment Information ---------------------------
Operating System: darwin
Node Version: 14.11.0
Framework Version: 2.0.0
Plugin Version: 4.0.2
SDK Version: 2.3.1
Components Version: 3.1.2
Following, my serverless.yml file:
service: es-alarms
frameworkVersion: '2'
provider:
name: aws
runtime: python3.8
stage: ${opt:stage, self:custom.defaultStage}
profile: ${self:custom.profiles.${opt:stage, self:provider.stage, 'local'}}
versionFunctions: false
memorySize: 128
region: eu-west-1
environment:
ES_CLUSTER: ${opt:stage, self:custom.defaultStage}
iamManagedPolicies:
- "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
custom:
defaultStage: local
profiles:
local: local
dev: dev
uat: uat
prod: prod
pythonRequirements:
dockerizePip: non-linux
localstack:
debug: true
stages:
# list of stages for which the plugin should be enabled
- local
host: http://localhost # optional - LocalStack host to connect to
autostart: true # optional - start LocalStack in Docker on Serverless deploy
endpoints:
# This section is optional - can be used for customizing the target endpoints
# S3: http://localhost:4572
# DynamoDB: http://localhost:4570
# CloudFormation: http://localhost:4581
# Elasticsearch: http://localhost:4571
# ES: http://localhost:4578
# SNS: http://localhost:4575
# SQS: http://localhost:4576
# Lambda: http://localhost:4574
# Kinesis: http://localhost:4568
lambda:
# Enable this flag to improve performance
mountCode: false
docker:
# Enable this flag to run "docker ..." commands as sudo
sudo: False
functions:
notify-jira:
handler: handler_notify_jira.open_jira_kdh
package:
artifact:
events:
- sns: es-cloudwatch-alarms
iamRoleStatements:
- Effect: 'Allow'
Action:
- 'secretsmanager:GetResourcePolicy'
- 'secretsmanager:GetSecretValue'
- 'secretsmanager:DescribeSecret'
- 'secretsmanager:List*'
Resource: 'arn:aws:secretsmanager:eu-west-1:62********09:secret:abc/sls-alarms/jira_*'
notify-teams:
handler: handler_notify_teams.send_teams_notif
package:
artifact:
events:
- sns: es-cloudwatch-alarms
iamRoleStatements:
- Effect: 'Allow'
Action:
- 'secretsmanager:GetResourcePolicy'
- 'secretsmanager:GetSecretValue'
- 'secretsmanager:DescribeSecret'
- 'secretsmanager:List*'
Resource: 'arn:aws:secretsmanager:eu-west-1:62********09:secret:abc/sls-alarms/teams/${opt:stage}/webhook_cloudwatch_alarms*'
plugins:
- serverless-python-requirements
- serverless-iam-roles-per-function
- serverless-localstack
Also, I noticed that when the errors above are triggered, the Localstack docker container logs few times this error:
2020-09-16T14:11:07:WARNING:bootstrap.py: Thread run method <function AdaptiveThreadPool.submit.<locals>._run at 0x7fa6e7b39b80>(None) failed: 'cloudformation' Traceback (most recent call last):
File "/opt/code/localstack/localstack/utils/bootstrap.py", line 533, in run
result = self.func(self.params)
File "/opt/code/localstack/localstack/utils/async_utils.py", line 28, in _run
return fn(*args, **kwargs)
File "/opt/code/localstack/localstack/services/generic_proxy.py", line 557, in handler
response = modify_and_forward(method=method, path=path_with_params, data_bytes=data, headers=headers,
File "/opt/code/localstack/localstack/services/generic_proxy.py", line 332, in modify_and_forward
listener_result = listener.forward_request(method=method,
File "/opt/code/localstack/localstack/services/edge.py", line 80, in forward_request
return do_forward_request(api, port, method, path, data, headers)
File "/opt/code/localstack/localstack/services/edge.py", line 85, in do_forward_request
result = do_forward_request_inmem(api, port, method, path, data, headers)
File "/opt/code/localstack/localstack/services/edge.py", line 95, in do_forward_request_inmem
service_name, backend_port, listener = PROXY_LISTENERS[api]
KeyError: 'cloudformation'