Hello, I’ve been searching around the forums in hopes someone had an answer to my question but basically I’ve noticed that at the moment, there doesn’t seem to be a means of deploying an Aurora Serverless Cluster through Serverless Components at the moment. I’ve sifted through all of the posted npm modules and I’m wondering if anyone knows when this will happen or if there’s some type of workaround with regards to this? Preferably; I would like to use Serverless Components for this thanks to its Open-Source Licensing as opposed to Amazon’s CloudFormation.
I wonder if that is because Aurora RDS would look like MySQL or PostgreSQL to npm? I’ve used node12.x/RDS/MySql done manually and used the mysql npm modules vs anything specific for RDS/Aurora.
Although you do have different end points for ReadOnly vs the Write.
-Andrew
Hi Nytingale,
I’ve managed to get an Aurora Serverless Cluster to deploy with CloudFormation (not Components). However, I should note there are some caveats and prerequisites.
One thing to note is that Aurora Serverless Clusters exist in a VPC and as far as I recall, cannot exist outside of one.
This means that in order to connect to the cluster, your serverless functions would also have to deploy in the same VPC with an Internet Gateway etc… This results in extraordinarily execution times and for a production service is not really fit for purpose, at least in my opinion, particularly when coupled with the cold start of the cluster and the lambda function it can easily be in excess of 10 seconds.
This can be solved using the RDSDataService which allows SQL commands to be executed on the Aurora RDS using HTTP requests authenticated with a Secret ARN. This removes the requirement for the lambda being deployed in the Cluster VPC.
Also, as serverless infrastructure itself, the RDS Cluster does have cold starts which in my experience will exceed the default APIGateway timeout when you attempt to connect from one of your serverless functions and results in a 500 error. This can be confusing at first so be sure to manage the timeouts on requests to the RDS.
Right, on to the actual explanation of how to get an Aurora Serverless Cluster deployed. You’re going to need a few things.
I’ve added a link to each of the corresponding template documentation pages. Please review this and adjust the templates accordingly, the examples below do not include delete protection etc…
VPC & Subnets CloudFormation Example
VPC:
Type: AWS::EC2::VPC
Properties:
CidrBlock: 172.32.0.0/16
EnableDnsSupport: true
EnableDnsHostnames: true
InstanceTenancy: default
SubnetA:
Type: AWS::EC2::Subnet
Properties:
CidrBlock: 172.32.0.0/20
AvailabilityZone: ${ self:provider.region }a
MapPublicIpOnLaunch: true
VpcId:
Ref: VPC
SubnetB:
Type: AWS::EC2::Subnet
Properties:
CidrBlock: 172.32.16.0/20
AvailabilityZone: ${ self:provider.region }b
MapPublicIpOnLaunch: true
VpcId:
Ref: VPC
SubnetC:
Type: AWS::EC2::Subnet
Properties:
CidrBlock: 172.32.32.0/20
AvailabilityZone: ${ self:provider.region }c
MapPublicIpOnLaunch: true
VpcId:
Ref: VPC
Security Group CloudFormation Example
SecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupName: ${ self:service }-security-group
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 3306
ToPort: 3306
VpcId:
Ref: VPC
RDS Subnet Group CloudFormation Example
RDSSubnetGroup:
Type: AWS::RDS::DBSubnetGroup
Properties:
SubnetIds:
- Ref: SubnetA
- Ref: SubnetB
- Ref: SubnetC
Secret Manager Secret CloudFormation Example
AuroraSecret:
Type: AWS::SecretsManager::Secret
Properties:
Name: ${ self:service }-aurora-secret
Description: ${ self:service } serverless aurora secret
GenerateSecretString:
SecretStringTemplate: !Join ['', ['{ "username": "', '${ self:provider.environment.DATABASE_USERNAME }', '" }']]
GenerateStringKey: 'password'
PasswordLength: 30
ExcludeCharacters: '"@/\'
Serverless Aurora Cluster CloudFormation Example
AuroraServerlessPostgresCluster:
Type: AWS::RDS::DBCluster
Properties:
MasterUsername: !Join ['', ['{{resolve:secretsmanager:', !Ref AuroraSecret, ':SecretString:username}}' ]]
MasterUserPassword: !Join ['', ['{{resolve:secretsmanager:', !Ref AuroraSecret, ':SecretString:password}}' ]]
DatabaseName: ${ self:provider.environment.DATABASE_NAME }
Engine: aurora-postgresql
EngineMode: serverless
DBSubnetGroupName:
Ref: RDSSubnetGroup
VpcSecurityGroupIds:
- Fn::GetAtt: SecurityGroup.GroupId
EnableHttpEndpoint: true # Enable the Data API
AuroraServerlessPostgresClusterSecretAttachment:
Type: AWS::SecretsManager::SecretTargetAttachment
Properties:
SecretId: !Ref AuroraSecret
TargetId: !Ref AuroraServerlessPostgresCluster
TargetType: AWS::RDS::DBCluster
For the purposes of this example, I’m assuming that the above templates are defined in separate yaml files.
With that, in your serverless.yml they would need to be included in resources section.
resources:
- ${ file(infrastructure/vpc/vpc.yml) }
- ${ file(infrastructure/aurora/subnet-group.yml) }
- ${ file(infrastructure/aurora/security-group.yml) }
- ${ file(infrastructure/aurora/secret.yml) }
- ${ file(infrastructure/aurora/serverless-cluster.yml) }
You will also need to add the relevant IAMRoleStatements for the RDS Data API and the Secrets Manager.
The AWS SDK includes an RDSDataService that can be used to connect to and excute commands to your RDS Instance.
Here’s the Javascript RDSDataService SDK for reference.
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/RDSDataService.html
I hope this was helpful, it took me a while to initially figure out.
1 Like
Thankfully enough the really long wait times is no longer an issue with Lambda as AWS has made some changes under the hood to correct that issue as of November last year if I recall.
I wasn’t aware of this, that’s awesome!
You can see the massive difference that this has made in function execution duration, with it dropping to 933 ms from 14.8 seconds!