Aurora Serverless Cluster

Hello, I’ve been searching around the forums in hopes someone had an answer to my question but basically I’ve noticed that at the moment, there doesn’t seem to be a means of deploying an Aurora Serverless Cluster through Serverless Components at the moment. I’ve sifted through all of the posted npm modules and I’m wondering if anyone knows when this will happen or if there’s some type of workaround with regards to this? Preferably; I would like to use Serverless Components for this thanks to its Open-Source Licensing as opposed to Amazon’s CloudFormation.

I wonder if that is because Aurora RDS would look like MySQL or PostgreSQL to npm? I’ve used node12.x/RDS/MySql done manually and used the mysql npm modules vs anything specific for RDS/Aurora.

Although you do have different end points for ReadOnly vs the Write.

-Andrew

Hi Nytingale,

I’ve managed to get an Aurora Serverless Cluster to deploy with CloudFormation (not Components). However, I should note there are some caveats and prerequisites.

One thing to note is that Aurora Serverless Clusters exist in a VPC and as far as I recall, cannot exist outside of one.

This means that in order to connect to the cluster, your serverless functions would also have to deploy in the same VPC with an Internet Gateway etc… This results in extraordinarily execution times and for a production service is not really fit for purpose, at least in my opinion, particularly when coupled with the cold start of the cluster and the lambda function it can easily be in excess of 10 seconds.

This can be solved using the RDSDataService which allows SQL commands to be executed on the Aurora RDS using HTTP requests authenticated with a Secret ARN. This removes the requirement for the lambda being deployed in the Cluster VPC.

Also, as serverless infrastructure itself, the RDS Cluster does have cold starts which in my experience will exceed the default APIGateway timeout when you attempt to connect from one of your serverless functions and results in a 500 error. This can be confusing at first so be sure to manage the timeouts on requests to the RDS.

Right, on to the actual explanation of how to get an Aurora Serverless Cluster deployed. You’re going to need a few things.

I’ve added a link to each of the corresponding template documentation pages. Please review this and adjust the templates accordingly, the examples below do not include delete protection etc…

VPC and Subnets

VPC & Subnets CloudFormation Example
VPC: 
  Type: AWS::EC2::VPC

  Properties:
    CidrBlock: 172.32.0.0/16
    EnableDnsSupport: true
    EnableDnsHostnames: true
    InstanceTenancy: default

SubnetA:
  Type: AWS::EC2::Subnet

  Properties:
    CidrBlock: 172.32.0.0/20
    AvailabilityZone: ${ self:provider.region }a
    MapPublicIpOnLaunch: true
    
    VpcId:
      Ref: VPC

SubnetB:
  Type: AWS::EC2::Subnet

  Properties:
    CidrBlock: 172.32.16.0/20
    AvailabilityZone: ${ self:provider.region }b
    MapPublicIpOnLaunch: true
    
    VpcId:
      Ref: VPC

SubnetC:
  Type: AWS::EC2::Subnet

  Properties:
    CidrBlock: 172.32.32.0/20
    AvailabilityZone: ${ self:provider.region }c
    MapPublicIpOnLaunch: true
    
    VpcId:
      Ref: VPC

Security Group

Security Group CloudFormation Example
SecurityGroup:
  Type: AWS::EC2::SecurityGroup

  Properties:
    GroupName: ${ self:service }-security-group

    SecurityGroupIngress:
      - IpProtocol: tcp
        FromPort: 3306
        ToPort: 3306

    VpcId: 
      Ref: VPC

RDS Subnet Group

RDS Subnet Group CloudFormation Example
RDSSubnetGroup:
  Type: AWS::RDS::DBSubnetGroup

  Properties: 
    SubnetIds:
      - Ref: SubnetA
      - Ref: SubnetB
      - Ref: SubnetC

Secret Manager Secret

Secret Manager Secret CloudFormation Example
AuroraSecret:
  Type: AWS::SecretsManager::Secret

  Properties:
    Name: ${ self:service }-aurora-secret
    Description: ${ self:service } serverless aurora secret

    GenerateSecretString:
      SecretStringTemplate: !Join ['', ['{ "username": "', '${ self:provider.environment.DATABASE_USERNAME }', '" }']]
      GenerateStringKey: 'password'
      PasswordLength: 30
      ExcludeCharacters: '"@/\'

Serverless Aurora Cluster

Serverless Aurora Cluster CloudFormation Example
AuroraServerlessPostgresCluster:
  Type: AWS::RDS::DBCluster

  Properties:
    MasterUsername: !Join ['', ['{{resolve:secretsmanager:', !Ref AuroraSecret, ':SecretString:username}}' ]]
    MasterUserPassword: !Join ['', ['{{resolve:secretsmanager:', !Ref AuroraSecret, ':SecretString:password}}' ]]

    DatabaseName: ${ self:provider.environment.DATABASE_NAME }

    Engine: aurora-postgresql
    EngineMode: serverless

    DBSubnetGroupName: 
      Ref: RDSSubnetGroup

    VpcSecurityGroupIds:
      - Fn::GetAtt: SecurityGroup.GroupId

    EnableHttpEndpoint: true # Enable the Data API

AuroraServerlessPostgresClusterSecretAttachment:
  Type: AWS::SecretsManager::SecretTargetAttachment

  Properties:
    SecretId: !Ref AuroraSecret
    TargetId: !Ref AuroraServerlessPostgresCluster
    TargetType: AWS::RDS::DBCluster

For the purposes of this example, I’m assuming that the above templates are defined in separate yaml files.

With that, in your serverless.yml they would need to be included in resources section.

resources:
  - ${ file(infrastructure/vpc/vpc.yml) }
  - ${ file(infrastructure/aurora/subnet-group.yml) }
  - ${ file(infrastructure/aurora/security-group.yml) }
  - ${ file(infrastructure/aurora/secret.yml) }
  - ${ file(infrastructure/aurora/serverless-cluster.yml) }

You will also need to add the relevant IAMRoleStatements for the RDS Data API and the Secrets Manager.

The AWS SDK includes an RDSDataService that can be used to connect to and excute commands to your RDS Instance.

Here’s the Javascript RDSDataService SDK for reference.
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/RDSDataService.html

I hope this was helpful, it took me a while to initially figure out.

Thankfully enough the really long wait times is no longer an issue with Lambda as AWS has made some changes under the hood to correct that issue as of November last year if I recall.

I wasn’t aware of this, that’s awesome!

You can see the massive difference that this has made in function execution duration, with it dropping to 933 ms from 14.8 seconds!