How to create DeploymentBucket when manually specified if it doesn't exist

Serverless Framework
1

In production environment, ops team pre-creates deployment buckets. In other environments, they don’t exercise such control. How can I configure serverless such that: pre-defined deployment bucket is used in production environment and bucket is created if needed in other environments.

Suppose I have something like this:

custom:
  defaultStage: stage
  stage: ${opt:stage, self:custom.defaultStage}
  # Use config for given stage (defaults to 'stage' if unspecified)
  config: ${file(./lib/Configuration.js):${self:custom.stage}}
provider:
  name: aws
  role: ${self:custom.config.LAMBDA_ROLE}
  deploymentBucket: ${self:custom.config.SERVERLESS_DEPLOYMENT_BUCKET}

In the above, the deploymentBucket name is expected to vary based on the ‘-s’ value passed on command line. I have to specify the deploymentBucket key for production, but I can’t ‘unspecify’ that key in other deployment environments… If instead I explicitly specify some unique value for each environment, then I don’t get the desired auto-creation of the bucket in the non-production environments…

Is there anyway to solve this?

It seems like I need some way to ‘unset’ keys within the config system – for example in ‘dev’ environment, I could specify a value like ‘SERVERLESS_UNSET_KEY_SENTINEL’

custom:
    deploymentBucket: SERVERLESS_UNSET_KEY_SENTINEL
provider:
   deploymentBucket: ${self:custom.deploymentBucket}

Which would have the effect of removing the deploymentBucket key (reverting to the behavior you’d get if you’d left it unspecified…)

2

I think you’re going to struggle getting this to work, because you’re effectively asking for non-idempotent deploys which is a Bad Idea.

Couldn’t you just pre-create/define buckets in all your environments (with the production environment owned by ops, and the rest owned by you), and then just create an environment/bucket name mapping? That way the deployment steps are the same, regardless of the environment.

3

I can pre-create the deployment bucket but I would much rather expess the deployment process/requirements within the serverless configuration … It seems like that’s a big part of the point of using a deployment framework …?

I spelunked around in the framework code trying to understand how it works to see if I could find a clean way to achieve this – the logic around the handling of provider.deploymentBucket is bananas and very hard to follow.

This is the approach I tried first and seems the cleanest – … but I couldn’t find a way to get it to work …

resources:
  Conditions:
    # CreateDeploymentBucket:
    #   Fn::Equals:
    #     - ${self:custom.config.CONDITION_CREATE_DEPLOYMENT_BUCKET}
    #     - true
    CreateRoles:
      Fn::Equals:
        - ${self:custom.config.CONDITION_CREATE_ROLES}
        - true
    CreateTables:
      Fn::Equals:
        - ${self:custom.config.CONDITION_CREATE_TABLES}
        - true
  Resources:
    # ServerlessDeploymentBucket:
    #   Type: AWS::S3::Bucket
    #   Condition: CreateDeploymentBucket
    #   BucketName: ${self:custom.config.SERVERLESS_DEPLOYMENT_BUCKET}
    SomeTable:
      Type: AWS::DynamoDB::Table
      Condition: CreateTables
      Properties: ${self:custom.dynamodb.SomeTable.meta}
    someRole:
      Type: AWS::IAM::Role
      Condition: CreateRoles
      Properties:
            ...

The idea being that CONDITION_CREATE_DEPLOYMENT_BUCKET would be false in environments where an ops team manages the creation of required resources … I have to work within some external constraints regarding ops policies for deployment - but I want to implement the project in such a way that I can incrementally over time move as much of that external deployment logic into project code over time as the understanding with ops evolves …

4

I’m up for a challenge but it comes with a warning: While I regularly use stage specific settings I haven’t tried anything this complex so it may fail spectacularly. Also, I haven’t tested this so you may have a bit of debugging to do.

I would start with something like this inside your serverless.yml

custom:
  defaultStage: stage
  stage: ${opt:stage, self:custom.defaultStage}
  provider:
    default: &default_provider
      name: aws
    dev:
      <<: *default_provider
    prod:
      <<: *default_provider
      role: LAMBDA_ROLE
      deploymentBucket: SERVERLESS_DEPLOYMENT_BUCKET

provider: ${self:custom.provider.${self:custom.stage}}

What does this do?

provider will gets its values from customer.provider.STAGE where STAGE is either the stage from the command line or custom.defaultStage.

custom.provider.STAGE then contains the entire provider section for that stage.

To make this a little more DRY I’m using YAML anchors (a feature of YAML and not specific to Serverless). custom.provider.default should contain any settings that are common to all stages. The other stages inherit these through <<: *default_provider. The default values can be override per stage by simply setting the value again in that stage. In this example I’m adding role and deploymentBucket in custom.provider.prod because they’re only applicable to prod.

There’s a lot that could go wrong here, especially around the use of YAML anchors. If they’re causing you problems then you could try removing them or moving the providers to their own file.

Hopefully this will help you.

PS: I wrote a about per stage environment variables over here and I’m basically applying that to the provider section.

5

Yikes my brain will break if I try to use yaml anchors inside serverless.yml …

I ended up adding a plugin that attaches to the package lifecyle and tries to create the deployment bucket given by provider.DeploymentBucket … Im not 100% sure which event is appropriate to attach to, but this event occurs before the code that checks to verify the existence of the deploymentBucket s3 resource so it works …

import { AwsProviderFromServerless } from "./AwsProviderFromServerless";

/**
 * Serverless Plugin for creating s3 deployment bucket
 */
export class CreateDeploymentBucketPlugin {
  public commands = {
    "deploy-bucket": {
      lifecycleEvents: ["deploymentBucket"],
      usage: "Ensure the deployment bucket is created",
      options: {
        stage: {
          usage:
            "Create the deployment bucket for a given stage (dev, staging, prod)",
          required: true,
          shortcut: "s"
        }
      }
    }
  };

  public get hooks() {
    return {
      "before:package:initialize": this.ensureExistsDeploymentBucket,
      "deploy-bucket:deploymentBucket": this.ensureExistsDeploymentBucket
    };
  }

  private serverless: any;
  private options: any;

  constructor(serverless: any, options: any) {
    this.serverless = serverless;
    this.options = options;
  }

  public ensureExistsDeploymentBucket = async () => {
    const awsprovider = new AwsProviderFromServerless(this.serverless);
    awsprovider.validate();
    const s3 = awsprovider.S3();

    this.serverless.cli.log(
      `Ensure DeploymentBucket: ${awsprovider.deploymentBucket} exists in region: ${awsprovider.region}`
    );

    try {
      // try to create the bucket
      const createResponse = await s3
        .createBucket({
          Bucket: awsprovider.deploymentBucket
        })
        .promise();

      if (createResponse.Location) {
        this.serverless.cli.log(
          `DeploymentBucket: ${awsprovider.deploymentBucket} exists in region: ${awsprovider.region}`
        );
      }
    } catch (err) {
      if (
        err.code === "BucketAlreadyExists" &&
        err.region === awsprovider.region
      ) {
        this.serverless.cli.log(
          `DeploymentBucket already exists in region: ${err.region}`
        );
      } else if (
        err.code === "BucketAlreadyExists" &&
        err.region === awsprovider.region
      ) {
        this.serverless.cli.log(
          `DeploymentBucket exists in unexpected region: ${err.region} expected: ${awsprovider.region}`
        );
      } else {
        this.serverless.cli.log(
          `Could not create deployment bucket: ${awsprovider.deploymentBucket}`
        );
        throw err;
      }
    }
  };
}

AWSProviderFromServerless for reference … (just a class I reuse in various plugins for getting typed and correctly configured objects for interacting with various aws-sdk apis)

import { IsString } from "class-validator";
import { AWSProvider } from "../Helpers";

export class AwsProviderFromServerless extends AWSProvider {
  get profile(): string {
    return this.serverless.service.provider.profile;
  }

  get stage(): string {
    return this.serverless.service.provider.stage;
  }

  get region(): string {
    return this.serverless.service.provider.region;
  }

  @IsString({
    message:
      "Serverless property: provider.deploymentBucket must be a string, Found $value"
  })
  get deploymentBucket(): string {
    return this.serverless.service.provider.deploymentBucket;
  }

  protected serverless: any;

  constructor(serverless: any) {
    super();
    this.serverless = serverless;
  }
}

Guess I need to include AwsProvider also …

import {
  Config as AWSConfig,
  DynamoDB,
  LexModelBuildingService as Lex,
  S3,
  SharedIniFileCredentials
} from "aws-sdk";

import { IsString, Validator } from "class-validator";

/**
 * One stop shop for getting AWS interface objects
 */
export abstract class AWSProvider {
  @IsString({
    message: "Property: profile must be a string.  Found $value"
  })
  public abstract profile: string;

  @IsString({
    message: "Property: region must be a string.  Found $value"
  })
  public abstract region: string;

  @IsString({
    message: "Property: stage must be a string.  Found $value"
  })
  public abstract stage: string;

  public awsConfig = (): AWSConfig => {
    return new AWSConfig({
      credentials: new SharedIniFileCredentials({ profile: this.profile }),
      region: this.region
    });
  };

  public S3 = (): S3 => {
    return new S3(this.awsConfig);
  };

  public Lex = (): Lex => {
    return new Lex(this.awsConfig);
  };

  public DynamoDB = (): DynamoDB => {
    return new DynamoDB(this.awsConfig);
  };

  public validate() {
    const validator = new Validator();
    const errors = validator.validateSync(this);

    if (errors.length > 0) {
      let errMessage =
        "AwsConfigFromServerless: Error validating serverless config\n";
      for (const err of errors) {
        errMessage += Object.keys(err.constraints)
          .map(key => {
            return err.constraints[key];
          })
          .join("");
      }

      throw errMessage;
    }
  }
}