Monitoring Blue/Green deployment in CICD

Hi,

I use the plugin serverless-plugin-canary-deployments to perform Blue/Green deployments of lambda functions using Code Deploy.
The Cloud Formation stack doesn’t wait the result of Blue/Green deployment to go into UPDATE_COMPLETE state.
I would like to monitor Blue/Green deployments in CICD to get the final result of the deployment, in order to prevent to deploy in later stages some lambda that failed the validation tests.

I consider to use the following command from AWS CLI :
‘aws deploy wait deployment-successful --deployment-id $DEPLOYMENT_ID’

My problem is that I don’t know how to get $DEPLOYMENT_ID from the serverless deployment result.

Did someone know how to do that ?

1 Like

I’m interested in what you do, if you find a solution please post it here.

Thanks!

Hi @a4xrbj1

I found a workaround that could be used, even if it is not the best way to do it.
The AWS CLI or NodeJS sdk has the following methods that are usefull:

  • list-deployment-groups
  • list-deployments
  • get-deployment

Reference : deploy — AWS CLI 1.31.3 Command Reference

I used the NodeJS sdk, because I think it is easier to parse JSON result.

To monitor Blue/Green deployments of a Serverless app, I did the following steps:

  1. Configure serverless file with canary plugin, and run a first deployment.
  2. From step 1, a Code Deploy application is created. Keep the name of this application for later use.
  3. Install the aws sdk for Code Deploy : AWS SDK for JavaScript v3
  4. Create a file “monitorServerlessDeployments.js” and add the code below.
  5. Update the IAM role used for deployment if it doesn’t have the permission to perform the three methods listed above.
  6. Add the environment variable “AWS_DEFAULT_REGION” and “SERVERLESS_DEPLOY_APP_NAME” (the name of the Code Deploy application created at step 1).
  7. Update your CI deployment job to run the NodeJS script after the “serverless deploy” line.
  8. Test your job and verify that the script monitor the deployment results.

NB: It looks for the deployment in the paste 10 minutes, change the window if it doesn’t fit your needs.

Below is the code of the NodeJS script :

// This script is used to monitor the deployment of the serverless application
// It will check the status of the deployment every 10 seconds
// It will exit with a non-zero exit code if at least one of the deployments fails

const {
    CodeDeployClient,
    ListDeploymentGroupsCommand,
    ListDeploymentsCommand,
    GetDeploymentCommand,
} = require("@aws-sdk/client-codedeploy");

async function main() {
    if (!process.env.SERVERLESS_DEPLOY_APP_NAME || !process.env.AWS_DEFAULT_REGION) {
        throw new Error(
            "Missing environment variables SERVERLESS_DEPLOY_APP_NAME or AWS_DEFAULT_REGION",
        );
    }
    // Get the list of deploymentGroupId of the serverless application of Code Deploy
    console.log(
        `Fetching deployment groups of ${process.env.SERVERLESS_DEPLOY_APP_NAME} in region ${process.env.AWS_DEFAULT_REGION}`,
    );

    const codeDeployClient = new CodeDeployClient({
        region: process.env.AWS_DEFAULT_REGION,
    });
    let listDeploymentGroupsCommand = new ListDeploymentGroupsCommand({
        applicationName: process.env.SERVERLESS_DEPLOY_APP_NAME,
    });

    let deploymentGroupList = [];
    let currentDeploymentGroupList = {};
    // Result is paginated, need a loop to get all results
    do {
        currentDeploymentGroupList = await codeDeployClient.send(
            listDeploymentGroupsCommand,
        );
        deploymentGroupList = deploymentGroupList.concat(
            currentDeploymentGroupList.deploymentGroups,
        );
        listDeploymentGroupsCommand = new ListDeploymentsCommand({
            applicationName: process.env.SERVERLESS_DEPLOY_APP_NAME,
            nextToken: currentDeploymentGroupList.nextToken,
        });
    } while (currentDeploymentGroupList.nextToken);

    console.log(`Found ${deploymentGroupList.length} deployments`, deploymentGroupList);

    // Get the limit UNIX timestamps to fetch the deployments from the last 10 minutes
    const endDate = new Date() - 0;
    const startDate = new Date(endDate - 600000) - 0;

    console.log(
        `Fetching deployments from ${startDate} to ${endDate} of ${process.env.SERVERLESS_DEPLOY_APP_NAME} in region ${process.env.AWS_DEFAULT_REGION}`,
    );

    let recentDeploymentList = [];
    // get recent deployments for each deployment group
    for (const currentDeploymentGroup of deploymentGroupList) {
        // Get the list of deploymentId of the serverless application of Code Deploy
        let listDeploymentsCommand = new ListDeploymentsCommand({
            applicationName: process.env.SERVERLESS_DEPLOY_APP_NAME,
            deploymentGroupName: currentDeploymentGroup,
            createTimeRange: {
                start: new Date(startDate),
                end: new Date(endDate),
            },
        });

        let currentDeploymentList = {};
        let deploymentList = [];
        // result is paginated, need a loop to get all results
        do {
            currentDeploymentList = await codeDeployClient.send(listDeploymentsCommand);

            deploymentList = deploymentList.concat(currentDeploymentList.deployments);

            listDeploymentsCommand = new ListDeploymentsCommand({
                applicationName: process.env.SERVERLESS_DEPLOY_APP_NAME,
                deploymentGroupName: currentDeploymentGroup,
                createTimeRange: {
                    start: new Date(startDate),
                    end: new Date(endDate),
                },
                nextToken: currentDeploymentList.nextToken,
            });
        } while (currentDeploymentList.nextToken);

        // add the deployments to the list of recent deployments
        recentDeploymentList.push(...deploymentList);

        console.log(
            `Found ${deploymentList.length} deployments for deployment group ${currentDeploymentGroup}`,
            deploymentList,
        );
    }

    console.log(
        `Found ${recentDeploymentList.length} recent deployments`,
        recentDeploymentList,
    );

    if (recentDeploymentList.length === 0) {
        console.warn("\nNo deployment to monitor... 🤔\n");
    } else {
        const succeededDeployments = [];
        const stoppedDeployments = [];
        const failedDeployments = [];
        // do while all deployments are not finished
        do {
            // get the status of each deployment
            for (const deploymentId of recentDeploymentList) {
                const deploymentInfo = await codeDeployClient.send(
                    new GetDeploymentCommand({
                        deploymentId: deploymentId,
                    }),
                );

                // if the deployment is finished
                if (deploymentInfo.deploymentInfo.status !== "InProgress") {
                    // if the deployment failed
                    if (deploymentInfo.deploymentInfo.status === "Failed") {
                        failedDeployments.push(
                            deploymentInfo.deploymentInfo.deploymentGroupName +
                                " : " +
                                deploymentId,
                        );
                        recentDeploymentList = recentDeploymentList.filter((item) => {
                            return item !== deploymentId;
                        });
                        console.log(
                            `\nDeployment ${deploymentInfo.deploymentInfo.deploymentGroupName} failed`,
                            deploymentInfo.deploymentInfo.errorInformation,
                        );
                    } else if (deploymentInfo.deploymentInfo.status === "Stopped") {
                        stoppedDeployments.push(
                            deploymentInfo.deploymentInfo.deploymentGroupName +
                                " : " +
                                deploymentId,
                        );
                        recentDeploymentList = recentDeploymentList.filter((item) => {
                            return item !== deploymentId;
                        });
                        console.log(
                            `\nDeployment ${deploymentInfo.deploymentInfo.deploymentGroupName} has been stopped`,
                            deploymentInfo.deploymentInfo.deploymentStatusMessages,
                        );
                    } else if (deploymentInfo.deploymentInfo.status === "Succeeded") {
                        succeededDeployments.push(
                            deploymentInfo.deploymentInfo.deploymentGroupName +
                                " : " +
                                deploymentId,
                        );
                        recentDeploymentList = recentDeploymentList.filter((item) => {
                            return item !== deploymentId;
                        });
                        console.log(
                            `\nDeployment ${deploymentInfo.deploymentInfo.deploymentGroupName} succeeded`,
                            deploymentInfo.deploymentInfo.deploymentStatusMessages,
                        );
                    }
                }
            }

            if (recentDeploymentList.length > 0) {
                console.log("\n...");
                // sleep 10sec
                await new Promise((resolve) => {
                    setTimeout(resolve, 10000);
                });
            }
        } while (recentDeploymentList.length > 0);

        console.log("\n");
        console.log(
            `Found ${succeededDeployments.length} succeeded deployments`,
            succeededDeployments,
        );
        console.log(
            `Found ${stoppedDeployments.length} stopped deployments`,
            stoppedDeployments,
        );
        console.log(
            `Found ${failedDeployments.length} failed deployments`,
            failedDeployments,
        );

        if (failedDeployments.length > 0) {
            throw new Error(
                `Found ${failedDeployments.length} failed deployments`,
                failedDeployments,
            );
        } else if (stoppedDeployments.length > 0) {
            throw new Error(
                `Found ${stoppedDeployments.length} stopped deployments`,
                stoppedDeployments,
            );
        } else {
            console.log("\nAll deployments succeeded 😎");
        }
    }
}

// call the main function.
main();

I hope that it will be helpful to someone. :slight_smile: