Getting awsKmsKeyArn to work

I’m trying to get encrypted parameters to work with a python app, but not having much success.

In serverless.yml I have:

service:
  name: my-sql
  awsKmsKeyArn: arn:aws:kms:eu-west-1:...

provider:
  name: aws
  runtime: python3.6
  ...
  environment: ${file(env_dev.yml)}

(aside: I wonder why awsKmsKeyArn goes under “service” rather than “provider”?)

And in env_dev.yml I have:

DATABASE_URL: mysql+pymysql://user:pass@etc

This all works fine without the encryption.

Now, adding the awsKmsKeyArn doesn’t seem to have made any difference to the behaviour of my app. When I deploy the stack, I find that the variable is not encrypted - that is, it is visible in the AWS lambda console, and the app finds the plain value it needs in the environment.

I can click to enable encryption in the AWS console, but after “sls deploy” it is back to plaintext.

So it’s not clear to me: if I specify “awsKmsKeyArn” by itself, what is supposed to happen? The setting does end up in the cloudformation template.

Next: when I enable encryption helper manually in the console, it gives me some suggested code to use in my application:

import boto3
import os

from base64 import b64decode

ENCRYPTED = os.environ['DATABASE_URL']
# Decrypt code should run once and variables stored outside of the function
# handler so that these are decrypted once per container
DECRYPTED = boto3.client('kms').decrypt(CiphertextBlob=b64decode(ENCRYPTED))['Plaintext']

def lambda_handler(event, context):
  # TODO handle the event here

This is good, it means the decryption is supposed to take place in the application itself. But I need cloudformation to deploy with the encrypted key.

So: should I put the already-encrypted and base64-encoded data in the YAML? Or should my YAML contain the plaintext, and the encryption takes place when Cloudstack deploys it?

The next thing I tried was encrypting the data locally.

aws kms encrypt --key-id arn:aws:kms:eu-west-1:... --plaintext "mysql+pymysql://user:pass@etc"

This gave me a nice encrypted base64 blob. I put it in the YAML, added the decode logic to the python, and deployed. But now the lambda function just gives me an execution timeout error (even after upping it to 20 seconds). Adding print() statements shows it’s the decrypt() step which is hanging.

Furthermore, the AWS console displays this base64 data as if it were unencrypted text. That is, it still has the “Encrypt” button available.

I wondered if I might have to add something like this:

  iamRoleStatements:
    - Effect: Allow
      Action:
        - kms:Decrypt
      Resource: arn:aws:kms:eu-west-1:...

but (a) the sls documentation doesn’t mention this, and (b) I would have expected sls to add this automatically, if this was needed, since I set awsKmsKeyArn. And anyway, I added it and it didn’t make any difference.

So I’ve clearly missed the point as to how this is supposed to work. Any clues gratefully received!

Thanks,

Brian.

P.S. I did read https://serverless.com/framework/docs/providers/aws/guide/functions#secrets-using-environment-variables-and-kms

It links to
http://docs.aws.amazon.com/lambda/latest/dg/env_variables.html#env-storing-sensitive-data
and
http://docs.aws.amazon.com/lambda/latest/dg/tutorial-env_console.html
but these aren’t very helpful, as they assume you’re using the console to encrypt the key.

Well, I found one of the answers here: the reason for the timeout is because my lambda function is running in a VPC, without an Internet gateway, but KMS has to be accessed over the Internet. Grr.

I’m still not clear how the buttons in the AWS Lambda console to “encrypt/decrypt” individual environment variables are related to the decryption done within the lambda; nor what role the awsKmsKeyArn setting in serverless plays.

After finally finding this:
https://github.com/awslabs/serverless-application-model/issues/48

… it seems that CloudFormation doesn’t support the encryption of individual keys, which the Lambda console does.

The awsKmsKeyArn seems to be just for the bulk encryption-at-rest of the settings.

Anyway, next thing is to deploy NAT gateways for VPC subnets.

1 Like

I found this confusing too but can share what I found in using KMS with serverless for encrypting sensitive environment variables.

  • I don’t use the console encryption helpers - these are unnecessary when using Serverless or CloudFormation for defining the Lambda functions
  • I encrypt the data using the AWS CLI as you have done
  • I decrypt explicitly in code using the KMS API in the AWS SDK (JavaScript in my case)
  • The cleartext environment variables therefore never have to be committed to source control as part of Serverless YML files
  • AWS Lambda does not know that these values are encrypted once the functions are deployed, unlike variables that were encrypted using the console encryption helpers
  • Defining the awsKmsKeyArn in your service or functions, from what I can tell, populates the “KMS key” section in the the Lambda Function’s Advanced Settings with the KMS key you specify. This key is then used when your Lambda function uses the KMS API to decrypt, so you don’t have to explicitly select the key again. I’m not actually sure whether this setting also results in AWS using this key to encrypt every environment at rest when the function is created and updates. If this is the case, it means you have double encryption of some variables at rest. Either way, it doesn’t matter to me as the problem I’m solving is avoiding cleartext sensitive environment variables in source control, in transit and at rest.

I hope this helps. It would be great if some of the assumptions here can be clarified.

I agree with most of the above, I just mention this part:

I believe that the encrypted data block actually includes the key ID, because you can use “aws kms decrypt” to decrypt a block, without having to specify the key ID.

So as you say, the awsKmsKeyArn parameter might be to do with how data is encrypted at rest in the absence of explicit encryption; but it’s not at all clear!

1 Like

That makes sense, thanks.