All Articles

Prevent Secrets Leakage During CloudFormation Deployments

Photo by https://unsplash.com/@nicoli_
Photo by https://unsplash.com/@nicoli_

TLDR: Add the NoEcho property to sensitive CloudFormation parameters to mask their value

Introduction

I decided recently to slowly but surely convert all my custom deploy scripts to CloudFormation Custom Resources.

At its core, a Custom Resource is a Lambda function that adheres to a specific contract and used to add custom logic to a CloudFormation deployment.

If you’d like to learn more about Custom Resources check out this great post by Alex DeBrie.

Learn By Example

Here is a usage example for a Custom Resource I wrote to create a CI/CD setup with GitHub, CircleCI and AWS (I’m using serverless framework for deployment):

# serverless.yml

service: aws-custom-resources

provider:
  name: aws

  stage: ${opt:stage, 'dev'}
  region: ${opt:region, 'us-east-1'}

resources:
  Resources:
    CircleCI:
      Type: 'Custom::CircleCI'
      Version: '1.0'
      Properties:
        ServiceToken:
          Fn::ImportValue: 'circleci-custom-resource:${self:provider.stage}:ServiceToken'
        ApiToken: ${env:CIRCLECI_API_TOKEN} # CircleCI REST API token
        Owner: ${env:CIRCLECI_OWNER} # GitHub repository owner
        Repo: ${env:CIRCLECI_REPO} # GitHub repository name
        EnvironmentVariables: # Coveralls configuration
          - name: COVERALLS_SERVICE_NAME
            value: CircleCI
          - name: COVERALLS_REPO_TOKEN
            value: ${env:COVERALLS_REPO_TOKEN}

Looks pretty straightforward right?

I’m configuring an API Token for the CircleCI REST API, the GitHub repository details and some additional variables to report tests coverage to Coveralls.

Let move on to the next step to deploy our shiny new stack.

Deployment

Running the following command:

serverless deploy --stage prod --region us-east-1

Deploys the stack and configures the CI/CD. Woohoo emoji-fire

Here is the command output:

Serverless: Packaging service...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Validating template...
Serverless: Updating Stack...
Service Information
service: aws-custom-resources
stage: prod
region: us-east-1
stack: aws-custom-resources-prod
resources: 2
api keys:
  None
endpoints:
  None
functions:
  None
layers:
  None
Serverless: Run the "serverless" command to setup monitoring, troubleshooting and testing.
Done in 10.75s.

On first glance everything seems fine, but running the describe-stack-events command will expose an issue with the deployment.

Revealing the Secrets

Running:

aws cloudformation describe-stack-events --stack-name 'aws-custom-resources-prod' --region us-east-1

Will provide information about the deployment (only relevant parts are displayed):

{
  ...
  "ResourceStatus": "CREATE_IN_PROGRESS",
  "ResourceProperties": "{\"ServiceToken\":\"arn:aws:lambda:us-east-1:123456789:function:circleci-custom-resource-prod-circleci\",\"Owner\":\"erezrokah\",\"EnvironmentVariables\":[{\"name\":\"COVERALLS_SERVICE_NAME\",\"value\":\"CircleCI\"},{\"name\":\"COVERALLS_REPO_TOKEN\",\"value\":\"Y*************************L\"}],\"Repo\":\"aws-custom-resources\",\"ApiToken\":\"7*************************c\"}"
}

If you look closely you’ll notice the COVERALLS_REPO_TOKEN and ApiToken properties are visible via the describe-stack-events API call (I replaced the real values with asterisks).

This behavior is unwanted and can lead to secrets leakage.

Protecting the Secrets

Luckily, CloudFormation provides a very simple solution for this issue. By using the NoEcho attribute when specifying CloudFormation Parameters we can mask sensitive values:

# serverless.yml

service: aws-custom-resources

provider:
  name: aws

  stage: ${opt:stage, 'dev'}
  region: ${opt:region, 'us-east-1'}

resources:
  Parameters:
    ApiTokenParameter:
      Type: String
      Default: ${env:CIRCLECI_API_TOKEN}
      NoEcho: true # Mask the token value
    CoverallsRepoTokenParameter:
      Type: String
      Default: ${env:COVERALLS_REPO_TOKEN}
      NoEcho: true # Mask the token value

  Resources:
    CircleCI:
      Type: 'Custom::CircleCI'
      Version: '1.0'
      Properties:
        ServiceToken:
          Fn::ImportValue: 'circleci-custom-resource:${self:provider.stage}:ServiceToken'
        ApiToken: { Ref: ApiTokenParameter }
        Owner: ${env:CIRCLECI_OWNER}
        Repo: ${env:CIRCLECI_REPO}
        EnvironmentVariables:
          - name: COVERALLS_SERVICE_NAME
            value: CircleCI
          - name: COVERALLS_REPO_TOKEN
            value: { Ref: CoverallsRepoTokenParameter }

Running describe-stack-events again will generate a similar output, but with the token values masked:

{  
  ...
  "ResourceStatus": "CREATE_IN_PROGRESS",
  "ResourceProperties": "{\"ServiceToken\":\"arn:aws:lambda:us-east-1:123456789:function:circleci-custom-resource-prod-circleci\",\"Owner\":\"erezrokah\",\"EnvironmentVariables\":[{\"name\":\"COVERALLS_SERVICE_NAME\",\"value\":\"CircleCI\"},{\"name\":\"COVERALLS_REPO_TOKEN\",\"value\":\"****\"}],\"Repo\":\"aws-custom-resources\",\"ApiToken\":\"****\"}"
}

Summary

We’ve seen how secrets can easily leak through CloudFormation deployments, and provided a simple solution to mask them.

When dealing with secrets it is important to understand how they are handled every step of the way to make sure they are not exposed.