CI/CD for terraform scanning

This article describes how to start a terraform file scan using Data Theorem’s API.

This can for example be used as part of a CI/CD pipeline in order to verify that the terraform file won’t create any resource with some urgent policy violation. This way customers can prevent the deployment of a cloud resource in production.

The feature was deployed on 2021/05/26. At that time we are only checking for urgent policy violations in S3 buckets. Alban (Unlicensed) Thomas Sileo and Marc Tranzer (Unlicensed) will have to prioritize what potential terraform issues should be checked first.

See https://www.terraform.io/docs/language/index.html for an explanation on terraform files.

Running a terraform scan

Data Theorem’s API can be used to run a terraform scan against a specific terraform configuration file

Retrieve an API Key that has the permission “API Security Results API” enabled; API keys are available in the Data Theorem portal at https://pr-1593.demotheorem.com/mobile/sdlc/results_api_access.

A terraform file scan can then be run using the following CURL command:

curl -X POST 'https://api.securetheorem.com/apis/devops/v1/iac_scans' \
--header 'Content-Type: multipart/form-data' \
--header 'Authorization: APIKey ABCACBA=' \
--form 'file=@"terraform_example_configuration:file.tf"' \
--form 'scan_type="TERRAFORM"'

3. Check the output for any issues in the files.

response = {
    "status": "COMPLETED" // if value is not COMPLETED then datatheorem was not able to scan the received file
    "issues_count": "1" // if value > 0 then the file is not safe to run on production.
    "result_as_json": {
        "issues": [
            {
                "resource_name": "aws_s3_bucket.public_read_acp_from_grant",
                "linked_policy_rule_type_id": 16
            }
        ]
    }
    // An explanation of the issues encounter in a printable markdown format. 
    "result_as_markdown": "##terraform_example_file.tf contains the following issues:\n###In aws_s3_bucket.public_read_acp_from_grant###\n**AWS S3 Bucket has Publicly Accessible ACLs**  \nThe S3 bucket's Access Control List (ACL) permissions allow anyone to read (but not modify) the bucket's\nACL permissions. This includes anyone on the Internet who knows the bucket's URL.\n\nWhile this permission does not allow accessing\nthe content of the bucket, there is usually no business need for exposing the list of users and their level of access.\n\nThe ACL group that has the `READ_ACP` permission is the `http://acs.amazonaws.com/groups/global/AllUsers` group.\n\nFor more information see the relevant AWS documentation\n[ACL Overview](https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html) and\n[How to restrict Amazon S3 Bucket Access to an\nIAM Role](https://aws.amazon.com/blogs/security/how-to-restrict-amazon-s3-bucket-access-to-a-specific-iam-role/).\n\n"
}

4. To check that the file is safe to deploy, please run

export terraform_issues_count=$(curl -X POST 'https://api.securetheorem.com/apis/devops/v1/iac_scans' \
--header 'Content-Type: multipart/form-data' \
--header 'Authorization: APIKey ABCACBA=' \
--form 'file=@"terraform_example_configuration:file.tf"' \
--form 'scan_type="TERRAFORM"'| jq -r ".issues_count")

// Then deploy your file only if terraform_issues_count is equal to 0

Use Public API to see all recent IAC scans

TODO: Alban (Unlicensed) Spencer James (Unlicensed) We should design a portal page to see recent IAC Scans results

curl -X POST 'https:///prod-horizon.appspot.com//public/v2/iac_scans' \
--header 'Authorization: Session ABCACBA=' \

response:
class IacScansResponse(BasePaginatedResponse):
    iac_scans: List[IacScanSummaryField]
  
class IacScanSummaryField:
    id: UUID
    date_created: datetime
    status: IacScanStatusEnum
    scan_type: IacScanTypeEnum
    scanned_files_name: List[str]
    issue_count: int

Integrating into a CI/CD pipeline

GitHub Actions

For terraform files hosted on GitHub, a GitHub Actions workflow can be configured. The workflow will perform terraform scans every time the repository is tagged with a new version.

To setup this workflow:

Create a new secret containing your API key (called DATATHEOREM_API_RESULT_API_KEY in the below example).
Create a new workflow by creating a file at .github/workflows/datatheorem.yaml with the following content:

name: Data Theorem Terraform Scans
on:
  push:
    tags:
       - '*'
jobs:
  datatheorem-terraform-scan:
    runs-on: ubuntu-latest
    steps:
      - env:
          WEB_APPLICATION_ID: 14df3c48-ff3a-4324-b4a6-13d2904aa785
          DATATHEOREM_WEB_SECURE_SCANS_API_KEY: ${{ secrets.DATATHEOREM_API_RESULT_API_KEY }}
        run: |
          curl -X POST 'https://api.securetheorem.com/apis/devops/v1/iac_scans' \
              --header 'Content-Type: multipart/form-data' \
              --header 'Authorization: APIKey $DATATHEOREM_API_RESULT_API_KEY' \
              --form 'file=@"terraform_example_configuration:file.tf"' \
              --form 'scan_type="TERRAFORM"'

Bitbucket Pipelines

For terraform files hosted on Bitbucket, a similar workflow can be configured in Bitbucket Pipelines:

Create a new secure “Repository variables” in Pipelines configuration (from the repository setting).
Create a Pipeline by creating a .bitbucket-pipelines.yml within your repository with the following content:

pipelines:
  tags:
    '*':
      - step:
          script:
            - |
              curl -X POST 'https://api.securetheorem.com/apis/devops/v1/iac_scans' \
                  --header 'Content-Type: multipart/form-data' \
                  --header 'Authorization: APIKey $DATATHEOREM_API_RESULT_API_KEY' \
                  --form 'file=@"terraform_example_configuration:file.tf"' \
                  --form 'scan_type="TERRAFORM"'