Page Comparison

...

This can for example be used as part of a CI/CD pipeline in order to verify that the terraform file won’t create any resource with some urgent policy violation. This way customers can prevent the deployment of a cloud resource can be prevented to be deployed in production.

The feature was deployed on  2021/05/26. At that time we are only checking for urgent policy violations in S3 buckets. Alban (Unlicensed) Thomas Sileo and Marc Tranzer (Unlicensed) will have to prioritize what potential terraform issues should be checked first.

See https://www.terraform.io/docs/language/index.html for an explanation on terraform Terraform files.

Running a terraform scan

Data Theorem’s API can be used to run a terraform scan against a specific terraform configuration file

1. Retrieve an API Key that has the permission “API Security Results API” enabled; API keys are available in the Data Theorem portal at

...

1. API Key. 

2. A terraform file scan can then be run using the following CURL command:

   Code Block
   curl -X POST 'https://api.securetheorem.com/apis/devops/v1/iac_scans' \ --header 'Content-Type: multipart/form-data' \ --header 'Authorization: APIKey ABCACBA=' \ --form 'file=@"terraform_example_configuration:file.tf"' \ --form 'scan_type="TERRAFORM"'

3. Check the output for any issues in the files.

...

export terraform_issues_count=$(curl -X POST 'https://api.securetheorem.com/apis/devops/v1/iac_scans' \
--header 'Content-Type: multipart/form-data' \
--header 'Authorization: APIKey ABCACBA=' \
--form 'file=@"terraform_example_configuration:file.tf"' \
--form 'scan_type="TERRAFORM"'| jq -r ".issues_count")

// Then deploy your file only if terraform_issues_count is equal to 0

Use Public API to see all recent IAC scans

1. TODO: Alban (Unlicensed) Spencer James (Unlicensed) We should design a portal page to see recent IAC Scans results

curl -X POST 'https:///prod-horizon.appspot.com//public/v2/iac_scans' \
--header 'Authorization: Session ABCACBA=' \

response:
class IacScansResponse(BasePaginatedResponse):
    iac_scans: List[IacScanSummaryField]
  
class IacScanSummaryField:
    id: UUID
    date_created: datetime
    status: IacScanStatusEnum
    scan_type: IacScanTypeEnum
    scanned_files_name: List[str]
    issue_count: intif [ $TERRAFORM_ISSUES_COUNT == 0 ]; then
  echo "Deploying file: terraform_example_configuration"
else
  exit 1

Integrating into a CI/CD pipeline

GitHub Actions

For terraform files hosted on GitHub, a GitHub Actions workflow can be configured. The workflow will perform terraform scans every time the repository is tagged with a new version.

To setup this workflow:

1. Create a new secret containing your API key (called DATATHEOREM_API_RESULT_API_KEY in the below example).

2. Create a new workflow by creating a file at .github/workflows/datatheorem.yaml with the following content:

name: Data Theorem Terraform Scans
on:
  push:
    tags:
       - '*'
jobs:
  datatheorem-terraform-scan:
    runs-on: ubuntu-latest
    steps:
      - env:
          DATATHEOREM_WEB_SECURE_SCANS_API_KEY: ${{ secrets.DATATHEOREM_API_RESULT_API_KEY }}
        run: |
          curl -X POST 'https://api.securetheorem.com/apis/devops/v1/iac_scans' \
              --header 'Content-Type: multipart/form-data' \
              --header 'Authorization: APIKey $DATATHEOREM_API_RESULT_API_KEY' \
              --form 'file=@"terraform_example_configuration:file.tf"' \
              --form 'scan_type="TERRAFORM"'

Bitbucket Pipelines

For terraform files hosted on Bitbucket, a similar workflow can be configured in Bitbucket Pipelines:

1. Create a new secure “Repository variables” in Pipelines configuration (from the repository setting).

2. Create a Pipeline by creating a .bitbucket-pipelines.yml within your repository with the following content:

pipelines:
  tags:
    '*':
      - step:
        script:
            - apt-get update
            - apt-get install -y jq
      script:      - export FILEPATH="$BITBUCKET_CLONE_DIR/my_terraform_file.tf"
            - if [ -f "$FILEPATH" ]; then echo "File exists" ; else exit 1; fi
            - |
                export TERRAFORM_DATA_THEOREM_RESPONSE=$(curl -X POST 'https://api.securetheorem.com/apis/devops/v1/iac_scans' \
                                --header 'Content-Type: multipart/form-data' \
                                --header '"Authorization: APIKey $DATATHEOREM_APITERRAFORM_RESULT_API_KEY'" \
                                --form 'file=@"terraform_example_configuration:file.tf'"$FILEPATH"'"' \
                                --form 'scan_type="TERRAFORM"')
                export TERRAFORM_ISSUES_COUNT=$(echo $TERRAFORM_DATA_THEOREM_RESPONSE | tr '\r\n' ' '  | jq -r ".issues_count")
                export MARKDOWN_RESULT=$(echo $TERRAFORM_DATA_THEOREM_RESPONSE | tr '\r\n' ' ' | jq -r ".result_as_markdown")
                if [ $TERRAFORM_ISSUES_COUNT == 0 ]; then
                  echo "Deploying file: terraform_example_configuration"
                else
                  echo "Terraform file contains $TERRAFORM_ISSUES_COUNT issues, abort deployment..."
                  echo "Terraform file issues report: $MARKDOWN_RESULT"
                  exit 1
                fi