Pre-Production Scans: Uploads via Screwdriver CD

This article describes the process for configuring automated uploads of mobile app builds to Data Theorem, from a CI/CD pipeline in Screwdriver. At the high-level the process is the following:

1. Retrieve the Data Theorem Upload API Key
2. Save the Upload API Key as a Secret in Screwdriver
3. Create a new Screwdriver job for uploading a build to Data Theorem
4. Create a bash script for uploading a build to Data Theorem
5. Enable caching of the mobile app build
6. Deploy and test in CI

Step 1: Retrieve the Data Theorem Upload API Key

To be able to use the Upload API provided by Data Theorem, you must first retrieve the Upload API Key to authenticate against the API.

Please visit https://www.securetheorem.com/sdlc/api_access and copy the key labeled as “Upload API Key”.

Step 2: Save the Upload API Key as a Secret in Screwdriver

To be able to use it for sending mobile app builds to Data Theorem, we have to save the Upload API Key as a Secret in Screwdriver.

To do so, please visit the secrets tab within your pipeline to create a new secret:

Enter DT_UPLOAD_API_KEY into the field with the value set to the Upload API Key that was retrieved in Step 1.

Step 3: Create a new Screwdriver job for uploading a build to Data Theorem

Next is creating a new job specifically to upload the newly built binary to Data Theorem.

In screwdriver.yaml please create the following:

# ....
jobs:
  upload_binary_to_datatheorem:
    requires: job_that_builds_binary # Please read below for more information
    secrets:
      - DT_UPLOAD_API_KEY
    steps:
      - run_upload_script: ./upload_mobile_binaries_to_datatheorem.sh $PATH_TO_BINARY_TO_UPLOAD # We will create the bash script in the next step
# ....

The two things to pay attention here is in the requires section.

The first is to find out the path of where the binary will be created. You will need to find this out for step 5.

Note: For this example we'll assume it's located in $SD_SOURCE_DIR/build/apps/binary.apk.

Second, is the value that should go in the requires section, which is the name of the job that is in charge of building the binary.

For example if we have the following in a screwdriver.yaml:

# ....
jobs:
  create_binary:
    steps:
      - build: ./gradlew build # Let's assume it creates a binary in $SD_SOURCE_DIR/build/apps/
  upload_binary_to_datatheorem:
    requires: create_binary # Rely on the job that created the binary
    secrets:
      - DT_UPLOAD_API_KEY
    steps:
      - run_upload_script: ./upload_mobile_binaries_to_datatheorem.sh $SD_SOURCE_DIR/build/apps/binary.apk # Upload the binary that was created by the `create_binary` job
# ....

Step 4: Create a bash script for uploading a build to Data Theorem

Create and add the following contents to a file called: upload_mobile_binaries_to_datatheorem.sh

#!/usr/bin/env bash
# Purpose of this script is to send mobile binary builds to Data Theoerm's Upload API
# Example call:
# ./upload_mobile_binaries_to_datatheorem.sh path/to/mobile/binary/to/upload

# Fail if any commands fails
set -ex

maxRetries=3
for (( retry = 0; retry < maxRetries; retry++ ))
do
  # Step 1: get the upload URL
  echo "Get upload url"
  step1_response=$(curl -s -w "%{http_code}" -X POST -H "Authorization: APIKey ${DT_UPLOAD_API_KEY}"  --data ""  https://api.securetheorem.com/uploadapi/v1/upload_init)
  http_code=${step1_response: -3}
  response_body=${step1_response::-3}

  # For older versions of bash e.g. GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin21)
  # response_body=${step1_response%???}
  # http_code=${step1_response#${response_body}}

  # Check that http status code is 200
  [ ! ${http_code} -eq 200 ] && echo ${response_body} && exit 1
  upload_url=$(echo ${response_body} | jq -r ".upload_url")
  echo ${upload_url}

  # Step 2: upload the APK
  echo "Upload app"
  step2_response=$(curl -F file=@${1} ${upload_url}) && echo ${step2_response} && break
done

if [ ${retry} -ge ${maxRetries} ]; then
  echo "Upload failed after ${maxRetries} attempts"
  exit 1
fi

The script will fail if any of the commands are unsuccessful. 

Please save this script where deemed appropriate, such as a directory with other scripts.

Step 5: Enable caching of the mobile app build

The final thing to complete this process is to enable caching in between jobs. This is needed to make the mobile app build available across multiple jobs, including the job that will upload it to Data Theorem.

To allow caching of the directory where the mobile app build will be created, you will first need to identify the path where it will be created, and then add this path to the cache: section of the screwdriver.yaml file:

# ....
# Top level
cache:
  event: [$SD_SOURCE_DIR/build/apps/]  # Things to be cached/shared between jobs

# ....
jobs:
  main:
    # ...  
# ....

Step 6: Deploy and test in CI

Please deploy all changes to CI to ensure the mobile app builds are automatically uploaded to Data Theorem.

If you encounter any issues or have any questions, please email support at support@datatheorem.com.