Page Comparison

This article describes the process for configuring automated uploads of mobile app builds to Data Theorem, from a CI/CD pipeline in Screwdriver. At the high-level the process is the following:

1. Retrieve the Data Theorem Upload API Key
2. Save the Upload API Key as a Secret in Screwdriver
3. Create a new Screwdriver job for uploading a build to Data Theorem
4. Create a bash script for uploading a build to Data Theorem
5. Enable caching of the mobile app build
6. Deploy and test in CI

Step 1: Retrieve the Data Theorem Upload API Key

To be able to use the Upload API provided by Data Theorem, you must first retrieve the Upload API Key to authenticate against the API.

Please visit https://www.securetheorem.com/sdlc/api_access and copy the key labeled as “Upload API Key”.

Step 2: Save the Upload API Key as a Secret in Screwdriver

To be able to use it for sending mobile app builds to Data Theorem, we have to save the Upload API Key as a Secret in Screwdriver.

To do so, please visit the secrets tab within your pipeline to create a new secret:TODO: This picture is confusing? What are you trying to show? If it's the Secrets tab maybe we should remove the bottom half of the image

Image Modified

Enter DT_UPLOAD_API_KEY into the field with the value set to the Upload API Key that was retrieved in Step 1.

Image RemovedImage Added

Step 3: Create a new Screwdriver job for uploading a build to Data Theorem

Next is creating a new job specifically to upload the newly built binary to Data Theorem.

...

# ....
jobs:
  upload_binary_to_datatheorem:
    requires: job_that_builds_binary # Please read below for more information
    secrets:
      - DT_UPLOAD_API_KEY
    steps:
      - run_upload_script: ./upload_mobile_binaries_to_datatheorem.sh $PATH_TO_BINARY_TO_UPLOAD # We will create the bash script in the next step
# ....

The one thing two things to pay attention here is in the requires section.

The first is to find out the path of where the binary will be created. You will need to find this out for step 5.

Note: For this example we'll assume it's located in $SD_SOURCE_DIR/build/apps/binary.apk.

Second, is the value that should go here in the requires section, which is the name of the job that is in charge of building the binary.

...

# ....
jobs:
  create_binary:
    steps:
      - build: ./gradlew build # Let's assume it creates a binary in $SD_SOURCE_DIR/build/apps/
  upload_binary_to_datatheorem:
    requires: create_binary # Rely on the job that created the binary
    secrets:
      - DT_UPLOAD_API_KEY
    steps:
      - run_upload_script: ./upload_mobile_binaries_to_datatheorem.sh $SD_SOURCE_DIR/build/apps/binary.apk # Upload the binary that was created by the `create_binary` job
# ....

TODO: is build/apps/binary.apk always going to be the path?

Step 4: Create a bash script for uploading a build to Data Theorem

Create and add the following contents to a file called: upload_mobile_binaries_to_datatheorem.sh

#!/usr/bin/env bash
# Purpose of this script is to send mobile binary builds to Data Theoerm's Upload API
# Example call:
# ./upload_mobile_binaries_to_datatheorem.sh path/to/mobile/binary/to/upload

# Fail if any commands fails
set -eex

maxRetries=3
for (( retry = 0; retry < maxRetries; retry++ ))
do
  # Step 1: get the upload URL
  echo "Get upload url"
  step1_response=$(curl -s -w "%{http_code}" -X POST -H "Authorization: APIKey ${DT_UPLOAD_API_KEY}"  --data ""  https://api.securetheorem.com/uploadapi/v1/upload_init)
upload_url=$(echo 
  http_code=${step1_response: -3}
  response_body=${step1_response::-3}

  # For older versions of bash e.g. GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin21)
  # response_body=${step1_response%???}
  # http_code=${step1_response#${response} | cut -f 3 -d" " | tr -d '"')

_body}}

  # Check that http status code is 200
  [ ! ${http_code} -eq 200 ] && echo ${response_body} && exit 1
  upload_url=$(echo ${response_body} | jq -r ".upload_url")
  echo ${upload_url}

  # Step 2: upload the mobile binary APK
  echo "Upload app"
  step2_response=$(curl -F file=@${1} ${upload_url}) && echo ${step2_response} && break
done

if [ ${retry} --retry 3)

echo "Response from Data Theorem"
echo ${step2_response}ge ${maxRetries} ]; then
  echo "Upload failed after ${maxRetries} attempts"
  exit 1
fi

The script will fail if any of the commands are unsuccessful. 

Please save this script where deemed appropriate, such as a directory with other scripts.

Step 5: Enable caching of the mobile app build

The final thing to complete this process is to enable caching in between jobs. This is needed to make the mobile app build available across multiple jobs, including the job that will upload it to Data Theorem.

...

# ....
# Top level
cache:
  event: [path/to/built/binary$SD_SOURCE_DIR/build/apps/]  # Things to be cached/shared between jobs

# ....
jobs:
  main:
    # ...  
# ....

Step 6: Deploy and test in CI

Please deploy all changes to CI to ensure the mobile app builds are automatically uploaded to Data Theorem.

...