API Protect NGINX Agent Deployment

Owned by Alban (Unlicensed)
Jul 12, 2022
3 min read

Overview

The Data Theorem API Protect nginx agent can be deployed to one or several nginx instances in order to monitor and collect data about incoming API requests. The data is then sent to Data Theorem’s platform for further analysis, in order to automatically discover APIs and detect attacks.

Deployment

Deploying the agent requires running a python script in order to update the nginx configuration files. If you haven’t received the script yet, please contact support@datatheorem.com.

This script will enable traffic mirroring for all incoming requests; the request will be sent to Data Theorem’s traffic analyzer services.

Step 1: Deploy the traffic analyzer docker services

Requirements:

  • Docker

  • docker-compose

The necessary docker-compose.yml needed to deploy these services is contained within the zip archive you should receive for deployment, if you do not have this zip archive please contact support@datatheorem.io.

Extract the zip on the machine which you desire to run the traffic analyzer services.

Deploying the traffic analyzer docker services requires a BEARER_TOKEN, if you do not have one please contact support@datatheorem.io.

In the directory containing the docker-compose.yml file, to start the services of the network traffic anaylzer run the following command:

BEARER_TOKEN=[DATA_THEOREM_API_PROTECT_API_KEY] docker-compose up -d

To verify the network traffic analyzer services have stated properly run the following command:

docker ps

If the services have started properly you should see something mostly the same as the following:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0ab415d0064e parser_service:latest "uvicorn main:app --…" 2 days ago Up 0 days 0.0.0.0:8081->8081/tcp, :::8081->8081/tcp ec2-user-ps-1 3eaf19e5f489 openapi_service:latest "python main.py" 2 days ago Up 0 days ec2-user-oas-1 8d3aeedb0ec8 threat_detection_service:latest "python main.py" 2 days ago Up 0 days ec2-user-tds-1 5e74a81c0b34 redis:alpine "docker-entrypoint.s…" 2 days ago Up 0 days 6379/tcp ec2-user-redis-1

Step 2: Update nginx configuration to setup traffic mirroring

Requirements:

  • python3

  • python3-pip

  • python3-nginx (pip install python-nginx)

To correctly configure the mirror nginx configuration files, some environment variables are required to be set. Using the network IP address or hostname from the physical host from step one, export a variable as such:

Replacing ${IP_ADDRESS_FROM_STEP_1} with the actual IP address or hostname from step 1.

To mirror an existing nginx configuration file, run the following command:

This will create a new file ${existing_nginx_conf}_mirror.

Save a copy of the original ${existing_nginx_conf} and replace it with the new ${existing_nginx_conf}_mirror.

Lastly, restart NGINX, all incoming requests should be forwarded.

Appendix

Nginx configuration file guide:

Typically, nginx configuration files are stored at:

  • /etc/nginx/sites-available

    • These are text files, usually one per domain name, possibly includes subdomains.

  • /etc/nginx/sites-enabled

    • These are symlinks to files in /etc/nginx/sites-available.

    • Only files in this directory are served via nginx.

Nginx configuration files may also be stored in

  • /etc/nginx/conf.d

If there are no files in the above directories the user is probably using the global nginx configuration file to host websites, this can usually be found at:

  • /etc/nginx/nginx.conf