Data Theorem Splunk App

Owned by John Gerlock
Last updated: Oct 22, 2024
3 min read

Overview

The Data Theorem Splunk application is a private Splunk App distributed by Data Theorem, Inc. for API Security. It it design to analyze Splunk logs for API attacks, API abuses, and API threats. It analyzes events as defined by the Splunk Common Information Model (CIM) add-on and sends the resulting access logs to Data Theorem for analysis. All customer data stays “On-Prem” or in your cloud, where only the metadata on events is shown on the Data Theorem portal. The data flow diagram is below:

 

splunk_arch.png

 

System Requirements

  • Splunk Enterprise version 9.x (Python 3.9 recommended)

  • Supported Operating System

    • Linux

  • Architecture:

    • x86_64

  • Dependencies

  • Supported Splunk Deployment Types

    • Standalone deployment: The app can be be deployed in a standalone Splunk instance, where Splunk performs both the search head and indexer roles.

    • Distributed deployment: In a distributed setup, where the search head is separate from indexers, ensure the app is deployed on the search head to perform the scheduled queries.

App Permissions

  • Access to run scheduled queries

  • Access to the indexes storing web-related data(tag='web').

  • Access to export the data, making REST API calls is required.

  • Access to write and read the API key that is stored in Splunk secrets store.

Data Protection

The Data Theorem Splunk App analyzes only indexes containing Web CIM data, and uses only the below list of fields from the Web CIM model. The Web CIM fields contain metdata about requests, similar to the information in an nginx or webserver access log. By using only defined fields on indexed Web CIM logs, there is minimal risk of accidental disclosure of sensitive data.

 

The Web CIM model include a`cookie` field that may contain HTTP Cookie values. We exclude this field and do not store or transmit any HTTP cookies.

 

Complete list of Web CIM fields

Dataset name

Field name

Data type

Description

Abbreviated list of example values

Dataset name

Field name

Data type

Description

Abbreviated list of example values

Web

action

string

The action taken by the server or proxy.

  • recommended

  • required for pytest-splunk-addon

Web

app

string

The application detected or hosted by the server/site such as WordPress, Splunk, or Facebook.

 

Web

bytes

number

The total number of bytes transferred (bytes_in + bytes_out).

  • recommended

  • required for pytest-splunk-addon

Web

bytes_in

number

The number of inbound bytes transferred.

  • recommended

  • required for pytest-splunk-addon

Web

bytes_out

number

The number of outbound bytes transferred.

  • recommended

  • required for pytest-splunk-addon

Web

cached

boolean

Indicates whether the event data is cached or not.

prescribed values:
true, false, 1, 0

Web

category

string

The category of traffic, such as may be provided by a proxy server.

required for pytest-splunk-addon

Web

dest

string

The destination of the network traffic (the remote host). You can alias this from more specific fields, such as dest_host, dest_ip, or dest_name.

  • recommended

  • required for pytest-splunk-addon

Web

dest_bunit

string

These fields are automatically provided by asset and identity correlation features of applications like Splunk Enterprise Security. Do not define extractions for these fields when writing add-ons.

Web

dest_category

string

Web

dest_priority

string

Web

dest_port

number

The destination port of the web traffic.

required for pytest-splunk-addon

Web

duration

number

The time taken by the proxy event, in milliseconds.

 

Web

http_content_type

string

The content-type of the requested HTTP resource.

recommended

Web

http_method

string

The HTTP method used in the request.

  • recommended

  • prescribed values:
    GET, PUT,POST, DELETE, HEAD, OPTIONS, CONNECT, TRACE

Web

http_referrer

string

The HTTP referrer used in the request. The W3C specification and many implementations misspell this as http_referer. Use a FIELDALIAS to handle both key names.

recommended

Web

http_referrer_domain

string

The domain name contained within the HTTP referrer used in the request.

recommended

Web

http_user_agent

string

The user agent used in the request.

  • recommended

  • required for pytest-splunk-addon

Web

http_user_agent_length

number

The length of the user agent used in the request.

required for pytest-splunk-addon

Web

response_time

number

The amount of time it took to receive a response, if applicable, in milliseconds.

 

Web

site

string

The virtual site which services the request, if applicable.

 

Web

src

string

The source of the network traffic (the client requesting the connection).

  • recommended

  • required for pytest-splunk-addon

Web

src_bunit

string

These fields are automatically provided by asset and identity correlation features of applications like Splunk Enterprise Security. Do not define extractions for these fields when writing add-ons.

Web

src_category

string

Web

src_priority

string

Web

status

string

The HTTP response code indicating the status of the proxy request.

  • recommended

  • required for pytest-splunk-addon

Web

tag

string

This automatically generated field is used to access tags from within datamodels. Do not define extractions for this field when writing add-ons.

 

Web

uri_path

string

The path of the resource served by the webserver or proxy.

other:
/CertEnroll/Blue%20Coat%20Systems
%20Internal.crl

Web

uri_query

string

The path of the resource requested by the client.

other:
?return_to=%2Fen-US%2Fapp%2Fsimple_xml_examples%2Fcustom_viz_
forcedirected%3Fearliest%3D0%26latest%3D

Web

url

string

The URL of the requested HTTP resource.

  • recommended

  • required for pytest-splunk-addon

  • other:
    http://0.channel36.facebook.com/x/1746719903/ false/p_1243021868=11

Web

url_domain

string

The domain name contained within the URL of the requested HTTP resource.

recommended

Web

url_length

number

The length of the URL.

 

Web

user

string

The user that requested the HTTP resource.

recommended

Web

user_bunit

string

These fields are automatically provided by asset and identity correlation features of applications like Splunk Enterprise Security. Do not define extractions for these fields when writing add-ons.

Web

user_category

string

Web

user_priority

string

Web

vendor_product

string

The vendor and product of the proxy server, such as Squid Proxy Server. This field can be automatically populated by vendor and product fields in your data.

recommended

Installation

 

  • Download the Data Theorem Splunk App from the Data Theorem portal

  • Copy the Data Theorem API Key

 

splunk.png

  • Install the app on Splunk deployment

  • When prompted, paste the API Key from the Data Theorem portal