Skip to content

Data Warehouse (DWH)

Overview

The Blackwall Data Warehouse (DWH) stores and processes historical traffic and security event data collected by GateKeeper and the Detection layer. While GateKeeper and the Detection layer analyse incoming traffic and make real-time decisions about whether to allow or block requests, the Data Warehouse retains information about those decisions for later analysis. Importantly, the Data Warehouse can be hosted and managed by Blackwall or deployed within your own infrastructure. It enables you to:

  • Investigate previous security incidents.
  • Monitor traffic and attack trends over time.
  • Review blocked requests and security events.
  • Generate reports on domain activity.
  • Access historical statistics through the Blackwall Dashboard.
  • Use the Alerting System to receive notifications about security events.

How the Data Warehouse works

GateKeeper and the Detection layer prioritise real-time traffic processing. Security decisions must be made quickly, without waiting for historical data to be stored or analysed. The Data Warehouse operates separately from this process. When GateKeeper and the Detection layer process traffic:

  1. GateKeeper intercepts incoming requests.
  2. The Detection layer determines whether requests should be allowed or blocked.
  3. Traffic and security event information is sent asynchronously to the Data Warehouse.
  4. The Data Warehouse stores this information and makes it available for analysis, reporting, and other supported services.

Because the Data Warehouse operates outside the live traffic-processing path, delays or interruptions in data storage do not affect traffic filtering or security enforcement. The Data Warehouse maintains both individual event records and aggregated statistics. Individual records support detailed investigations, while aggregated statistics provide the data used by Dashboard graphs and reports.

The Data Warehouse stores several types of traffic and security information: - Request logs: Records of allowed and blocked requests. - Bot detection events: Information about identified bots and their classifications. - WAF events: Web Application Firewall matches and custom rule triggers. - Rate-limiting and DDoS events: Records of rate-limit triggers and Layer 7 DDoS mitigation activity. - SSL/TLS and performance metadata: Additional information associated with traffic processing.

This information supports historical analysis without increasing the processing load on GateKeeper or the Detection layer.

Alerting System

The Alerting System analyses information stored in the Data Warehouse to identify conditions that require attention and send security notifications to partners by email, webhook, Slack, Teams, etc. In addition to providing security visibility, these notifications can support partners offering Blackwall protection through free and paid service tiers.

For example, security notifications can help partners demonstrate the value of Blackwall's protection to customers using a free service, supporting opportunities to upgrade those customers to paid protection. The Alerting System requires a Partner On-Prem Data Warehouse deployment. Contact your Blackwall representative for information about enabling and configuring the Alerting System.

Deployment options

Blackwall supports two Data Warehouse deployment models, both which use the same underlying event structure and support access to historical traffic information.

  • Blackwall-managed (default deployment model) - Blackwall hosts and manages the Data Warehouse within its own cloud infrastructure. Partners do not need to provide or maintain hosting infrastructure. The Blackwall-managed deployment is suitable for partners who want access to historical traffic and security data without managing additional infrastructure. Blackwall is responsible for hosting and managing the Data Warehouse, including its data retention and access arrangements.

  • Partner On-Prem - The Data Warehouse is deployed within the partner's infrastructure, allowing partners to host their own traffic and security event data. The Partner On-Prem deployment is intended for hosting partners who require their traffic and security data to remain within their own infrastructure. This provides partners with greater control over where their historical event data is stored.

Alerting System availability

The Partner On-Prem deployment is currently the only deployment model that supports the Blackwall Alerting System.

Before you begin

Data Warehouse provisioning and installation are performed by Blackwall. You do not need to install the Data Warehouse software or modify your website or application. Before requesting a deployment, determine the following:

  1. System Requirements:
  2. Deployment location: Whether you want to use the default Blackwall-managed deployment or host the Data Warehouse within your own infrastructure.
  3. Historical data requirements: Whether you need to preserve existing security events and historical statistics, particularly when switching from a Blackwall-managed deployment.
  4. Alerting requirements: Whether you intend to use the Alerting System, which requires a Partner On-Prem deployment.
  5. Data access requirements: Which authorised personnel or systems need access to stored data.

If you select Partner On-Prem, coordinate the infrastructure requirements with Blackwall before deployment.

Set up a Data Warehouse

Blackwall provisions and configures Data Warehouse instances using its internal deployment processes. To request and establish a Data Warehouse deployment:

  1. Contact your Blackwall representative and request a Data Warehouse deployment.

  2. Specify your preferred deployment model:

    • Blackwall-managed: Blackwall hosts and manages your Data Warehouse.
    • Partner On-Prem: The Data Warehouse is hosted within your infrastructure.

  3. If you select Partner On-Prem, coordinate the deployment requirements and hosting environment with Blackwall. Confirm the appropriate hardware requirements for your deployment with a Blackwall Solutions Architect, as the requirements vary depending on the deployment size.

  4. Confirm whether you require the Alerting System. This feature is currently supported only with Partner On-Prem deployments.

  5. Blackwall provisions and configures the Data Warehouse for the selected environment.

  6. When provisioning is complete, obtain the applicable access details and credentials from Blackwall.

  7. Sign in to the Blackwall Dashboard and open the relevant statistics pages to verify that traffic and security event information is available.

  8. If you use an API or direct database connection to access stored data, verify access using the applicable connection details and credentials supplied by Blackwall.

Your Data Warehouse is ready for use when new traffic and security events are being recorded and are accessible through the configured interfaces.

Note

No changes to your protected website or application are required. Blackwall handles Data Warehouse provisioning and the necessary integration with its traffic-processing services.

Switch from Blackwall-managed to Partner On-Prem

Existing partners using a Blackwall-managed Data Warehouse can request a switch to a Partner On-Prem deployment. Switching to an on-premises deployment changes where new traffic and security events are stored.

Important: Historical data availability

Historical event data is not automatically transferred when switching from Blackwall-managed to Partner On-Prem by default. You must agree this before migration, so that the required data along with the intended retention policy is moved to the local DWH during migration. If you do not agree and plan this before migration, historical data will be lost.

When Blackwall redirects your GateKeeper engines to the on-premises Data Warehouse, new events are stored in the on-premises environment. Historical events previously stored in Blackwall's managed Data Warehouse will not automatically appear in the new deployment.

This can affect your ability to investigate previous incidents, review historical trends, or access information needed for reporting.

If you rely on historical data, contact Blackwall before switching to discuss whether the existing information can be migrated or otherwise preserved.

Do not proceed with the switch until you understand how access to your existing historical data will be affected.

On-Prem system requirements

For a Partner On-Prem Data Warehouse deployment, the recommended starting configuration is 8 vCPUs and 32 GB RAM, supporting approximately 10,000 domains and 3 billion logs per month. For larger deployments starting at 50,000 domains, we recommend 16 vCPUs and 64 GB RAM. Increasing the CPU allocation from 8 to 16 vCPUs is the recommended first step when additional processing capacity is required. These specifications are indicative and may be revised as further performance data becomes available. Contact Blackwall to confirm the appropriate hardware requirements for your deployment.

Making the switch to Partner On-Prem:

  1. Contact your Blackwall representative and request a switch from Blackwall-managed to Partner On-Prem.

  2. Review your historical data requirements. Identify whether you need to retain access to existing traffic logs, security events, or historical statistics.

  3. If historical data must be preserved, discuss the available migration or preservation arrangements with Blackwall. Confirm the agreed approach before proceeding.

  4. Coordinate the requirements for your on-premises environment with Blackwall.

  5. Blackwall provisions and configures the new Data Warehouse within your infrastructure.

  6. Blackwall redirects the relevant GateKeeper engines to the new Data Warehouse.

  7. After the switch, open the Blackwall Dashboard and verify that new traffic statistics and security events are available.

  8. If historical data migration was agreed, verify that the required historical information is accessible according to the agreed migration arrangements.

  9. If you intend to use the Alerting System, coordinate its enablement and configuration with Blackwall.

The switch is complete when new events are being recorded in the on-premises Data Warehouse and the required data access has been verified.

Note

Switching from Blackwall-managed to Partner On-Prem is supported. Switching from Partner On-Prem back to Blackwall-managed is not currently supported.

Data retention and security

Data retention

By default, the Data Warehouse retains event data for one year. This provides historical information for incident investigations, security analysis, and reporting without retaining records indefinitely. Events older than the configured retention period are removed according to the applicable retention policy.

Access security

Direct access to the Data Warehouse is protected using database credentials. Only authorised systems and personnel with valid credentials can query or export stored information. Blackwall provides the relevant access details when the Data Warehouse is provisioned.

Using your Data Warehouse

Once your Data Warehouse is operational, its stored information can support several activities.

Use case Description
Dashboard statistics Review traffic volumes, blocked requests, and security activity through the Blackwall Dashboard.
Incident investigation Examine historical events associated with attacks, traffic spikes, or other security incidents.
Trend analysis Identify changes in bot activity, attack patterns, or traffic volumes over time.
Monthly reporting Generate reports summarising domain traffic and security activity.
API access Retrieve stored information through supported API interfaces for further analysis or integration.
Alerting Generate security notifications based on stored events. Available only with Partner On-Prem deployments.

For information about the Alerting System, including its capabilities and configuration, refer to the dedicated Alerting System documentation or contact your Blackwall representative.

Feedback