Skip to content

Deploying Redis Sentinel for a Three-Server GateKeeper Cluster

Purpose

A GateKeeper cluster provides redundancy by running multiple GateKeeper servers together in the same cluster. A clustered deployment requires at least two GateKeeper servers, but Blackwall recommends that you configure three or more so that cluster coordination services, such as Redis Sentinel, can be configured reliably. The more GateKeeper servers included, the higher the overall throughput of the cluster. The cluster ensures that traffic continues to be processed, should one server become unavailable.

This guide describes how to configure a highly available Redis backend for three GateKeeper servers, where each server runs:

  • GateKeeper
  • Redis Server
  • Redis Sentinel
  • HAProxy, providing the local Redis endpoint 127.0.0.1:6380

GateKeeper doesn't connect directly to an individual Redis server. Instead, each GateKeeper instance uses its local HAProxy, as demonstrated below:

GateKeeper
    |
    v
127.0.0.1:6380
    |
    v
HAProxy
    |
    v
Current Redis master

Redis Sentinel monitors the Redis instances and if the master fails, it promotes one of the replicas to become the new master. HAProxy checks the actual role of each Redis instance and routes new connections only to the current master. Three Sentinel instances with a quorum of 2 allow automatic failover if one Redis node fails.

Tip

This guide provides Redis high availability, but it does not configure load balancing for public HTTP/HTTPS traffic across the GateKeeper servers. Public traffic requires a separate load balancer, DNS failover, or another ingress mechanism with GateKeeper health checks.

Example topology

The following addresses are used in the examples:

Server Hostname Private IP Components
GK1 gk1 10.114.0.11 GateKeeper, Redis, Sentinel, HAProxy
GK2 gk2 10.114.0.12 GateKeeper, Redis, Sentinel, HAProxy
GK3 gk3 10.114.0.13 GateKeeper, Redis, Sentinel, HAProxy

During the initial deployment:

  • GK1 is the initial Redis master;
  • GK2 is a Redis replica;
  • GK3 is a Redis replica;
  • the Sentinel quorum is 2;
  • GateKeeper accesses Redis through 127.0.0.1:6380.

After the first failover, no specific server should be treated as the permanent master. Redis Sentinel determines the current role.

Warning

After Redis Sentinel has been started, do not replace /etc/redis/sentinel.conf or restore it from the original template. Replacing the active file with the initial configuration may return Sentinel to an outdated view of the cluster.

Prerequisites

Before starting, make sure that:

  • all three servers are installed and accessible over SSH;
  • bidirectional network connectivity is available between the servers' private IP addresses;
  • the private IP addresses do not change after a reboot;
  • all servers maintain accurate time using NTP;
  • GateKeeper is already installed;
  • ports 6379/tcp and 26379/tcp are not accessible from the internet;

Prepare your servers

Create passwords

Two different passwords are required and the following placeholders are used throughout this guide:

  • <REDIS_PASSWORD> — for connecting to Redis;
  • <SENTINEL_PASSWORD> — for connecting to Redis Sentinel.

Warning

Do not put real passwords in shell history, tickets, public documentation, or version control. For the command examples below, load Redis CLI passwords from local root-only files instead of typing passwords directly in shell commands. Once generated, store the generated values in a secure secrets-management system.

To simplify HAProxy health checks, use passwords without spaces, quotation marks, backslashes, or newline characters. Hexadecimal values are suitable.

  1. Generate both passwords on a secure administrative machine using the following commands. 
    openssl rand -hex 32
openssl rand -hex 32
  2. Using the following commands on each of the three GateKeeper servers, create credential helper files, which make administrative commands safer and easier. 
    sudo install -o root -g root -m 0600 /dev/null /root/.redis-cli-redis.env
sudo install -o root -g root -m 0600 /dev/null /root/.redis-cli-sentinel.env
  3. Open the created Redis CLI password file: 
    sudoedit /root/.redis-cli-redis.env
  4. Add your previously generated password to the open file: 
    export REDISCLI_AUTH='<REDIS_PASSWORD>'
  5. Open the created Sentinel CLI password file: 
    sudoedit /root/.redis-cli-sentinel.env
  6. Add your previously generated password to the open file:: 
    export REDISCLI_AUTH='<SENTINEL_PASSWORD>'

Install packages

  1. Run the following commands on GK1, GK2, and GK3: 
    sudo apt update

sudo DEBIAN_FRONTEND=noninteractive apt install -y \
redis-server \
redis-sentinel \
redis-tools \
haproxy
  2. Check the installed versions: 
    redis-server --version
redis-sentinel --version
haproxy -v
iptables --version

  3. Create backup copies of the configuration files: 

    sudo cp -a /etc/redis/redis.conf \
/etc/redis/redis.conf.before-gk-cluster

sudo cp -a /etc/redis/sentinel.conf \
/etc/redis/sentinel.conf.before-gk-cluster 2>/dev/null || true

sudo cp -a /etc/haproxy/haproxy.cfg \
/etc/haproxy/haproxy.cfg.before-gk-cluster

  4. Stop Redis Sentinel until the initial configuration is complete: 

    sudo systemctl stop redis-sentinel

Configure iptables

The following connections must be allowed between the GateKeeper servers:

Port Purpose
6379/tcp Redis replication and Redis management by Sentinel
26379/tcp communication between Sentinel instances

Port 6380/tcp must listen only on 127.0.0.1 and must not be opened in iptables.

  1. Run the following commands on GK1: 

    sudo iptables -I INPUT -p tcp -s 10.114.0.12 -d 10.114.0.11 --dport 6379 -j ACCEPT
sudo iptables -I INPUT -p tcp -s 10.114.0.13 -d 10.114.0.11 --dport 6379 -j ACCEPT

sudo iptables -I INPUT -p tcp -s 10.114.0.12 -d 10.114.0.11 --dport 26379 -j ACCEPT
sudo iptables -I INPUT -p tcp -s 10.114.0.13 -d 10.114.0.11 --dport 26379 -j ACCEPT

  2. Run the following commands on GK2: 

    sudo iptables -I INPUT -p tcp -s 10.114.0.11 -d 10.114.0.12 --dport 6379 -j ACCEPT
sudo iptables -I INPUT -p tcp -s 10.114.0.13 -d 10.114.0.12 --dport 6379 -j ACCEPT

sudo iptables -I INPUT -p tcp -s 10.114.0.11 -d 10.114.0.12 --dport 26379 -j ACCEPT
sudo iptables -I INPUT -p tcp -s 10.114.0.13 -d 10.114.0.12 --dport 26379 -j ACCEPT

  3. Run the following commands on GK3: 

    sudo iptables -I INPUT -p tcp -s 10.114.0.11 -d 10.114.0.13 --dport 6379 -j ACCEPT
sudo iptables -I INPUT -p tcp -s 10.114.0.12 -d 10.114.0.13 --dport 6379 -j ACCEPT

sudo iptables -I INPUT -p tcp -s 10.114.0.11 -d 10.114.0.13 --dport 26379 -j ACCEPT
sudo iptables -I INPUT -p tcp -s 10.114.0.12 -d 10.114.0.13 --dport 26379 -j ACCEPT

  4. Review and persist the rules on each server: 

    sudo iptables -S INPUT
sudo netfilter-persistent save

Configure Redis

Configure the Redis instances

On each server, open and amend the Redis configuration file.

Warning

Find the existing directives and replace them. Do not leave multiple active versions of the same directive.

  1. Open the Redis configuration file pn all three GateKeeper servers: 
    sudoedit /etc/redis/redis.conf

  2. For the open Redis configuration file on GK1, ensure that the following directives are active: 

    bind 127.0.0.1 10.114.0.11
protected-mode yes
port 6379

pidfile "/run/redis/redis-server.pid"
logfile "/var/log/redis/redis-server.log"
dir "/var/lib/redis"

requirepass <REDIS_PASSWORD>
masterauth <REDIS_PASSWORD>

appendonly yes
appendfsync everysec

replica-read-only yes
replica-priority 100
    During the initial deployment, do not add a replicaof directive on GK1.

  3. For the open Redis configuration file on GK2, ensure that the following directives are active: 

    bind 127.0.0.1 10.114.0.12
protected-mode yes
port 6379

pidfile "/run/redis/redis-server.pid"
logfile "/var/log/redis/redis-server.log"
dir "/var/lib/redis"

requirepass <REDIS_PASSWORD>
masterauth <REDIS_PASSWORD>

appendonly yes
appendfsync everysec

replica-read-only yes
replica-priority 100

replicaof 10.114.0.11 6379

  4. For the open Redis configuration file on GK3, ensure that the following directives are active: 
    bind 127.0.0.1 10.114.0.13
protected-mode yes
port 6379

pidfile "/run/redis/redis-server.pid"
logfile "/var/log/redis/redis-server.log"
dir "/var/lib/redis"

requirepass <REDIS_PASSWORD>
masterauth <REDIS_PASSWORD>

appendonly yes
appendfsync everysec

replica-read-only yes
replica-priority 100

replicaof 10.114.0.11 6379

Allow Redis configuration changes

Redis Sentinel changes the roles of Redis instances during failover. Redis must be able to persist the updated configuration by using CONFIG REWRITE.

  1. Run the following commands on all three servers: 
    sudo chown root:redis /etc/redis/redis.conf
sudo chmod 0660 /etc/redis/redis.conf
  2. On all three servers, verify the owner and permissions: 
    stat -c '%U %G %a %n' /etc/redis
stat -c '%U %G %a %n' /etc/redis/redis.conf
  3. Check that the /etc/redis directory is owned by root, and the redis user is not able to create or delete files in it.
  4. Check that the /etc/redis/redis.conf file is writable by the redis group.

Start the initial Redis master

  1. On GK1, enable the Redis service to start automatically, restart it to apply the configuration, and verify that it is running, using the following commands: 
    sudo systemctl enable redis-server
sudo systemctl restart redis-server
sudo systemctl is-active redis-server

  2. Test your connection, using: 

    . /root/.redis-cli-redis.env
redis-cli -h 127.0.0.1 -p 6379 PING

    Expected result: 

    PONG

  3. Check the role, using: 

    . /root/.redis-cli-redis.env
redis-cli -h 127.0.0.1 -p 6379 ROLE

    Expected result from GK1: 

    master
    4. Verify that the configuration can be persisted: 
    . /root/.redis-cli-redis.env
redis-cli -h 127.0.0.1 -p 6379 CONFIG REWRITE

    Expected result:

    OK

Start the Redis Replicas

  1. On GK2 and GK3, enable the Redis service to start automatically, restart it to apply the configuration, and verify that it is running: 
    sudo systemctl enable redis-server
sudo systemctl restart redis-server
sudo systemctl is-active redis-server

  2. On both servers, check the replication status: 

    . /root/.redis-cli-redis.env
redis-cli -h 127.0.0.1 -p 6379 INFO replication

    Expected parameters: 

    role:slave
master_host:10.114.0.11
master_port:6379
master_link_status:up

    Depending on the Redis version, secondary nodes may be referred to as a replica, although the role field in the INFO replication output may still return slave.

  3. On GK1, check the number of connected replicas: 

    . /root/.redis-cli-redis.env
redis-cli -h 127.0.0.1 -p 6379 INFO replication

    Expected parameters: 

    role:master
connected_slaves:2

  4. Do not proceed to the Sentinel configuration until both replicas show: 

    master_link_status:up

Verify successful data replication

  1. For testing purposes, create a temporary key on GK1: 
    . /root/.redis-cli-redis.env
redis-cli -h 127.0.0.1 -p 6379 \
SET gk-cluster-bootstrap-check "$(date -Is)"
  2. Check that the key created in the previous step exists on GK2 and GK3: 
    . /root/.redis-cli-redis.env
redis-cli -h 127.0.0.1 -p 6379 \
GET gk-cluster-bootstrap-check
  3. Ensure that both replicas return the same value.
  4. Delete the temporary key on GK1: 
    . /root/.redis-cli-redis.env
redis-cli -h 127.0.0.1 -p 6379 \
DEL gk-cluster-bootstrap-check

Configure Redis Sentinel

Create configuration

  1. Create a separate /etc/redis/sentinel.conf file on GK1, GK2, and GK3.
  2. On each of the three servers, open the file created in the previous step: 
    sudoedit /etc/redis/sentinel.conf

  3. Make the following amendments on each of the three servers:

    port 26379
bind 127.0.0.1 10.114.0.11
protected-mode yes

requirepass <SENTINEL_PASSWORD>

sentinel monitor mymaster 10.114.0.11 6379 2
sentinel auth-pass mymaster <REDIS_PASSWORD>

sentinel down-after-milliseconds mymaster 10000
sentinel failover-timeout mymaster 60000
sentinel parallel-syncs mymaster 1

    port 26379
bind 127.0.0.1 10.114.0.12
protected-mode yes

requirepass <SENTINEL_PASSWORD>

sentinel monitor mymaster 10.114.0.11 6379 2
sentinel auth-pass mymaster <REDIS_PASSWORD>

sentinel down-after-milliseconds mymaster 10000
sentinel failover-timeout mymaster 60000
sentinel parallel-syncs mymaster 1

    port 26379
bind 127.0.0.1 10.114.0.13
protected-mode yes

requirepass <SENTINEL_PASSWORD>

sentinel monitor mymaster 10.114.0.11 6379 2
sentinel auth-pass mymaster <REDIS_PASSWORD>

sentinel down-after-milliseconds mymaster 10000
sentinel failover-timeout mymaster 60000
sentinel parallel-syncs mymaster 1

    The same SENTINEL_PASSWORD must be used by all three Sentinel instances.

  4. Set the permissions on each server: 

    sudo chown redis:redis /etc/redis/sentinel.conf
sudo chmod 0640 /etc/redis/sentinel.conf

Start Sentinel sequentially

  1. Start Sentinel on GK1 first: 

    sudo systemctl enable redis-sentinel
sudo systemctl restart redis-sentinel
sudo systemctl is-active redis-sentinel

  2. Check the current master: 

    . /root/.redis-cli-sentinel.env
redis-cli -h 127.0.0.1 -p 26379 \
SENTINEL get-master-addr-by-name mymaster

    Expected result:

    10.114.0.11
6379

  3. Then start Sentinel on GK2: 

    sudo systemctl enable redis-sentinel
sudo systemctl restart redis-sentinel
sudo systemctl is-active redis-sentinel

  4. Wait until GK1 and GK2 discover each other. Then start Sentinel on GK3: 

    sudo systemctl enable redis-sentinel
sudo systemctl restart redis-sentinel
sudo systemctl is-active redis-sentinel

Verifying Sentinel deployment

  1. Run the following command on each server and check that the result confirms that a quorum and a majority sufficient for failover are available. 

    . /root/.redis-cli-sentinel.env
redis-cli -h 127.0.0.1 -p 26379 \
SENTINEL CKQUORUM mymaster

  2. Check the current master: 

    . /root/.redis-cli-sentinel.env
redis-cli -h 127.0.0.1 -p 26379 \
SENTINEL master mymaster

  3. Check the discovered replicas: 

    . /root/.redis-cli-sentinel.env
redis-cli -h 127.0.0.1 -p 26379 \
SENTINEL replicas mymaster

  4. Check the discovered Sentinel instances: 

    . /root/.redis-cli-sentinel.env
redis-cli -h 127.0.0.1 -p 26379 \
SENTINEL sentinels mymaster

Make sure that each Sentinel sees:

  • one Redis master
  • two Redis replicas
  • the other two Sentinel instances
  • a quorum of 2

Configure HAProxy

HAProxy must listen for the Redis endpoint only on the local address 127.0.0.1:6380. It must then:

  1. connect to each Redis instance;
  2. authenticate with Redis;
  3. request replication information;
  4. consider only the server with the master role to be available;
  5. close existing connections if a server is no longer the master.

Configure the local Redis endpoint

To configure HAProxy:

  1. Open the HAProxy configuration file on each server: 

    sudoedit /etc/haproxy/haproxy.cfg

  2. Use the following configuration: 

    global
    log /dev/log local0
    log /dev/log local1 notice
    user haproxy
    group haproxy
    daemon
    maxconn 4096

defaults
    log global
    mode tcp
    retries 3
    timeout connect 3s
    timeout client 1m
    timeout server 1m

frontend redis_local
    bind 127.0.0.1:6380
    mode tcp
    default_backend redis_current_master

backend redis_current_master
    mode tcp
    option tcp-check

    tcp-check connect
    tcp-check send AUTH\ <REDIS_PASSWORD>\r\n
    tcp-check expect string +OK

    tcp-check send INFO\ replication\r\n
    tcp-check expect string role:master

    tcp-check send QUIT\r\n
    tcp-check expect string +OK

    server gk1 10.114.0.11:6379 check inter 2s fall 2 rise 1 on-marked-down shutdown-sessions
    server gk2 10.114.0.12:6379 check inter 2s fall 2 rise 1 on-marked-down shutdown-sessions
    server gk3 10.114.0.13:6379 check inter 2s fall 2 rise 1 on-marked-down shutdown-sessions

  3. Restrict access to the configuration file because it contains the Redis password: 

    sudo chown root:haproxy /etc/haproxy/haproxy.cfg
sudo chmod 0640 /etc/haproxy/haproxy.cfg

  4. Validate the syntax: 

    sudo haproxy -c -f /etc/haproxy/haproxy.cfg

  5. If the configuration is valid, enable and start HAProxy: 

    sudo systemctl enable haproxy
sudo systemctl restart haproxy
sudo systemctl is-active haproxy

Verify local HAProxy endpoint

  1. Run the following command on each of the three GateKeeper servers: 

    . /root/.redis-cli-redis.env
redis-cli -h 127.0.0.1 -p 6380 PING

    Expected result: 

    PONG

  2. Check the role of the Redis server selected by HAProxy: 

    . /root/.redis-cli-redis.env
redis-cli -h 127.0.0.1 -p 6380 ROLE

    The result must begin with:

    master

  3. Test writing and reading through HAProxy: 

    . /root/.redis-cli-redis.env
redis-cli -h 127.0.0.1 -p 6380 \
SET gk-haproxy-check "$(hostname)-$(date -Is)"

redis-cli -h 127.0.0.1 -p 6380 \
GET gk-haproxy-check

redis-cli -h 127.0.0.1 -p 6380 \
DEL gk-haproxy-check

  4. Check the listener: 

    sudo ss -lntp | grep ':6380'

    The endpoint must listen only on: 

    127.0.0.1:6380

    It must not listen on 0.0.0.0:6380 or on the server's private IP address.

Configure GateKeeper

  1. Open the following file on each GateKeeper server: 

    sudoedit /etc/default/botguard

  2. Add or update: 

    REDIS_HOST=127.0.0.1:6380
REDIS_PASSWORD=<REDIS_PASSWORD>

  3. Make sure the file does not contain multiple active REDIS_HOST or REDIS_PASSWORD lines.

  4. Check the available GateKeeper services: 

    systemctl list-unit-files | grep -E '^(botguard|gatekeeper)'

  5. Restart GateKeeper: 

    sudo systemctl restart botguard-controller
sudo systemctl restart botguard-apiserver
sudo systemctl restart botguard-certmanager
sudo systemctl restart botguard-webapp

  6. Check the status: 

    sudo systemctl is-active botguard-controller
sudo systemctl is-active botguard-apiserver
sudo systemctl is-active botguard-certmanager
sudo systemctl is-active botguard-webapp

  7. All expected services must start successfully.

Verify the cluster

  1. On GK1, run the following command: 

    . /root/.redis-cli-redis.env
redis-cli -h 127.0.0.1 -p 6379 \
INFO replication | grep -E 'role:|connected_slaves:'

    Initially, the expected output is:

    role:master
connected_slaves:2

  2. On GK2 and GK3, run the following command: 

    . /root/.redis-cli-redis.env
redis-cli -h 127.0.0.1 -p 6379 \
INFO replication | grep -E 'role:|master_host:|master_link_status:'

    Initially, the expected output is:

    role:slave
master_host:10.114.0.11
master_link_status:up

  3. On all three servers, verify the local Redis endpoint: 

    . /root/.redis-cli-redis.env
redis-cli -h 127.0.0.1 -p 6380 PING

    Expected result:

    PONG

  4. On all three servers, verify the current master through Sentinel: 

    . /root/.redis-cli-sentinel.env
redis-cli -h 127.0.0.1 -p 26379 \
SENTINEL get-master-addr-by-name mymaster

    All three Sentinel instances must return the same master address.

  5. On all three servers, verify the quorum: 

    . /root/.redis-cli-sentinel.env
redis-cli -h 127.0.0.1 -p 26379 \
SENTINEL CKQUORUM mymaster

  6. Verify the services 

    systemctl status \
redis-server \
redis-sentinel \
haproxy \
botguard-controller \
botguard-apiserver \
--no-pager

Test automatic failover

This tests may be performed during an approved maintenance window to check automatic failover success.

  1. Identifying the Current Master

Run the following command on any server:

. /root/.redis-cli-sentinel.env
redis-cli -h 127.0.0.1 -p 26379 \
SENTINEL get-master-addr-by-name mymaster

With the initial configuration, the result should point to:

10.114.0.11
6379

  1. Make a test value by creating a key through the local HAProxy instance: 

    . /root/.redis-cli-redis.env
redis-cli -h 127.0.0.1 -p 6380 \
SET gk-failover-check before-failover

  2. Stop the current Redis master server. If GK1 is the current master, run the following command on GK1: 

    sudo systemctl stop redis-server

    Do not stop Sentinel when testing only a Redis process failure.

  3. Verify that a new master is selected. Run the following command on GK2 or GK3: 

    watch -n 1 \
'. /root/.redis-cli-sentinel.env && redis-cli -h 127.0.0.1 -p 26379 SENTINEL get-master-addr-by-name mymaster'

    If failover is completed successfully, the returned address must change to:

    10.114.0.12

    or:

    10.114.0.13

  4. Check the local HAProxy endpoint: 

    . /root/.redis-cli-redis.env
redis-cli -h 127.0.0.1 -p 6380 PING

    If failover is completed successfully, the expected result is:

    PONG

  5. Check the existing key: 

    . /root/.redis-cli-redis.env
redis-cli -h 127.0.0.1 -p 6380 \
GET gk-failover-check

    Verify that writes are possible:

    . /root/.redis-cli-redis.env
redis-cli -h 127.0.0.1 -p 6380 \
SET gk-failover-check after-failover

  6. Bring the stopped Redis instance back online by running the following commands on the stopped server: 

    sudo systemctl start redis-server
sudo systemctl is-active redis-server

    Do not manually assign the recovered server as the master. Sentinel must automatically reconfigure the recovered former master as a replica of the current master.

  7. Check the role of the recovered node: 

    . /root/.redis-cli-redis.env
redis-cli -h 127.0.0.1 -p 6379 \
INFO replication | grep -E 'role:|master_host:|master_link_status:'

    Expected output:

    role:slave
master_link_status:up

    Verify that all three Sentinel instances point to the same master.

  8. After the failover test is completed successfully, reboot each node one at a time using the following sequence:

    1. Make sure that SENTINEL CKQUORUM mymaster completes successfully.
    2. Reboot one Redis replica.
    3. Wait for it to return.
    4. Make sure that the replica shows master_link_status:up.
    5. Check the Sentinel quorum again.
    6. Only then proceed to the next node and repeat these steps.

    Do not reboot two Sentinel instances at the same time.

  9. After rebooting each server, run: 

    sudo systemctl is-active redis-server
sudo systemctl is-active redis-sentinel
sudo systemctl is-active haproxy

    Check the local endpoint:

    . /root/.redis-cli-redis.env
redis-cli -h 127.0.0.1 -p 6380 PING

    Expected result:

    PONG
Feedback