The source of this step-by-step manual: https://github.com/nordic-institute/X-Road/blob/master/doc/Manuals/LoadBalancing/ig-xlb_x-road_external_load_balancer_installation_guide.md

load_balancing_state_replication.png

Illustrative picture from NIIS's full manual


MASTER security server setup (some commands must be done on the slave too):

  1. Install the X-Road security server packages using the normal installation procedure or use an existing standalone node.
  2. Stop the xroad services.
    1. service xroad-* stop

  3. Create a separate PostgreSQL instance for the serverconf database (see section 4. Database replication setup for details in GITHUB MANUAL, link above):

    1. Create replication keys for every node (1 slave in example):

      openssl req -new -x509 -days 7300 -nodes -sha256 -out ca.crt -keyout ca.key -subj '/O=cluster/CN=CA'
openssl req -new -nodes -days 7300 -keyout server.key -out server.csr -subj "/O=cluster/CN=master"
openssl req -new -nodes -days 7300 -keyout server_slave1.key -out server_slave1.csr -subj "/O=cluster/CN=slave1"
openssl x509 -req -in server.csr -CAcreateserial -CA ca.crt -CAkey ca.key -days 7300 -out server.crt
openssl x509 -req -in server_slave1.csr -CAcreateserial -CA ca.crt -CAkey ca.key -days 7300 -out server_slave1.crt
      CODE

    2. on all nodes:

      sudo mkdir -p -m 0755 /etc/xroad/postgresql; sudo chmod o+x /etc/xroad
      CODE

    3. Copy ca.crt server.crt server.key to /etc/xroad/postgresql/

      #on master
cp ca.crt server.crt server.key /etc/xroad/postgresql/
# copy to every slave node from master: 
scp ca.crt server_slave1.crt server_slave1.key [email protected]:/tmp/
# on slave server:
cd /tmp; cp ca.crt server_slave1.crt server_slave1.key /etc/xroad/postgresql/
# on all servers
sudo chown postgres:postgres /etc/xroad/postgresql/*; sudo chmod 400 /etc/xroad/postgresql/*
      CODE

    4. On master server:

      sudo -u postgres pg_createcluster -p 5433 10 serverconf

# make next changes in the file:
vi /etc/postgresql/10/serverconf/postgresql.conf

ssl = on
ssl_ca_file = '/etc/xroad/postgresql/ca.crt'
ssl_cert_file = '/etc/xroad/postgresql/server.crt'
ssl_key_file = '/etc/xroad/postgresql/server.key'

listen_addresses = '*' 
wal_level = replica
max_wal_senders = 4
wal_keep_segments = 10


# vi /etc/postgresql/10/serverconf/pg_hba.conf
#add to the end of the file:
hostssl replication +slavenode samenet cert

# start a new serverconf database (change the password):
systemctl start [email protected]
sudo -u postgres psql -p 5433 -c "CREATE ROLE slavenode NOLOGIN";
sudo -u postgres psql -p 5433 -c "CREATE USER slave1 REPLICATION PASSWORD NULL IN ROLE slavenode";
sudo -u postgres psql -p 5433 -c "CREATE USER serverconf PASSWORD '<take password from the master /etc/xroad/db_properties file>'";

# if using remote database, then be sure that serverconf_admin account is described in /etc/xroad.propterties file
sudo -u postgres psql -p 5433 -c "CREATE USER serverconf_admin PASSWORD '<password from /etx/xroad.properties file>'";

# continue with a new serverconf's database creation
sudo -u postgres pg_dump -C serverconf | sudo -u postgres psql -p 5433 -f -
sudo -u postgres psql -p 5432 -c "ALTER DATABASE serverconf RENAME TO serverconf_old";
      CODE

    5. on the SLAVE:

      SLAVE:
# execute commands (last is on two lines):
sudo -u postgres pg_createcluster -p 5433 10 serverconf;
cd /var/lib/postgresql/10/serverconf/;
rm -rf *;
sudo -u postgres PGSSLMODE=verify-ca PGSSLROOTCERT=/etc/xroad/postgresql/ca.crt PGSSLCERT=/etc/xroad/postgresql/server_slave1.crt PGSSLKEY=/etc/xroad/postgresql/server_slave1.key pg_basebackup -h master-ss-server-IP-or-hostname -p 5433 -U slave1 -D .;

# copy next lines to a file 
vi /var/lib/postgresql/10/serverconf/recovery.conf

standby_mode = 'on'
primary_conninfo = 'host=master-ss-server-IP-or-hostname port=5433 user=slave1 sslmode=verify-ca sslcert=/etc/xroad/postgresql/server_slave1.crt sslkey=/etc/xroad/postgresql/server_slave1.key sslrootcert=/etc/xroad/postgresql/ca.crt'
trigger_file = '/var/lib/xroad/postgresql.trigger'

# change permissions:
chown postgres:postgres recovery.conf; chmod 0600 recovery.conf

# change next lines in the file:
vi /etc/postgresql/10/serverconf/postgresql.conf

listen_addresses = 'localhost'
ssl = on
ssl_ca_file = '/etc/xroad/postgresql/ca.crt'
ssl_cert_file = '/etc/xroad/postgresql/server_slave1.crt'
ssl_key_file = '/etc/xroad/postgresql/server_slave1.key'
# no need to send WAL logs
wal_level = minimal
max_wal_senders = 0
# wal_keep_segments = 0

hot_standby = on
hot_standby_feedback = on

# start a new serverconf database
systemctl start [email protected]
      CODE
  4. Change /etc/xroad/db.properties to point to the separate database instance (MASTER):
    1. vi /etc/xroad/db.properties
    2. serverconf.hibernate.connection.url : Change the url port number from 5432 to 5433
  5. If you are using an already configured server as the master, the existing configuration was replicated to the slaves in step 3. Otherwise, proceed to configure the master server: install the configuration anchor, set up basic information, create authentication and signing keys and so on. See the security server installation guide [IG-SS] for help with the basic setup.

  6. Set up the configuration file replication:

    adduser --system --shell /bin/bash --ingroup xroad xroad-slave
sudo mkdir -m 755 -p /home/xroad-slave/.ssh && sudo touch /home/xroad-slave/.ssh/authorized_keys

# on SLAVE:
su xroad
ssh-keygen
less ~/.ssh/id_rsa.pub

# on MASTER:
vi /home/xroad-slave/.ssh/authorized_keys
# copy here lines from the SLAVE file id_rsa.pub (opened with previous less command)

# SLAVE (still under xroad user):
#connect to the master host using ssh and accept the host key.
ssh -i /var/lib/xroad/.ssh/id_rsa [email protected]
exit
exit (back to root user)

# on SLAVE server copy next lines to file (change host's address) /etc/systemd/system/xroad-sync.service:
[Unit]
Description=X-Road Sync Task
After=network.target
Before=xroad-proxy.service
Before=xroad-signer.service
Before=xroad-confclient.service
Before=xroad-jetty.service
[Service]
User=xroad
Group=xroad
Type=oneshot
Environment=XROAD_USER=xroad-slave
Environment=MASTER=master-ss-server-IP-or-hostname
ExecStartPre=/usr/bin/test ! -f /var/tmp/xroad/sync-disabled
ExecStart=/usr/bin/rsync -e "ssh -o ConnectTimeout=5 " -aqz --timeout=10 --delete-delay --exclude db.properties --exclude "/conf.d/node.ini" --exclude "*.tmp" --exclude "/postgresql" --exclude "/nginx" --exclude --exclude "/gpghome" "/globalconf" --exclude "/jetty" --delay-updates --log-file=/var/log/xroad/slave-sync.log ${XROAD_USER}@${MASTER}:/etc/xroad/ /etc/xroad/
[Install]
WantedBy=multi-user.target
WantedBy=xroad-proxy.service

# on SLAVE copy next lines to a file
vi /etc/systemd/system/xroad-sync.timer
[Unit]
Description=Sync X-Road configuration
[Timer]
OnBootSec=60
OnUnitActiveSec=60
[Install]
WantedBy=timers.target

# on SLAVE:
systemctl enable xroad-sync.timer xroad-sync.service
systemctl start xroad-sync.timer

# on SLAVE copy next lines to a file
vi /etc/logrotate.d/xroad-slave-sync
/var/log/xroad/slave-sync.log {
	daily
	rotate 7
	missingok
	compress
	su xroad xroad
	nocreate
}
    CODE

  7. Configure the node type as master in /etc/xroad/conf.d/node.ini (MASTER)

    # create a file /etc/xroad/conf.d/node.ini containing:
[node]
type=master

#change permissions
chown xroad:xroad /etc/xroad/conf.d/node.ini
    CODE

  8. Disable support for client-side pooled connections (HTTP connection persistence) in /etc/xroad/conf.d/local.ini

    # vi /etc/xroad/conf.d/local.ini
[proxy]
server-support-clients-pooled-connections=false
    CODE

  9. Setup op-monitor to work in one of the nodes (master in the example) (https://www.x-tee.ee/docs/live/xroad/ug-ss_x-road_7_security_server_user_guide.html#1524-installing-an-external-operational-monitoring-daemon)

    1. # vi /etc/xroad/conf.d/local.ini
[op-monitor]
host = <master's server address>
      CODE

    2. Stop op-monitor process on other nodes:

      # on other (e.g. SLAVE) machines:
service xroad-opmonitor stop
systemctl disable xroad-opmonitor
      CODE
  10. Start the X-Road services.
    1. service xroad-* start


SLAVE security server setup:

  1. Install security server packages using the normal installation procedure.
  2. Stop the xroad services.
    1. service xroad-* stop
  3. Create a separate PostgreSQL instance for the serverconf database (has been done already)

  4. Change /etc/xroad/db.properties to point to the separate database instance and change password to match the one defined in the master database

    # vi /etc/xroad/db.properties
serverconf.hibernate.connection.url : Change the url port number from 5432 to 5433
serverconf.hibernate.connection.password: Change to match the master db's password (take from master's server same file).
    CODE

  5. Set up SSH between the master and the slave (already done)

  6. Set up state synchronization using rsync+ssh (mostly done):

    rsync -e "ssh -i /var/lib/xroad/.ssh/id_rsa" -avz --delete --exclude db.properties --exclude "/postgresql" --exclude "/conf.d/node.ini" --exclude "/gpghome" --exclude "/nginx" [email protected]:/etc/xroad/ /etc/xroad/

# if got an error "rsync: opendir "/etc/xroad/signer" failed: Permission denied (13)" then:
chmod 750 /etc/xroad/signer/
    CODE

  7. Configure the node type as slave in /etc/xroad/conf.d/node.ini

    # vi /etc/xroad/conf.d/node.ini
[node]
type=slave

# change permissions
chown xroad:xroad /etc/xroad/conf.d/node.ini
    CODE

  8. Start the X-Road services.

    1. service xroad-* start

  9. For a nightly autobackup to work a new GPG keys must be created:

    sudo -i -u xroad /usr/share/xroad/scripts/generate_gpg_keypair.sh /etc/xroad/gpghome 'Security Server's full ID'
#to get a Security Server's ID:
/usr/share/xroad/scripts/get_security_server_id.sh
    CODE

If a Security Server uses remote database then the superuser credentials must be added before installing a software: https://x-tee.ee/docs/live/xroad/ig-ss_x-road_v7_security_server_installation_guide.html#26-remote-database-setup-optional