Turvaserveri klasterdamine

Allpool olev juhend on konkreetne samm-sammuline kokkuvõte üldjuhendist: https://github.com/nordic-institute/X-Road/blob/master/doc/Manuals/LoadBalancing/ig-xlb_x-road_external_load_balancer_installation_guide.md

Pilt NIISi juhendist master-slave klastrist.

MASTER (osa käske vaja ka SLAVE peal teha; vaikimisi kõik on MASTER käsud):

1. Install the X-Road security server packages using the normal installation procedure or use an existing standalone node. (Soovitus kohe SLAVE peal ka X-Road tarkvara paigaldada, initsialiseerima ei pea)
2. Stop the xroad services.
   
   1. service xroad-* stop

3. Create a separate PostgreSQL instance for the serverconf database (see section 4. Database replication setup for details in GITHUB MANUAL, link above; use the newest PostgreSQL version if possible):

   1. Create replication keys for every node (on master node):

      openssl req -new -x509 -days 7300 -nodes -sha256 -out ca.crt -keyout ca.key -subj '/O=cluster/CN=CA'
      openssl req -new -nodes -days 7300 -keyout server.key -out server.csr -subj "/O=cluster/CN=master"
      openssl req -new -nodes -days 7300 -keyout server_slave1.key -out server_slave1.csr -subj "/O=cluster/CN=slave1"
      openssl x509 -req -in server.csr -CAcreateserial -CA ca.crt -CAkey ca.key -days 7300 -out server.crt
      openssl x509 -req -in server_slave1.csr -CAcreateserial -CA ca.crt -CAkey ca.key -days 7300 -out server_slave1.crt

      CODE

   2. kõigi node'de peal:

      sudo mkdir -p -m 0755 /etc/xroad/postgresql; sudo chmod o+x /etc/xroad

      CODE

   3. Copy ca.crt server.crt server.key to /etc/xroad/postgresql/

      # masteris
      cp ca.crt server.crt server.key /etc/xroad/postgresql/
      # igasse slave node-i masteri pealt: 
      scp ca.crt server_slave1.crt server_slave1.key domain-user@slave-ss-server:/tmp/
      # slave peal:
      cd /tmp; cp ca.crt server_slave1.crt server_slave1.key /etc/xroad/postgresql/
      # kõigis masinates
      sudo chown postgres:postgres /etc/xroad/postgresql/*; sudo chmod 400 /etc/xroad/postgresql/*

      CODE

   4. Master masinas:

      sudo -u postgres pg_createcluster -p 5433 14 serverconf
      
      # muuta alljärgnevad read failis:
      vi /etc/postgresql/14/serverconf/postgresql.conf
      
      ssl = on
      ssl_ca_file = '/etc/xroad/postgresql/ca.crt'
      ssl_cert_file = '/etc/xroad/postgresql/server.crt'
      ssl_key_file = '/etc/xroad/postgresql/server.key'
      
      listen_addresses = '*' 
      wal_level = replica
      max_wal_senders = 4 
      wal_keep_size = 8
      # wal_keep_segments = 10 (on Postgre 10 instead)
      
      
      # vi /etc/postgresql/14/serverconf/pg_hba.conf
      #lisada lõppu rida
      hostssl replication +slavenode samenet cert
      
      # käivitab uue eraldatud serverconf baasi ja loob sinna kasutaja (kasutajale vaja parool määrata): 
      systemctl start postgresql@14-serverconf
      
      sudo -u postgres psql -p 5433 -c "CREATE ROLE slavenode NOLOGIN";
      sudo -u postgres psql -p 5433 -c "CREATE USER slave1 REPLICATION PASSWORD NULL IN ROLE slavenode";
      sudo -u postgres psql -p 5433 -c "CREATE USER serverconf PASSWORD '<parool tuleb /etc/xroad/db.properties failist>'";
      
      # ainult remote baasi korral on vaja hoolitseda, et baasi loomiseks vajalik serverconf_admin kasutaja oleks olemas /etc/xroad.properties failis.
      sudo -u postgres psql -p 5433 -c "CREATE USER serverconf_admin PASSWORD '<parool tuleb /etc/xroad.properties failist>'";
      
      # jätkame uue serveconfi baasi loomisega
      sudo -u postgres pg_dump -C serverconf | sudo -u postgres psql -p 5433 -f -
      sudo -u postgres psql -p 5432 -c "ALTER DATABASE serverconf RENAME TO serverconf_old";

      CODE

   5. SLAVE peal:

      SLAVE:
      # sisesta käsud (viimane käsk on kaherealine):
      sudo -u postgres pg_createcluster -p 5433 14 serverconf;
      cd /var/lib/postgresql/14/serverconf/;
      rm -rf *;
      sudo -u postgres PGSSLMODE=verify-ca PGSSLROOTCERT=/etc/xroad/postgresql/ca.crt PGSSLCERT=/etc/xroad/postgresql/server_slave1.crt PGSSLKEY=/etc/xroad/postgresql/server_slave1.key pg_basebackup -h master-ss-server-IP-or-hostname -p 5433 -U slave1 -D .;
      
      # Vaja luua replikeerimise jaoks stanby.signal: 
      touch /var/lib/postgresql/14/serverconf/standby.signal
      chown postgres:postgres /var/lib/postgresql/14/serverconf/standby.signal
      chmod 0600 /var/lib/postgresql/14/standby.signal
      
      # muuta alljärgnevad read failis:
      vi /etc/postgresql/14/serverconf/postgresql.conf
      
      listen_addresses = 'localhost'
      ssl = on
      ssl_ca_file = '/etc/xroad/postgresql/ca.crt'
      ssl_cert_file = '/etc/xroad/postgresql/server_slave1.crt'
      ssl_key_file = '/etc/xroad/postgresql/server_slave1.key'
      
      # leave all WAL settings be default
      
      primary_conninfo = 'host=<master-ss-server-IP-or-hostname> port=5433 user=slave1 sslmode=verify-ca sslcert=/etc/xroad/postgresql/server.crt sslkey=/etc/xroad/postgresql/server.key sslrootcert=/etc/xroad/postgresql/ca.crt'
      
      hot_standby = on
      hot_standby_feedback = on
      
      
      # käivitada uus baas
      systemctl start postgresql@14-serverconf

      CODE
4. Change /etc/xroad/db.properties to point to the separate database instance (MASTER):
   
   1. vi /etc/xroad/db.properties
   2. serverconf.hibernate.connection.url : Change the url port number from 5432 to 5433
5. If you are using an already configured server as the master, the existing configuration was replicated to the slaves in step 3. Otherwise, proceed to configure the master server: install the configuration anchor, set up basic information, create authentication and signing keys and so on. See the security server installation guide [IG-SS] for help with the basic setup.

6. Set up the configuration file replication:

   adduser --system --shell /bin/bash --ingroup xroad xroad-slave
   sudo mkdir -m 755 -p /home/xroad-slave/.ssh && sudo touch /home/xroad-slave/.ssh/authorized_keys
   
   # SLAVE peal:
   su xroad
   ssh-keygen
   less ~/.ssh/id_rsa.pub
   
   # MASTER peal:
   vi /home/xroad-slave/.ssh/authorized_keys
   # kopeeri avanenud faili SLAVE-s olev id_rsa.pub sisu
   
   # SLAVE (ikka xroad kasutaja õigustes):
   #connect to the master host using ssh and accept the host key.
   ssh -i /var/lib/xroad/.ssh/id_rsa xroad-slave@master-ss-server-IP-or-hostname
   exit
   exit (tagasi root õigustesse)
   
   # SLAVE peal kopeeri allpool olevad read faili (sisesta õige master serveri aadress) 
   vi /etc/systemd/system/xroad-sync.service
   
   [Unit]
   Description=X-Road Sync Task
   After=network.target
   Before=xroad-proxy.service
   Before=xroad-signer.service
   Before=xroad-confclient.service
   Before=xroad-jetty.service
   [Service]
   User=xroad
   Group=xroad
   Type=oneshot
   Environment=XROAD_USER=xroad-slave
   Environment=MASTER=master-ss-server-IP-or-hostname
   ExecStartPre=/usr/bin/test ! -f /var/tmp/xroad/sync-disabled
   ExecStart=/usr/bin/rsync -e "ssh -o ConnectTimeout=5 " -aqz --timeout=10 --delete-delay --exclude db.properties --exclude "/conf.d/node.ini" --exclude "/gpghome" --exclude "*.tmp" --exclude "/postgresql" --exclude "/nginx" --exclude "/globalconf" --exclude "/jetty" --delay-updates --log-file=/var/log/xroad/slave-sync.log ${XROAD_USER}@${MASTER}:/etc/xroad/ /etc/xroad/
   [Install]
   WantedBy=multi-user.target
   WantedBy=xroad-proxy.service
   
   # SLAVE peal tee samamoodi veel failiga 
   vi /etc/systemd/system/xroad-sync.timer
   
   [Unit]
   Description=Sync X-Road configuration
   [Timer]
   OnBootSec=60
   OnUnitActiveSec=60
   [Install]
   WantedBy=timers.target
   
   # SLAVE peal taustatöö käivitamiseks:
   systemctl enable xroad-sync.timer xroad-sync.service
   systemctl start xroad-sync.timer
   
   # SLAVE peal loo fail /etc/logrotate.d/xroad-slave-sync alljärgneva sisuga:
   vi /etc/logrotate.d/xroad-slave-sync
   
   /var/log/xroad/slave-sync.log {
   	daily
   	rotate 7
   	missingok
   	compress
   	su xroad xroad
   	nocreate
   }

   CODE

7. Configure the node type as master in /etc/xroad/conf.d/node.ini (MASTER)

   # tekita fail /etc/xroad/conf.d/node.ini sisuga:
   [node]
   type=master
   
   #muuda faili õigused
   chown xroad:xroad /etc/xroad/conf.d/node.ini

   CODE

8. Disable support for client-side pooled connections (HTTP connection persistence) in /etc/xroad/conf.d/local.ini

   # vi /etc/xroad/conf.d/local.ini
   [proxy]
   server-support-clients-pooled-connections=false

   CODE

9. Seadista op-monitori teenus töötama ühes klastri masinas või eraldiseisvas masinas (https://www.x-tee.ee/docs/live/xroad/ug-ss_x-road_7_security_server_user_guide.html#1524-installing-an-external-operational-monitoring-daemon)

   1. Op-monitori masteriks valin hetkel master masina:

      # vi /etc/xroad/conf.d/local.ini
      [op-monitor]
      host = <masteri host aadress>

      CODE

   2. Peatan teenuse teistes masinates (minu näite puhul SLAVE-s):

      # on other (e.g. SLAVE) machines:
      service xroad-opmonitor stop
      systemctl disable xroad-opmonitor
      

      CODE
10. Start the X-Road services.
    
    1. service xroad-* start

SLAVE osa lisaks eespool tehtule:

1. Install security server packages using the normal installation procedure.
2. Stop the xroad services.
   1. service xroad-* stop
3. Create a separate PostgreSQL instance for the serverconf database (juba tehtud eespool)

4. Change /etc/xroad/db.properties to point to the separate database instance and change password to match the one defined in the master database

   # vi /etc/xroad/db.properties
   serverconf.hibernate.connection.url : Change the url port number from 5432 to 5433
   serverconf.hibernate.connection.password: Change to match the master db's password (vaata master masinast samast failist).

   CODE

5. Set up SSH between the master and the slave (juba tehtud)

6. Set up state synchronization using rsync+ssh (enamus tehtud, kuid lisaks):

   rsync -e "ssh -i /var/lib/xroad/.ssh/id_rsa" -avz --delete --exclude db.properties --exclude "/postgresql" --exclude "/conf.d/node.ini" --exclude "/gpghome" --exclude "/nginx" xroad-slave@master-ss-server-IP-or-hostname:/etc/xroad/ /etc/xroad/
   
   # if got an error "rsync: opendir "/etc/xroad/signer" failed: Permission denied (13)" then:
   chmod 750 /etc/xroad/signer/

   CODE

7. Configure the node type as slave in /etc/xroad/conf.d/node.ini

   # vi /etc/xroad/conf.d/node.ini
   [node]
   type=slave
   
   # muuda õigused
   chown xroad:xroad /etc/xroad/conf.d/node.ini

   CODE

8. Start the X-Road services.

   1. service xroad-* start
      
      

9. Selleks, et igaöine autobackup töötaks vaja slave õlal genereerida gpgvõtmed:

   sudo -i -u xroad /usr/share/xroad/scripts/generate_gpg_keypair.sh /etc/xroad/gpghome 'turvaserveri_pikk_id'#to get a Security Server's ID:/usr/share/xroad/scripts/get_security_server_id.sh

   CODE

Remote baasi korral on vaja hoolitseda, et baasi loomiseks vajalikud admin kontod (just serverconf_admin) oleksid määratud /etc/xroad.properties failis. Vastav õpetus juhendis: https://x-tee.ee/docs/live/xroad/ig-ss_x-road_v7_security_server_installation_guide.html#26-remote-database-setup-optional

Klasterdatud turviku õlas hooldusrežiimi aktiveerimine:

curl http://localhost:5566/maintenance?targetState=true

PS!
Kui muutub master või slave masina hostname, siis vaja genereerida uued ssh võtmed xroad-slave kasutaja jaoks. Muidu ei tööta xroad-sync.service sees olev rsync enam. (käsitsi rsync ssh -i töötab edasi)
Hostinime vahetus vaja teha ka /var/lib/postgresql/10/serverconf/recovery.conf sees.