Training Platform Backup

Overview

The goal of this topic is to train you in the backup and restore of xAPPnow Low Code Platform. The platform administrator is the responsible for this task. For more details see Operations (Backup/Restore)

Exercise

In this exercise you will work with the three different ways to back up the databases of xAPPnow Low Code Platform:

Offline Backup
Online Backup with Log Shipping
Streaming Replication

Exercise 1: Offline Backup

Step 1: Stop xAPPnow services

./appollo-stop.sh <INSTANCE_NAME>

Parameter Explanation:

CONTAINER_NAME is the name of the database container. For appollo instance should be stack<INSTANCE_NAME>dbhost_1
Step 2: Execute backup command inside directory APPOLLO_HOME

./appollo-backup-create.sh <CONTAINER_NAME> <APPOLLO_BACKUP_DIR>

Step 3: Start xAPPnow services

./appollo-start.sh <INSTANCE_NAME>

Exercise 2: Restore Offline Backup

Step 1: Stop xAPPnow services

./appollo-stop.sh <INSTANCE_NAME>

Step 2: Execute restore command inside directory APPOLLO_HOME

./appollo-backup-restore.sh <CONTAINER_NAME> <APPOLLO_BACKUP_DIR>/<BACKUP_DATE>/stack<INSTANCE_NAME >_db-data.tar

Parameter Explanation:

BACKUP_DATE is the directory with the name of the backup date
Step 3: Start xAPPnow services

./appollo-start.sh <INSTANCE_NAME>

Exercise 3: Configure Log Shipping for Online Backup

Step 1: Configure database for log shipping

./appollo-configure-log-shipping.sh <CONTAINER_NAME> <SECONDS>

Parameter Explanation:

SECONDS is the number of seconds between backup update with the last transactions on the database

Exercise 4: Create Base Backup

Step 1: Execute backup command

./appollo-backup-online-create-base.sh <CONTAINER_NAME> <APPOLLO_BACKUP_DIR>

Exercise 5: Verify Base Backup

Step 1: Check the backup file was succesfully created

<APPOLLO_BACKUP_DIR>/base/<DATE>/<BACKUP_FILE>

Parameter Explanation:

DATE is the date when backup was done
BACKUP_FILE is the backup file name

Exercise 6: Create Incremental Backup

Step 1: Execute backup command

./appollo-backup-online-create-incremental.sh <CONTAINER_NAME> <APPOLLO_BACKUP_DIR>

Exercise 7: Verify Incremental Backup

Step 1: Check the backup file was succesfully created

<APPOLLO_BACKUP_DIR>/incremental/<DATE>_<BASE_DATE>/<BACKUP_FILE>

Parameter Explanation:

BASE_DATE is the date of a base backup corresponding to that incremental backup

Exercise 8: Backup Archiving

Step 1: Create backup archive directory

./appollo-backup-archive.sh <MOUNT_VOLUME_PATH> <BACKUP_DIRECTORY> <NUMBER_BACKUPS>

Parameter Explanation:

MOUNT_VOLUME_PATH is the instance mounted volume for backups. See 4.2 xAPPnow Installation, using same parameter.
NUMBER_BACKUPS is the number of backups to preserve.
BACKUP_DIRECTORY is directory to archive backups

Exercise 9: Protect backup

Step 1: Configure ssh

This steps needs to be run as root user.

sudo su

If the folder /root/.ssh does not exists, the keys can be generated using the command:

ssh-keygen -t rsa

Step 2: Allow host public ssh key on the remote server

ssh-copy-id -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no <REMOTE_USER>@<REMOTE_IP>

Parameter Explanation:

REMOTE_USER is the user used to accede the remote server
REMOTE_IP is the ip of the remote server

Step 3: Test ssh connection and accept remote host as known

ssh <REMOTE_USER>@<REMOTE_IP>

The authenticity of host '192.168.88.26 (192.168.88.26)' can't be established.

ECDSA key fingerprint is SHA256:Qdu59Dpmm8zhPPXP0tEd4uOHdGoPeemtORP+z8cbGyw.

Are you sure you want to continue connecting (yes/no)? yes

Remark: It should access to the remote server without password. After that exit the remote server with exit command.

Step 4: Sync archive backup directory to remote server

./appollo-sync-backup.sh <APPOLLO_HOME> <ARCHIVE_BACKUP_PATH> <REMOTE_USER REMOTE_IP> <REMOTE_PATH>

Parameter Explanation:

REMOTE_PATH is a directory path on remote server for backup volume directory

Remark: That directory must be created previously.

Exercise 10: Generate backup used for restoring

Step 1: Backup directory, it must be compressed on a backup file to be restored

./appollo-compress-backup-directory.sh <BACKUP_DIRECTORY> <BACKUP_FILE>

Parameter Explanation:

BACKUP_DIRECTORY is the folder where base backup and wall files are.
BACKUP_FILE is a compressed backup file to be restored

If generated backup task finished successfully there should be created a backup file, the output should be similar to this:

….

var/lib/postgresql/archive/wal/

var/lib/postgresql/archive/wal/000000010000000000000016.gz

var/lib/postgresql/archive/wal/000000010000000000000015.gz

var/lib/postgresql/archive/wal/000000010000000000000016.00000028.backup.gz

archive/

archive/repo/

archive/repo/wal_files.tar

archive/repo/base_2021-10-26_1222.tar.gz

Removing temporal directory /tmp/tmp.DJAYGc1RIw

Backup file created succesfully on path /home/appollo/online-backup-master.tar

Exercise 11: Restore Online Backup

Step 1: Stop xAPPnow services

./appollo-stop.sh <INSTANCE_NAME>

Step 2: Make offline backup of current database information (See Exercise 1 Step 2)

./appollo-backup-create.sh <CONTAINER_NAME> <APPOLLO_BACKUP_DIR>

Step 3: Restore online backup

./appollo-backup-online-restore.sh <CONTAINER_NAME> <APPOLLO_BACKUP_DIR>/<DATE>/incremental/DATE_base-DATE/<BACKUP_FILE>

Step 4: Restore online backup using point in time recovery

./appollo-backup-online-restore.sh <CONTAINER_NAME> <APPOLLO_BACKUP_DIR>/incremental/DATE/<BACKUP_FILE> “TIME”

Parameter Explanation:

TIME is a moment after the backup base and before incremental backup was created. The format is:

"YEAR-MONTH-DAY HOUR:MINUTE:SECOND"

Step 5: Start database container

docker start stack<INSTANCE_NAME>_dbhost_1

Step 6: Check database container log

docker logs -f stackdevelop_dbhost_1

In case of restoring using point in time recovery, execute the following command to resume database operations after recovery is paused, showing message “LOG: recovery has paused”

./appollo-backup-online-restore-resume.sh stackdevelop_dbhost_1

Step 7: Start xAPPnow services

./appollo-start.sh <INSTANCE_NAME>

Exercise 12: Configure Streaming Replication

Step 1: Configure master server for replication

./appollo-configure-streaming-replication-master.sh <CONTAINER_NAME> <SLAVE_IP>

Parameter Explanation:

CONTAINER_NAME is postgresql container.
SLAVE_IP is the ip address for slave server.

Step 2: Create master base backup (See Exercise 4 Create base backup)
Step 3: On slave server copy base backup from master server
Step 4: Stop xAPPnow services

./appollo-stop.sh <INSTANCE_NAME>

Step 5: Execute following command to setup replication on slave

./appollo-configure-streaming-replication-slave.sh <CONTAINER_NAME> <MASTER_IP> <PORT> <BASE_BACKUP>

Parameter Explanation:

MASTER_IP is the ip address for master server.
PORT is the port used for replication on the master server.
BASE_BACKUP is the base backup path.

Exercise 13: Verify Streaming Replication

Step 1: Execute the following command on slave server

./appollo-verify-replication-standby.sh <CONTAINER_NAME>

Step 2: Verify it looks like this

pg_is_in_recovery

-------------------

(1 row)

Exercise 14: Recovery after master fails

Step 1: Stop xAPPnow services on master server

./appollo-stop.sh <INSTANCE_NAME>

Step 2: Promote slave to master, execute the following command on slave server

./appollo-promote-master.sh <CONTAINER_NAME>

Exercise 15: Recover master server

Step 1: Restart xAPPnow services (Stop and Start xAPPnow services)
Step 2: Create base backup on slave server now working as master (See Exercise 4 Create Base Backup)
Step 3: Copy backup created on previous step to master server

On master server:

Step 4: Restore base backup created on slave server (See Exercise 8 Restore Online Backup)
Step 5: Start xAPPnow services

On slave server:

Repeat step to configure streaming replication for slave using the last base backup created on slave server so streaming replication Is ready again (See Exercise 9 Step 5 Configure slave server for replication)

If the system was configured to create incremental backups, this needs to be configured again (See Exercise 4 and Exercise 6 Create base and incremental backup)