Defying Murphy’s Law: Automated & Secure Off-Site Backups

Estimated reading time: 13 minutes

Making My LEMP Server Resilient

Building my Ultimate LEMP Server was a rewarding project. With the server running perfectly, my attention turned to my existing backup strategy. For years, I’ve had a robust disaster recovery plan in place, relying on snapshots, full VM/Container backups, and custom-built bash scripts like my WordPress backup script. It all has served me well for scheduled, nightly secure off-site backups.

However, as my workflow has evolved, I’ve consequently noticed a few friction points. While it may be true that my script is great for a full restore, it feels clunky for two increasingly common scenarios. For example, restoring a single file requires downloading a large, compressed archive, then extracting it, and after that, finding the file I need. Moreover, creating an “instant” backup before a major site change meant manually running the entire script. As a result, I wanted something better – something that provided truly flexible and secure off-site backups for not only my sites but, equally important, for the server’s entire personality – its configuration files.

My Checklist for an Even Better Secure Off-Site Backup Solution

With this in mind, as I set out to find a replacement, I defined an evolution of my requirements. Specifically, the new solution had to keep all the strengths of my old script while, at the same time, addressing its weaknesses. For this reason, my ideal backup solution had to be:

• Flexible: It should support different backends, whether it’s modern cloud object storage or a traditional self-hosted server via SSH.
• Granular & Fast Restores: The ability to browse backup history like a file system and restore a single file or directory instantly.
• Comprehensive: It must back up not only the website files and databases but also all critical server configuration files (Nginx, PHP, firewall, etc.).
• On-Demand Snapshots: The ability to run a quick, manual backup at any time with a simple command.
• Fully Automated: It still needed to run on a daily schedule without any manual intervention from me.
• Secure: This was non-negotiable. Encrypted bakups before they leave the server.
• Efficient: I still wanted a solution that used deduplication to save a massive amount of storage space and bandwidth over time.

The Solution: Restic for Secure Off-Site backups

After some research, the choice was clear: Restic. It’s a modern, open-source backup program that fits my philosophy perfectly. It’s written in Go, it’s a single, fast binary with no complex dependencies. More importantly, it has client-side encryption and content-defined deduplication built-in from the ground up.

The best part is its flexibility. It natively supports a variety of backends, from S3-compatible cloud services to a simple SSH connection. This meant I could choose the solution that best fit my needs. I’ll walk through setting up two of the most popular options: Backblaze B2 for cost-effective (in this case: free) cloud storage, and SSH/SFTP for a self-hosted approach.

Step 1: Setting Up the Restic Repository

First I need to install Restic on the LEMP server. Most of the commands in this blog post, with little adaptation, will work on most distributions. As I’m doing this on Alpine Linux, I need to install Restic with:

# Install Restic
sudo apk add restic

Restic is installed, now a repository needs to be initialized. I decided that I need the felxibility to choose between the two options below.

Option A: Using Cloud Storage (Backblaze B2)

For this method, I need to sign up for a free Backblaze B2 account, create a bucket (I’ve enabled “Default Encryption”, double encryption at least sounds cool and if Backblaze is doing it for me, no overhead there, right?), Add an Application Key, set “Allow List All Bucket Names”, save the Application Key ID, the Application Key, the bucket name and endpoint. For security, I stored these credentials in a secure file that only the root user can read.

# Create a secure file for our B2 credentials
sudo tee /root/.restic-b2.env > /dev/null <<'EOF'
export AWS_ACCESS_KEY_ID="Application_Key_ID"
export B2_ACCOUNT_KEY="applicationKey"
export RESTIC_REPOSITORY="b2:endppoint.backblazeb2.com/your-bucket-name"
export RESTIC_PASSWORD="a-very-strong-and-long-password"
EOF
# Set strict permissions on the file
sudo chmod 600 /root/.restic-b2.env
# Switch to root
sudo su
# Source the credentials and initialize the repository
source /root/.restic-b2.env
restic init
# Exit to the regular user
exit

Option B: Using a Secure SSH Connection (SFTP)

This method is perfect if you have another server (like a home NAS or a simple VPS) that you can connect to via SSH to store the secure off-site backups. First, we need to set up passwordless SSH key authentication.

On the backup server: Create a dedicated user and directory for backups.

# Run these commands on the backup server
sudo adduser -D restic-backup
sudo mkdir -p /var/backups/restic
sudo chown restic-backup:restic-backup /var/backups/restic

On the LEMP server: Generate a new, dedicated SSH key for the backup process.

# Run this on your LEMP server
ssh-keygen -t ed25519 -f /root/.ssh/restic_key -C "Restic Backup Key"

On the LEMP server: Copy the new public key to your backup server.

# Replace user@backup-server with your details
ssh-copy-id -i /root/.ssh/restic_key.pub restic-backup@your-backup-server-ip

Now, create the Restic environment file for the SFTP connection on the LEMP server.

# Create a secure file for our SFTP configuration
sudo tee /root/.restic-sftp.env > /dev/null <<'EOF'
export RESTIC_REPOSITORY="sftp:restic-backup@your-backup-server-ip/var/backups/restic"
export RESTIC_PASSWORD="a-very-strong-and-long-password"
EOF

# Set strict permissions on the file
sudo chmod 600 /root/.restic-sftp.env

# Source the credentials and initialize the repository
source /root/.restic-sftp.env
# Tell Restic to use our new SSH key
restic init -o --repo-sftp-command="ssh -i /root/.ssh/restic_key"

After choosing one of the options, a secure, off-site repository is ready to receive backups.

Step 2: Setting Up Email Notifications (Optional)

A silent backup is good, but a backup that reports its status is even better. I wanted to get a simple success or failure email every time the script runs. The easiest way to send emails on Alpine Linux is with ssmtp.

First, I installed ssmtp.

sudo apk add ssmtp

Then, I configured it by editing /etc/ssmtp/ssmtp.conf. You’ll need the SMTP details from your email provider for this step. For an example using Gmail, see the details below.

# Example ssmtp.conf for Gmail
#
# The user that gets all the mails for userids < 1000
[email protected]
# The mail server for Gmail
mailhub=smtp.gmail.com:587
# Where will the mail seem to come from?
rewriteDomain=gmail.com
# The full hostname of your server
hostname=your-server.com
# Use STARTTLS (required by Gmail)
UseTLS=YES
UseSTARTTLS=YES
# Username and password for your Gmail account
[email protected]
AuthPass=YOUR_GMAIL_APP_PASSWORD
# FromLineOverride=YES allows the script to set the From: address
FromLineOverride=YES

Step 3: Scripting the Automated and Secure Off-Site Backups

The core of this setup is a comprehensive and now configurable bash script. I created the script at /usr/local/bin/backup.sh. the below can be copied and pasted directly in the terminal after all variables have been adjusted. Download the script here from my GitHub.
To download it with curl, I do the following:

sudo curl -o /usr/local/bin/backup.sh https://raw.githubusercontent.com/ramonvanraaij/Scripts/main/linux/Alpine%20Linux/restic_backup.sh

The variables below, at the top of the script, need to be changed in the script, you can use your favorite editor to do that, vim or nano for example:

# --- Backup Method ---
# Choose your backup method: "B2" or "SFTP"
readonly BACKUP_METHOD="SFTP"

# --- SFTP Specific Configuration ---
# Only used if BACKUP_METHOD is "SFTP"
readonly SFTP_SSH_KEY="/root/.ssh/restic_key" # Path to the SSH private key for SFTP

# --- Email Notifications ---
readonly EMAIL_ENABLED="true"
readonly FROM_ADDRESS="LEMP Backup <[email protected]>"
readonly RECIPIENT_ADDRESS="[email protected]"

# --- Database Credentials ---
readonly DB_USER="root"
readonly DB_PASSWORD='strong_root_password'

# --- Paths & Retention ---
# A list of all files and directories to back up.
readonly PATHS_TO_BACKUP=(
    "/home/site1"
    "/home/site2"
    "/etc/nginx"
    "/etc/php83"
    "/etc/php84"
    "/var/spool/cron/crontabs"
    "/etc/apk/repositories"
    "/etc/letsencrypt"
    "/etc/ssh/sshd_config"
    "/etc/sshguard.conf"
    "/etc/nftables.nft"
    "/etc/logrotate.d"
    "/etc/periodic"
    "/root/scripts"
    "/usr/local/bin"
)
# Restic retention policy: How many snapshots to keep.
readonly KEEP_DAILY=7
readonly KEEP_WEEKLY=4
readonly KEEP_MONTHLY=12
readonly KEEP_YEARLY=3

# --- Environment Files ---
# Paths to the files containing your Restic repository credentials.
readonly B2_ENV_FILE="/root/.restic-b2.env"
readonly SFTP_ENV_FILE="/root/.restic-sftp.env"

After changing the variables, the script needs to have the executible flag set, but don’t use chmod +x! as the root DB password is in the script and we don’t want anybody but root to be able to read it!

sudo chmod 700 /usr/local/bin/backup.sh

Create the First of Many Secure Off-Site Backups

Now lets test if the script actually works!

sudo /usr/local/bin/backup.sh

Success! Lets check the Backblaze bucket in the browser.

Note that for me it took a few minutes before these showed in the browser.

Performing an Instant, On-Demand Backup

One of the best features of this setup is how easy it is to create instant, on-demand snapshots. This is incredibly useful before you perform a risky action, like editing a critical configuration file or running a major website update. To do this, you first need to load your repository credentials into your shell session. Then, simply run the restic backup command, pointing it to the specific file or directory you want to save. Adding a specific tag like --tag manual is a great practice, as it helps you easily distinguish these special snapshots from your regular automated backups.

#  Switch to root
sudo su
# Load your credentials (use the SFTP or B2 version as appropriate)
source /root/.restic-sftp.env

Back up a single critical file

# Omit the '-o sftp.command="ssh -i /root/.ssh/restic_key"' option if using the B2 backend
restic -o sftp.command="ssh -i /root/.ssh/restic_key" backup --tag manual /etc/nginx/nginx.conf

Or back up an entire directory before a big change

# Omit the '-o sftp.command="ssh -i /root/.ssh/restic_key"' option if using the B2 backend
restic -o sftp.command="ssh -i /root/.ssh/restic_key" backup --tag manual /home/site1/site1.com/html

The difference explained

The -o sftp.command="…" part in these examples is specific to the SFTP backend. It’s used to tell Restic how to establish the SSH connection, especially which private key to use.
When using the Backblaze B2 backend, Restic communicates directly with the B2 API over HTTPS and doesn’t use SSH at all. For authentication, it relies on the B2_ACCOUNT_ID and B2_ACCOUNT_KEY environment variables set in the /root/.restic-b2.env file.
So, for any manual commands using the B2 repository, simply omit that part.

Step 4: Creating Recovery Helpers

This seems to be unnecessary, as I make snapshots and full backups of my VM’s and containers. However, it is always nice to have yet another option when Murphy’s Law pays you a visit. For a full server rebuild, we need a way to restore the installed software and enabled services. These two simple helper scripts will be backed up along with everything else in /usr/local/bin.

Package Recovery Script

This script reads the backed-up package list and uses apk to install anything that’s missing.

sudo tee /usr/local/bin/recover_packages.sh > /dev/null <<'EOF'
#!/bin/bash
set -e
# Use 'find' to locate the package list file
PACKAGES_FILE=$(find /tmp/backups-* -name "packages.list" -type f | head -n 1)
# Check if the file was found
if [ -z "$PACKAGES_FILE" ]; then
    echo "ERROR: Package list not found in /tmp/backups-*" >&2
    exit 1
fi
echo "--- Installing required packages from list... ---"
# apk add is idempotent; it won't reinstall packages that are already present.
xargs apk add < "$PACKAGES_FILE"
echo "--- Package installation check complete. ---"
EOF
sudo chmod +x /usr/local/bin/recover_packages.sh

Service Recovery Script

This script reads the backed-up service list and re-enables any services that were set to start at boot across all their original runlevels. This is critical for ensuring services like nftables start correctly at boot time.

sudo tee /usr/local/bin/recover_services.sh > /dev/null <<'EOF'
#!/bin/bash
set -e
# Use 'find' to locate the service list file, as the directory name is dynamic
SERVICES_FILE=$(find /tmp/backups-* -name "services.list" -type f | head -n 1)
# Check if the file was found
if [ -z "$SERVICES_FILE" ]; then
    echo "ERROR: Services list not found in /tmp/backups-*" >&2
    exit 1
fi
echo "--- Enabling services for their respective runlevels... ---"
# Read the file line by line, skipping any empty lines or the header
# The format is: service_name | runlevel1 runlevel2 ...
while read -r service bar runlevels; do
    # Skip lines that don't have at least 3 fields (e.g., empty lines or headers)
    if [ -z "$runlevels" ]; then
        continue
    fi
    # Loop through each runlevel for the service (e.g., boot, default)
    for runlevel in $runlevels; do
        echo "Enabling service: $service for runlevel: $runlevel"
        rc-update add "$service" "$runlevel"
    done
done < <(grep -v '^$' "$SERVICES_FILE")
echo "--- Service restoration complete. ---"
EOF
sudo chmod +x /usr/local/bin/recover_services.sh

Step 5: Set it and Forget It with Cron

To automate the backup, I’ll use Alpine’s built-in cron system. Just symlink the main backup script to the daily cron directory. It will now run automatically every day.

# Symlink the script to the daily cron directory
sudo ln -s /usr/local/bin/backup.sh /etc/periodic/daily/run-restic-backup

Step 6: The Moment of Truth: Disaster Recovery Plan

The beauty of Restic is that the restore commands are (almost) identical regardless of the backend. Whether your data is on Backblaze B2 or your own SSH server, the process to get it back is the same.

Source the environment

First, source the correct environment file for the repository you want to access.

# Switch to root
sudo su
# If you used B2:
source /root/.restic-b2.env
# OR if you used SFTP:
source /root/.restic-sftp.env

Now you’re ready to test a few scenarios.

Scenario 1: Restore a Single File

Imagine I accidentally deleted my WordPress wp-config.php file. Instead of blindly restoring the latest version, Restic allows me to browse my backup history to find the exact version I need.

First, I’ll list all the available snapshots to see when they were created. This is useful for finding a version of the file from before a specific change was made.

# Omit the '-o sftp.command="ssh -i /root/.ssh/restic_key"' option if using the B2 backend
restic -o sftp.command="ssh -i /root/.ssh/restic_key" snapshots

This command produces a table showing each snapshot’s ID, the time it was taken, any tags, and the paths that were backed up. It might look something like this:

ID        Time                 Host              Tags          Paths                                   Size
------------------------------------------------------------------------------------------------------------------
a1b2c3d4  2025-09-06 12:03:22  lemp-server  automated,B2  /etc/logrotate.d                        300.724 MiB
                                                               /etc/nftables.nft
                                                               /etc/nginx
                                                               /etc/php83
                                                               /etc/php84
                                                               /etc/ssh/sshd_config
                                                               /home/site1
                                                               /tmp/backups-1757160198
                                                               /usr/local/bin
e5f6g7h8  2025-09-06 12:45:16  lemp-server  manual        /etc/nginx/nginx.conf                   4.867 KiB
i9j0k1l2  2025-09-06 12:53:04  lemp-server  manual        /home/site1/site1.com/html  278.829 MiB
------------------------------------------------------------------------------------------------------------------
3 snapshots

I can see I have an manual snapshot at 12:03 (e5f6g7h8) and another at 12:53 (i9j0k1l2). I want the version from before I made a change this afternoon, so I’ll inspect the 12:53 snapshot.

Next, I can list the files within that specific snapshot to confirm the path is correct. I’ll use the ID i9j0k1l2.

# List the contents of an older snapshot to find the exact file path
# Omit the '-o sftp.command="ssh -i /root/.ssh/restic_key"' option if using the B2 backend
restic -o sftp.command="ssh -i /root/.ssh/restic_key" ls i9j0k1l2 /home/site1/site1.com/html

This will show me all the files and directories in that path, confirming that wp-config.php exists.

Finally, I’ll restore the file using the chosen snapshot ID.

# Restore the file from the specific snapshot
# Omit the '-o sftp.command="ssh -i /root/.ssh/restic_key"' option if using
restic -o sftp.command="ssh -i /root/.ssh/restic_key" restore a1b2c3d4 --target /tmp/restore --include /home/site1/site1.com/html/wp-config.php 
# Check that the file is there
ls -l /home/site1/site1.com/html/wp-config.php

It worked perfectly. By browsing my snapshots, I could confidently restore the exact version of the file I needed. For restoring a directory, you can use the same command, just omit the “/wp-config.php” part at the end and the whole /home/site1/site1.com/html would be restored.

Scenario 2: Restore an Individual Database Dump

Similarly, if a database got corrupted, I could easily retrieve the latest SQL dump.

# First, find the exact filename of the dump from the latest snapshot
# Omit the '-o sftp.command="ssh -i /root/.ssh/restic_key"' option if using
restic -o sftp.command="ssh -i /root/.ssh/restic_key" find site1_db.sql
# Then, restore the specific file (replace with the actual filename from the command above) and also the "a1b2c3d4" after restore.
# Omit the '-o sftp.command="ssh -i /root/.ssh/restic_key"' option if using
restic -o sftp.command="ssh -i /root/.ssh/restic_key" restore a1b2c3d4 --target /tmp/restore --path "/tmp/backups/site1_db.sql"

With the SQL file restored to /tmp/restore, I could then import it back into MariaDB.

Scenario 3: Disaster Recovery Plan for a Full Server Rebuild

This is the ultimate test, and hopefully with my snapshots and full VM/Container backups, never needed. But, could I rebuild it from my secure off-site backups? The answer is yes! This detailed process shows how.

# First, Update & Upgrade 
sudo apk update && apk upgrade
# Install restic, nano, sudo, shadow, bash, curl and wget
sudo apk add restic nano sudo shadow bash curl wget 

# Switch to root
sudo su
# Source the env file to load your repository details
source /root/.restic-sftp.env
# Restore the entire /usr/local/bin and /etc directories
restic -o sftp.command="ssh -i /root/.ssh/restic_key" restore latest --target / --include /usr/local/bin --include /etc

sudo /usr/local/bin/recover_packages.sh

# Re-create the user for the first site
sudo adduser -D site1
# Re-create the user for the second site
sudo adduser -D site2
# Also, ensure the nginx user is in their groups so it can access the PHP-FPM socket
sudo usermod -a -G site1 nginx
sudo usermod -a -G site2 nginx
# Recreate nginx cache directories
sudo mkdir -p /var/cache/nginx/site1 && sudo chown nginx:nginx /var/cache/nginx/site1
sudo mkdir -p /var/cache/nginx/site2 && sudo chown nginx:nginx /var/cache/nginx/site2
# Recreate log directories
sudo mkdir -p /var/log/nginx/site1.com && sudo chown nginx:nginx /var/log/nginx/site1.com
sudo mkdir -p /var/log/nginx/site2.com && sudo chown nginx:nginx /var/log/nginx/site2.com
# Exit to the regular user
exit

# Still as root user, restore /etc, /home, and the temporary backup directory with our lists
restic -o sftp.command="ssh -i /root/.ssh/restic_key" restore latest --target / --include /home --include /tmp/backups

# Initialize and secure MariaDB
sudo mariadb-install-db --user=mysql --datadir=/var/lib/mysql
sudo rc-service mariadb start && sudo rc-update add mariadb
sudo mysql_secure_installation

# Example of recreating a database and user
mariadb -u root -p <<EOF
CREATE DATABASE site1_db;
CREATE USER 'site1_user'@'localhost' IDENTIFIED BY 'your-retrieved-password';
GRANT ALL PRIVILEGES ON site1_db.* TO 'site1_user'@'localhost';
FLUSH PRIVILEGES;
EOF

# Find the latest SQL dump for each database
SITE1_SQL_DUMP=$(find /tmp/backups-* -name "site1_db.sql" -print | sort -r | head -n 1)
SITE2_SQL_DUMP=$(find /tmp/backups-* -name "site2_db.sql" -print | sort -r | head -n 1)
# Import the dumps into the respective databases
mariadb -u root -p site1_db < "$SITE1_SQL_DUMP"
mariadb -u root -p site2_db < "$SITE2_SQL_DUMP"

sudo /usr/local/bin/recover_services.sh

sudo reboot

Step 7: Restic GUI

One of the items on my checlist was to have the ability to browse the backup history. It is achieved via the command line, however, to make it complete, I wanted to at least test one GUI. I tested restic-browser for a bit. This is a tool you would install on your laptop, not the Alpine LEMP server. So after installing it from the Arch Linux AUR on my laptop – i use arch btw 😉 – it just works. The only thing you need to do is having a copy of /root/.restic-b2.env in your user’s home on your laptop, source it and run it:

source ~/.restic-b2.env
restic-browser

Et voilà

There are also other GUI’s like npbackup and backrest, I might explore them later…

Final Thoughts: A Truly Resilient and Flexible Server

This project was the logical and necessary step after my server migration. By implementing comprehensive, secure off-site backups, I now have a fully automated, encrypted, and versioned history of my entire server stored safely.

The flexibility of Restic is its greatest strength. Whether you prefer the simplicity of cloud object storage or the control of a self-hosted SSH server, you can build a professional-grade backup system that requires almost zero maintenance. The server isn’t just a masterpiece anymore; it’s a fortress, and I have complete control over where the keys to that fortress are kept.

Buy me a coffee 🙂
If you found this post helpful, informative, or if it saved or made you some money, consider buying me a coffee. Your support means a lot and motivates me to keep writing.
You can do so via bunq.me (bunq, iDeal, Bankcontact and Credit- or Debit cards) or PayPal (PayPal and Credit- or Debit cards). Thank you!

Disclaimer
The blog posts, guides, and scripts provided on this website are for informational and educational purposes only. They are provided “as is” and without any warranty of any kind, either express or implied, including, but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement.
By using the information or scripts from this blog, you agree that I am not liable for any direct, indirect, incidental, consequential, or any other damages or losses arising from the use of or inability to use the information, scripts, or instructions contained herein. You assume full responsibility for any and all risks associated with the use of this content.
The blog posts, guides, and pages may contain referral/affiliate links. If you make a purchase through these links, I may receive a commission at no additional cost to you.