Create Server Backup from Local Machine with Borg

How to back up your servers with borg? One way to do it, as documented by the fine borg documentation, is to use a pull backup using sshfs. The idea is to mount the remote filesystem in a local directory, chroot into it, and run borg inside the chroot.

Why the chroot?

Borg stores the owner and group. Only the client (= system that gets backed up) can map from UID/GID to user and group name by looking at /etc/passwd. If we do not chroot, then it will use the /etc/passwd of the backup server. You can bypass all this by just storing the numeric IDs always with borg create --numeric-ids

Create Backup From Remote Filesystem Using Pull

Make sure the borg repo is cryptsetuped and mounted.

Create a restricted root user on the remote system

On Debian, I need to create the /run/sshd directory first when I try to run sshd with a custom sshd_config:

# /usr/sbin/sshd -D -f /etc/ssh/sshd_config.p2p

Normally, Debian uses inetd-style Systemd sockets (see systemd.socket(5)).

The sshd_config(5) needs to contain:

PermitRootLogin forced-commands-only
AllowUsers root

Be careful to set PermitRootLogin to forced-commands-only:

If this option is set to forced-commands-only, root login with public key authentication will be allowed, but only if the command option has been specified (which may be useful for taking remote backups even if root login is normally not allowed). All other authentication methods are disabled for root.

It is questionable security to allow password authentication over SSH, so let’s disable that in sshd_config as well¹:

PasswordAuthentication no

Since we do not want to use passwords for SSH access, we create an SSH key:

$ C="$(hostname)-$USER-remotehost-remoteuser-$(date -I)"
$ ssh-keygen -t ed25519 -f ~/.ssh/"$C" -C "$C"

Copy the public key to the remote host and put it in /root/.ssh/authorized_keys. Let’s also put some restrictions on this key:

restrict,command="internal-sftp" ssh-ed25519 AAA...

restrict

Enable all restrictions, i.e. disable port, agent and X11 forwarding, as well as disabling PTY allocation and execution of ~/.ssh/rc. If any future restriction capabilities are added to authorized_keys files, they will be included in this set.

–sshd(8)

command enforces the given command (internal-sftp), even if the user provided another command.

Unlocking the Root User

The root user might be locked by default. Locked means the user account cannot be used. You cannot use a Unix password to log in, nor use any other means to access the account. Unlock root by setting the account expiry date to -1 (-e -1 = no expiry). root also needs a valid default login shell (see /etc/shells; using -s <shell> sets the shell):

# usermod -e -1 -s /bin/sh root

It is not necessary to set a password for the account nor to unlock the password since we will be using SSH keys to authenticate, not passwords.

It might be a good idea to lock the account again after the backup is done:

# usermod -L -e 1 -s /usr/bin/nologin root

This disables password login, expires the account and changes the user shell to the nologin program, which prints an error message and exits.

Make Actual Backup

Mount the remote filesystem:

$ mkdir /tmp/sshfs
$ sshfs \
    -o IdentifyFile=~/.ssh/<privkey> \
    -p 12345 \
    -o allow_other \
    -o reconnect \
    -o dir_cache=yes \
    root@remote:/ /tmp/sshfs

-o IdentityFile should point to the file containing the private key we created earlier.

We need -o allow_other for the root user to bind mount the borg repo into the sshfs. -o allow_other is documented in mount.fuse(5):

allow_other This option overrides the security measure restricting file access to the user mounting the filesystem. So all users (including root) can access the files. This option is by default only allowed to root, but this restriction can be removed with a configuration option described in the previous section.

Some other options to consider (from sshfs(1)):

-o reconnect automatically reconnect to server if connection is interrupted. Attempts to access files that were opened before the reconnection will give errors and need to be re-opened.

-o dir_cache=BOOL Enables (yes) or disables (no) the SSHFS directory cache. The directory cache holds the names of directory entries. Enabling it allows readdir(3) system calls to be processed without network access.

Mount the borg repository inside it.

$ mkdir sshfs/mnt/borg/borgrepo
# mount --bind /mnt/borg/borgrepo sshfs/mnt/borg/borgrepo

Install borg on the remote machine. Or download the borg standalone binary (the ARM version) and copy it on the remote machine. Do not forget to make it executable:

$ cp /path/to/standalone/borg /tmp/sshfs/usr/local/bin/borg
$ chmod u+x /tmp/sshfs/usr/local/bin/borg

Mount dev, proc and sys inside the sshfs²:

$ cd /tmp/sshfs
$ for fs in dev proc sys; do sudo mount --bind /$fs $fs; done

Mount borg config and cache directories:

$ mkdir -p /tmp/sshfs/root/.config/borg
$ chmod 0700 /tmp/sshfs/root/.config/borg
$ sudo mount --bind /root/.config/borg /tmp/sshfs/root/.config/borg
$ mkdir -p /tmp/sshfs/root/.cache/borg
$ chmod 0700 /tmp/sshfs/root/.cache/borg
$ sudo mount --bind /root/.cache/borg /tmp/sshfs/root/.cache/borg

Finally, enter the chroot:

# chroot /tmp/sshfs

Things to Pay Attention to

Do Not Back Up borg Repository

When running borg create, make sure to --exclude /mnt/borg/borgrepo.

Do Not Use Inode Numbers for borg Caching

Set --files-cache ctime,size. See borg-create(1). The reason is that, by default, borg uses ctime,size,inode to determine whether a file has been changed. But sshfs cannot guarantee stable inode numbers. If the inode numbers are not stable, the cache is useless: all files look like they have changed from invocation to invocation.

Provide borg Passphrase over Anonymous Pipe

If you do not want to type in your credentials when prompted, you can pass in your credentials with pass (or another program that outputs the secret over an anonymous pipe):

# pass show borg-secret | \
    chroot /tmp/sshfs \
    BORG_PASSPHRASE_FD=0 borg create repo::archive /home /etc

BORG_PASSPHRASE_FD takes a file descriptor number. 0 is stdin. See borg(1).

Tear Down

After the backup is done, exit the chroot and unmount all mounted filesystems recursively:

chroot# exit
local# umount -R /tmp/sshfs

This Was Tested on the Following Versions

$ borg --version
borg 1.4.0
$ sshfs --version
SSHFS version 3.7.3
FUSE library version 3.16.2
using FUSE kernel interface version 7.38
fusermount3 version: 3.16.2