Sync backups between multiple servers and a central server

September 20, 2008  |  linux, security

Hi folks… I’m back from vacations and from some weeks without blogging.

Today I’ll write about a problem that I had on managing multiple servers and their backups. Imagine that you manage multiple servers and you want to assure that the nightly disk backups are copied to a central server that’s responsible to backup them to tape.

So, I’ll reduce my problem to 3 servers (A, B, C) that need to copy data to a central server named X.

My requirements:

  1. information is confidential, so need to be a secure file transfer;
  2. fast syncing between the 3 servers and the X server;
  3. each server is in a different network, in a different city;
  4. servers A, B and C use DSL internet connection to connect to X;
  5. all process must be automated without user intervention;
  6. I have no money to spend on this task :D

First things first… secure transfer. I’ll use encripted connections! Fast syncing… I will user RSYNC. Different cities, different networks, using internet… I must use an SSH server, i mean… openssh! Automated job… A linux cron job firing up a script! Last but not least, it’s all free! :)

When you install Linux, usually an SSH server is installed too. Linux comes with OpenSSH, an implementation of the SSH protocol. By default the authentication method is password authentication. This means that we have to enter the password when we try connect the SSH servers. As this process must be automated I would need a method to bypass the password authentication, so I’ve choose to use Public key authentication to allow the server X access the other servers automatically.

Public key authentication (PKA) is an alternative method of identifying yourself to an server, instead of typing a password. It’s much more secure and flexible. In conventional password authentication, you prove you are who you claim to be by providing the correct password. If the server has been hacked, or spoofed the attacker can learn or modify your password.

PKA solves this problem. You generate a key pair, consisting of a public key (which everybody is allowed to know) and a private key (which you keep secret). The private key is able to generate signatures. A signature created using your private key cannot be forged by anybody who does not have that key; but anybody who has your public key can verify that a particular signature is genuine.

With PKA I had 2 choices:

  1. create a private key in each server (A, B, C) and put the respective public keys in the central server X… so that A, B and C can login and copy their backups to X, configuring the X SSH server to accept PKA.
  2. create a private key in X, deploying the public key to A, B and C and configure A, B and C SSH servers to accept PKA. This would allow X to login on these servers and grab the backup files from those servers.

I’ve choose the 2nd method because I wanted to minimize the number of places I deploy a private key.

In each of the servers (A, B, C, X) I had to create an user responsible for this process. I called it “backup” and createf the dump dir (/home/backup/dump) that I will sync the backups from A, B, C to X.

Let’s start working…

(repeat the following 3 steps for each server)
[root@serverX ~]$ adduser backup
[root@serverX ~]$ su – backup
[backup@serverX ~]$ mkdir dump

Now we have to generated the private/public key pair at the server X.

[root@serverX ~]$ su – backup
[backup@serverX ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/backup/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/backup/.ssh/id_rsa.
Your public key has been saved in /home/backup/.ssh/id_rsa.pub.
The key fingerprint is: XXXXXXXXXXXXXXXXXXXXXXXXX

SSH is very picky about permissions. I’ve successfully configured PKA with this permissions:

[backup@serverX ~]$ chmod 700 ~/.ssh
[backup@serverX ~]$ chmod 644 ~/.ssh/*
[backup@serverX ~]$ chmod 600 ~/.ssh/id_rsa

Now that the private (id_rsa) and the public key (id_rsa.pub) are generated I’ll to copy the public key to each of the servers. The first step copies the public to the tmp dir, and the second create the .ssh directory where by default we deploy the authorized_keys file containing the public keys that are authorized to connect in those servers with the user backup.

(repeat the following2 steps for each A, B and C)

[backup@serverX ~]$ scp ~/.ssh/id_rsa.pub backup@serverA.com:/tmp/id_rsa.pub
[backup@serverX ~]$ ssh backup@serverA.com “mkdir -p ~/.ssh;chmod 700 ~/.ssh;touch ~/.ssh/authorized_keys;cat /tmp/id_rsa.pub >> ~/.ssh/authorized_keys”

Next, let’s configure SSH server (on A, B and C) to accept PKA. In Linux edit the /etc/ssh/sshd_config and add these lines:

RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile    .ssh/authorized_keys

Let’s restart SSH server:

[root@serverA ~]$ /etc/init.d/sshd restart

And voilá, we’re done with the PKA. We can test it by trying to login as “backup” on A from X like this:

[root@serverX ~]$ su – backup
[backup@serverX ~]$ ssh backup@serverA.com
[backup@serverA ~]$ hostname
serverA.com
[backup@serverA ~]$

Ok, now the easy part of the problem… Copying/syncing the files. From server X, let’s try to copy/sync the content of the /home/backup/dump from each of the 3 servers. We’ll use rsync

The utility rsync is a software application for Unix/Linux systems which synchronizes files and directories from one location to another while minimizing data transfer using delta encoding when appropriate. The rsync utility uses an algorithm (invented by Andrew Tridgell) for efficiently transmitting data across a communications link when the receiving computer already has a different version of that data.

[root@serverX ~]$ su – backup
[backup@serverX ~]$ rsync -avu backup@serverA.com:/home/backup/dump/ /home/backup/dump/

So after we test successfully the rsync, let’s create a script, ran by crontab, that will rsync the 3 servers (A,B,C) with the central server X.

As user “backup” create a file, using your favorite text editor, that will be the script ran by crontab. In that script add these 3 lines, the 3 rsync operations with the 3 servers.

rsync -avu backup@serverA.com:/home/backup/dump/ /home/backup/dump/
rsync -avu backup@serverB.com:/home/backup/dump/ /home/backup/dump/
rsync -avu backup@serverC.com:/home/backup/dump/ /home/backup/dump/

After that, and supposing that your script is saved as /home/backup/my_rsync.sh and you want to run this job daily at 3:30 AM, you must have a line like this at your crontab:

30 3 * * * (/home/backup/my_rsync.sh)

To edit the crontab you should write issue the command “crontab -e”. Crontab editor works like “vi”. We can have multiple lines on crontab each one specifing a job to run at the scheduled time. After saving the crontab we have a job that will run the sync process between the 3 servers and the server X.

I hope this long post can help someone in this pretty but crazy place called Earth :)


1 Comment


  1. Greetings from Carolina! I’m bored to death at work so I decided to browse your blog on my iphone during lunch break. I love the info you present here and can’t wait to take a look when I get home. I’m shocked at how fast your blog loaded on my phone .. I’m not even using WIFI, just 3G .. Anyhow, good blog!

Leave a Reply