Backup Manager's behaviour is defined in configuration files. You can run Backup Manager with different configuration files (at the same time or not). This chapter will cover all the configuration keys supported in version 0.7.7 and will explain their meaning.
Backup Manager stores archives it builds in a repository. Archives are built by using a backup method.
The repository is the place in your filesystem where all archives are stored. This is a particular place for Backup Manager, it will be cleaned during backup sessions: archives older than the authorized lifetime will be purged. If the repository does not exist, it will be created at runtime.
Isolating the repository on a dedicated partition is a good idea. This can prevent the repository from eating all the disk space of the partition. With a bad configuration file, backup sessions can lead to huge archives, for many reasons, so take care.
For security reasons, the repository can be accessible by a specific user/group pair. This will prevent the archives from being readable (and writable) by any user in the system. This mode is enabled by default (owned by root:root).
You can also change the permission of the repository and the archives, that is possible with two configuration variables: BM_REPOSITORY_CHMOD and BM_ARCHIVE_CHMOD.
If you cannot trust the place where you store your archives, you can choose to encrypt them so you are the only one who can read their content. That's a very good idea for archives you plan to upload to some remote place, or even for the archives you want to daily export on removable media.
For Backup Manager, encryption is defined in one place in the configuration file. If the variable "BM_ENCRYPTION_METHOD" is not defined, no encryption occurs during the archive build process, if a method is defined there, then any archive built are encrypted through a pipeline with that method.
Backup Manager will encrypt your archive through a pipeline in order not to write any byte of unencrypted data on the physical media. The encryption will be performed with a command line like the following:
To decrypt an archive built with GPG encryption, you have to be the owner of the private GPG key for which the encryption was made. Then issue the following:
As explained in the previous section, that variable should contain the GPG recipient for the encryption, eg: your GPG ID.
Archives are produced by backup methods, they can be virtually anything, but will always be named like the following: prefix-name-date.filetype. An archive is a file that contains data, it can be compressed or not, in a binary form or not.
As explained in the BM_REPOSITORY_ROOT section, every archive built by Backup Manager will be purged when their lifetime expires. In versions prior to 0.7.6, any archive were purged.
You can now choose to purge only the archive built in the scope of the configuration file, that is: archives prefixed with BM_ARCHIVE_PREFIX.
This is useful if you share the same BM_REPOSITORY_ROOT with different instances of Backup Manager that have different purging rules (eg: a BM_REPOSITORY_ROOT shared over NFS for multiple Backup Manager configuration).
Backup Manager does handle several archive methods, which can use a lot of ressources (mostly CPU); although this can be acceptable if Backup Manager is run at night, on a always-running server, it can seriously slow-down a desktop computer. Indeed, most of the time, desktop users use anacron to run backup-manager when possible, and most of time this is when the desktop is actually used.
To enhance the desktop-experience when archives are built, you can adjust the niceness used for archive creation with this configuration variable.
If disk usage matters in your backup strategy, you might find useful to use Backup Manager's duplicates purging feature. When an archive is generated, Backup Manager looks at the previous versions of this archive. If it finds that a previous archive is the same file as the one it has just built, the previous one is replaced by a symlink to the new one. This is useful if you don't want to have the same archive twice in the repository.
One of the main concepts behind the handling of the repository is to purge deprecated archives automatically. The purge session is always performed when you launch Backup Manager. During this phase, all archives older than the authorized lifetime are dropped.
Since version 0.7.3, Backup Manager purges only files it has created whereas in previous versions, it used to purge also other files within the repository.
Note that when using the incremental method for building archives, Backup Manager will handle differently master backups and incremental ones. The incremental backups will be purged like any other archives (when exceeding the authorized lifetime). On the ohter hand, deprecated master backups won't be purged unless there is a younger master backup in the repository. Then, even with a lifetime set to three days, a master backup will live more than three days, until a newer master backup is built.
On most setups, all the archives are stored in the top-level directory specified by the configuration key BM_REPOSITORY_ROOT. But it can make sense to have subdirectories, for instance to store archives uploaded from other hosts running Backup Manager. In this case, it is possible to ask Backup Manager to purge those directories too, by setting BM_REPOSITORY_RECURSIVEPURGE to true.
Please note that the BM_ARCHIVE_TTL value is global, so if you want to have different lifetimes for some archives, this is not the way to go. In this case you should save them outside BM_REPOSITORY_ROOT and write a cron job to do the purge (possibly calling backup-manager --purge with an alternate configuration file).
The core feature of Backup Manager is to make archives, for doing this, a method is used. Each method can require a set of configuration keys. We will describe here every method supported in the version 0.7.7.
The method you choose must be defined in the configuration key BM_ARCHIVE_METHOD. You can put here a list of all the different methods you want to use. Take care to put every configuration key needed by all the methods you choose. Note that you can also choose none of the proposed methods, if you don't want to build archives with this configuration file, then just put none.
If all you want to do is to handle a couple of tarballs of your file system, you can use this method. This method takes a list of directories and builds the corresponding tarballs. This method is the default one, this is the easiest to use, it just builds tarballs as you could do with your own tar script. Its main drawback is to eat a lot of disk space: archives can be big from a day to another, even if there are no changes in their content. See the tarball-incremental method if you want to optimize archives' size.
When building full backups (when not building incremental ones), Backup Manager will append the keyword "master" to the name of the archive. This is very useful when using the tarball-incremental method for seeing where the full backups are quickly.
Basically, this configuration key defines the filetype of the resulting archive. In a way, it defines which compressor to use (zip, gzip, dar or bzip2). Here are the supported values: tar, tar.gz, tar.bz2, zip and dar. Note that depending on the filetype you choose, you will have to make sure you have the corresponding compressor installed.
Since version 0.7.1, Backup Manager supports dar archives. This archiver provides some interesting features like the archive slicing.
If you want to make sure your archives won't exceed a given size (for instance 2 GB) you can use that configuration variable, but only if you are using the dar BM_TARBALL_FILETYPE. Indeed this feature is only supported by dar.
If you want to provide extra options to "tar" or "dar" you may do so here. Leave blank unless you know what you are doing.
It is possible, when generating the tarball (or the zip file) to dereference the symlinks. If you enable this feature, every symbolic link in the file system will be replaced in the archive by the file it points to. Use this feature with care, it can quickly lead to huge archives, or even worse: if you have a circular symlink somewhere, this will lead to an infinite archive!
Since version 0.7.3, this variable is replaced by the array BM_TARBALL_TARGETS[], it's still supported for backward compatibility though. You can use this variable for defining the locations to backup, but you must not use this variable if one or more of the paths you want to archive contain a space.
If you want to backup some targets that have spaces in their name (eg "Program Files"), you must not use this variable, but the array BM_TARBALL_TARGETS[] instead.
This variable holds every place you want to backup. This is the recommanded variable to use for defining your backup targets (BM_TARBALL_DIRECTORIES is deprecated since version 0.7.3).
You can safely put items that contain spaces (eg: "Program Files") whereas you can't with BM_TARBALL_DIRECTORIES.
You can also put Bash patterns in BM_TARBALL_TARGETS[], it will be expanded at runtime to find the resulting targets. For instance : BM_TARBALL_TARGETS[0]="/home/*" will lead to backup every home's sub-directory.
It can be very useful to prevent some locations of your filesytem from being included in the archives. This is really useful when you use wildcards in BM_TARBALL_DIRECTORIES. Indeed, you may want to backup every top-level directory of your filesystem (/*) but without volatile locations like /tmp, /dev and /proc.
You can also use this variable for excluding every files of a given extension, like for instance mp3 or mpg files.
If you want to archive some remote locations from a server where Backup Manager is insalled, you can choose to build archives over SSH. This is useful if you don't want to install Backup Manager every where and setup some upload methods from all thoses servers to a central data storage server. This way, Backup Manager will build some archives directly over SSH and will store the resulting tarballs locally, as if it was built like any other archive. The resulting archive will be prefixed with the remote hostname instead of BM_ARCHIVE_PREFIX.
BM_UPLOAD_SSH_USER: the user to use for connecting to the remote server. Note that this user will run tar remotely, so take care to archive something this user can read!
If you enable this feature, note that the resulting configuration file will have the following restrictions:
Example: You have three hosts: host01, host02 and host03. You want to set up host01 as a data storage server, it has a big /var/archives partition. You want to archive "/etc", "/home" and "/var/log" on box02 and box03 and store the archives on host01.
Of course, for this to work correctly, `bamuser' should be a valid user on box02 and box03; it must be allowed to connect to them with SSH key autentication and has to be able to read those directories.
If you want to handle tarballs without wasting disk space, you should use this method. The concept of this method is simple: You choose a frequency when a full backup is made (exactly like the one made by the tarball mehod). All the days between two full backups, archives contain only the files that have changed from the previous archive.
For instance, let's say you want to backup /home with this method. Your /home directory is composed by two sub-directories: /home/foo and /home/bar. You choose a weekly frequency and say that monday will be the "fullbackup" day. Obviously, you will have a full tarball of /home on monday. Then, if a file changed inside /home/foo and if /home/bar remains unchanged, tuesday's archive will only contain the modified files of /home/foo. Using this method will save a lot of disk space.
To build incremental tarballs, Backup Manager uses tar's switch --listed-incremental. This will create a file for each target which will contain some statistics used by tar to figure out if a file should be backed up or not. When Backup Manager is run for the first time, this file doesn't exist, so the first tarballs made are always master backups. If the incremental list files get removed, the next backups won't be incremental.
Since version 0.7.3, it's possible to see at the first glance if a backup is a master or an incremental one: master backup have the keyword master appended to the date. When purging the repository, the master backups are not removed as the incremental ones. Backup Manager always keep a master backup that is older than incremental archives.
This method uses all the tarball's configuration keys and adds two more. One to define the kind of frequency, the other to choose on which day the full backups should be done.
This is the type of frequency you want to use. If you choose weekly, you'll have to choose a day number between 0 and 6 for the BM_TARBALLINC_MASTERDATEVALUE configuration key, if you choose monthly, the day number will be between 1 and 31.
The number of the day when making full backups. Note that its meaning directly depends on the BM_TARBALLINC_MASTERDATETYPE. For instance, 1 means "monday" if you choose a weekly frequency, but it means "the first day of the month" if you choose a monthly frequency.
This method provides a way to archive MySQL databases, the archives are made with mysqldump (SQL text files) and can be compressed.
In versions prior to 0.7.6, Backup Manager used to pass the MySQL client's password through the command line. As explained by the MySQL manual, that's a security issue as the password is then readable for a short time in the /proc directory (or using the ps command).
To close that vulnerability, the MySQL client password is not passed through the command line anymore, it is written in a configuration file located in the home directory of the user running Backup Manager : ~/.my.cnf.
If that file doesn't exist at runtime, Backup Manager will create it and will then write the password provided in BM_MYSQL_ADMINPASS inside.
This is the list of databases you want to archive. You can put the keyword __ALL__ if you like to backup every database without having to list them.
The best way to produce MySQL dumps is done by using mysqldump's --opt switch. This makes the dump directly usable with mysql (adds the drop table statements), locks tables during the dump generation and other cool things (see mysqldump). This is recommended for full-clean-safe backups, but needs a privileged user (for the lock permissions).
The MySQL login you want to use for connecting to the database. Make sure this login can read all the databases you've set in BM_MYSQL_DATABASES.
If you have already made your own ~/.my.cnf configuration file, you don't have to set that variable.
If you don't know what is the ~/.my.cnf configuration file, set the password, then Backup Manager will take care of creating the MySQL client configuration file.
The archive is made with mysqldump which renders SQL lines; the resulting text file can be compressed. If you want to compress the file, choose the compressor you want. Leave it blank if you want pure SQL files.
You can archive Subversion repositories with this method. The archive will be made with svnadmin and will contain XML data (text files). Like the mysql method, you can choose to compress it.
If you want to compress the resulting XML files, choose a compressor here. Leave this blank if you don't want any compression.
Even if most of the common needs are covered by the existing methods, there is always a case uncovered. Backup Manager provides a way for backing up anything, and can be used in such circumstances.
This method is called pipe, it is more complex to use but can virtually backup anything. The concept is simple, a pipe method is defined by the following items:
For each pipe method defined, Backup Manager will launch the command given and redirect the content sent to stdout by this command to a file named with the name of the method and its filetype. Then, if the method uses a compressor, the file will be compressed.
One of the most important thing to do when backing up file systems is to store the archives on different places. The more different physical spaces you have, the better. Backup Manager provides a way for achieving this goal : the upload methods.
There are different upload methods, each of them behaves differently and provides particular features. In Backup Manager 0.7.7 you can use FTP, SSH, RSYNC or Amazon S3 uploads.
In the same manner as for backup methods, you can choose to use as many upload methods as you like. If you don't want to use this feature at all, just put the keyword none in the configuration BM_UPLOAD_METHOD.
Note that the FTP, SSH and S3 methods are dedicated to upload archives, using those method depends on the use of at least one backup method.
On the opposite, the RSYNC method uploads a directory to remote locations, this directory can be your repository or whatever other location of your file sytem.
Each of the hosts defined in that list is used by all the upload methods when establishing connections. For instance if you want to perform SSH uploads of your archives and RSYNC upload of a location to the same host, put it in this list.
If you have installed installed Backup Manager on the remote host, a good idea is to choose a sub-directory of the repository. Then, during the remote host purge phase, your uploads will be cleaned at the same time.
You can also define a destination dedicated to your host: BM_UPLOAD_DESTINATION="/var/archives/$HOSTNAME"
Let's say you want that all your uploads are performed on the host mirror2.lan.mysite.net, in the sub-directory /var/archives/uploads
Method name: ssh, goal: upload archives to remote hosts over SSH. This method depends on a backup method.
If you want to upload your archives on remote locations, you can use the SSH method. This method is good if you like to use a secure tunnel between the two points of the upload.
The call to scp will be done with the identity of the user BM_UPLOAD_SSH_USER, thus, you have to make sure this user can have access to the repository (take care to the secure mode).
This is the user to use for performing the ssh connection. Make sure this user can access repository.
Put here the list of hosts to use for SSH-only uploads. Note that if you put some hosts in BM_UPLOAD_HOSTS, they will be used as well.
If you set this boolean to "true", the remote archives will be purged before the new ones are uploaded. The purging rules are the same as the ones Backup Manager uses for local purging. If BM_UPLOAD_SSH_TTL is defined, this time to live will be used, else BM_ARCHIVE_TTL will be used.
Method name: ssh-gpg, goal: encrypt arcives using public key encryption and upload the result to untrusted remote hosts over SSH. This method depends on a backup method.
The upload using SSH can also be combined with public key encryption provided by gpg. The archives will be encrypted using a public key prior to sending them over the network, so on the remote server your files are protected from inspection.
This method can be used to protect your data from inspection on untrusted remote servers. However, since the encrypted files are not signed, this does not protect you from archive manipulation. So the md5 hases are still needed.
This parameter sets the recipient for which the archive is encrypted. A valid specification is a short or long key id, or a descriptive name, as explained in the gpg man page. The public key for this identity must be in the key ring of the user running gpg, which is the same as specified by BM_UPLOAD_SSH_USER. To test this run the command gpg --list-keys ID as that user, where ID is the ID as you give it to this parameter. If gpg displays exactly one key, then you are fine. Refer to the gpg man page for further details.
If security does not matter much on your lan (between the two points of the upload) you can choose to use the FTP method. One of the main pros of this method is that it can perform purging independently. You can safely use this method for uploading files to a host where you just have an FTP account.
If this variable is set to true, FTP transfers will be performed in passive mode, which is mandatory in NATed/firewalled environments.
Using different time to live values for local and remote archives can be useful in certain situations. For instance, it's possible to install Backup Manager locally, make it build archives, upload them to a remote FTP host and then purge them locally (but not on the remote host). Doing this is possible with setting a null value to the local TTL (BM_ARCHIVE_TTL) and a non-null value to BM_UPLOAD_FTP_TTL.
Put here the list of hosts to use for FTP-only uploads. Note that if you put some hosts in BM_UPLOAD_HOSTS, they will be used as well.
You can choose to purge deprecated archives before uploading new ones. This purge is done over FTP and uses the configuration key BM_ARCHIVE_TTL in the same manner as the local purge behaves (the FTP purge is not recursive though).
Amazon's new Simple Storage Service (S3) is an Internet "web service" that permits you to store unlimited blocks of data on their replicated and managed systems. See http://aws.amazon.com for more information. Registration is free and the rates are quite reasonable.
Using the S3 upload method will permit your archives to be stored on Amazon's S3 service. You must allocate a "bucket" to the exclusive use of Backup Manager. Each of your created archives will be uploaded to S3 and stored within this bucket in a key name that matches the name of the archive.
As with the other backup methods Backup Manager does not assist you in restoring files from archives. You must retrieve archives from S3 using other mechanisms such as the S3Shell provided as an examle command line utility by Amazon.
Note that when using this upload method, the BM_UPLOAD_HOSTS variable is ignored as the only valid host for S3 uploads in s3.amazon.com.
This option is required for the S3 upload method. This specifies the bucket used to store backup data. If the bucket does not exist it will be created as a private bucket. This key overrides BM_UPLOAD_DESTINATION. Note that Amazon requires that bucket names be globally unique. Be creative picking one.
This option is required for the S3 upload method. After you have registered Amazon will provide you an access key. You must use this key to access your storage on S3.
This option is required for the S3 upload method. After you have registered Amazon will provide you a secret key. You must use this key to write to your storage on S3.
You can choose to purge deprecated archives before uploading new ones. This purge is done over S3 and uses the configuration key BM_ARCHIVE_TTL in the same manner as the local purge behaves (the S3 purge is not recursive though).
You may want to upload some parts of your file system to some remote hosts. In these cases, archives are not needed, you just want to synchronize some directories to remote places. This is where the RSYNC upload method is useful.
RSYNC uploads need a SSH user/key pair to behave correctly, thus there is a dependency against the keys BM_UPLOAD_SSH_USER and BM_UPLOAD_SSH_KEY.
Put here the list of hosts to use for RSYNC-only uploads. Note that if you put some hosts in BM_UPLOAD_HOSTS, they will be used as well.
You can choose to dereference files pointed by symlinks in your RSYNC snapshots. This feature should be used with care.
In version 0.7.7, only CDs and DVDs are supported as external media, so we will discuss in this section only the BM_BURNING features. Other exports are expected to come in next versions though.
In non-interactive mode (when backup-manager is not lauchned from a terminal), any of these methods will try to put the whole archive repository in the media, if it does not fit in the media, it will try to put only the archives built on the day, if that's not possible, nothing will be burnt.
In interactive mode (when backup-manager is launched from a terminal), the whole repository will be burnt into as many media as needed. When a medium is satured with archives, backup-manager will pause the process asking the user to put a new media inside.
The CDRW and DVD-RW methods will first blank the media, so you can safely use these methods if you want to use the same medium several times.
The CDR and DVD medthods won't blank the medium first (DVD+RW media doesn't need blanking, it's possible to re-burn data on-the-fly over such media)..
DVD media are handled by the tool dvd+rw-tools, problems can occur in CRON environment with dvd+rw-tools versions prior to 6.1, make sure to have 6.1 or later if you want to burn DVD media with Backup Manager.
All those burning methods share the same configuration keys, so it's easy to switch from a medium to another.
This is mandatory for using the burning feature, it's the device to use for mounting the media. It's needed by backup manager for performing the MD5 checks and for other needs.
Backup Manager uses cdrecord for burning CDs. If when you run cdrecord -scanbus you don't see your burning device, that means you will have to force the device in ATA mode. To tell Backup Manager to do so, just put here the path to your device, and a switch will be appended to the cdrecord commandline like the following : cdrecrord ... dev=$BM_BURNING_DEVFORCED ....
This is where you define the maximum size (in megabytes) of the media you will put in the device. Here is the list of the common sizes:
When Backup Manager looks in the repository for burning data, it will try to put the whole archive repository in the media. If the summarized size of the repository does not fit in BM_BURNING_MAXSIZE, Backup Manager will then try to put only the archives of the day.
If this boolean is set to a true value, every MD5 sum will be checked when the media is burned in order to make sure everything is ok.
If you want to log Backup Manager actions to syslog, you can enable the internal logger, this is done with the configuration key BM_LOGGER. You are also able to choose which syslog facility to use thanks to the key BM_LOGGER_FACILITY.
You have the availability to write your own hooks if you want to automate some special beaviours within the Backup Manager process. You may like to mount over NFS your archive repository before the backup session and unmount it after, or you may like to launch your own uploader script when the backup session is finished.
In order to let you implement any solution you like, Backup Manager provides two different hooks: the pre-command and post-command hooks.
Put here the path to a program (or a shell command) to launch before the backup session. If the command fails (exits with non zero value, or prints the keyword false on stdout) the backup session will stop. If the pre-command succeeds, the process can follow.
Put here the path to a program (or a shell command) to launch after the backup session. If the command fails (exits with non zero value, or prints the keyword false on stdout) Backup Manager will exit with an error code (and will log to syslog the post-command failure if the logger is enabled).
We can supply part# 6087T10P21, check the quantities shown below and then request a quote for pricing and delivery. 4 Star Electronics, Inc. is the world's leading supplier of obsolete electronic components. We are ISO 9001:2008 certified and ANSI/ESD-S20.20 certified. Our 30,000 sqft. facility houses over 150,000 line items of obsolete, allocated and discontinued electronic components. Part number 6087T10P21 and our other In-Stock parts ship same day when ordered by 2pm PST. We look forward to doing business with you.
* In-stock inventory can ship today if ordered by 2:00pm (PST). Available inventory can ship in 1-5 days depending on warehouse location. Call today and a sales rep can provide you price and delivery on Part# 6087T10P21
We can supply part# 940070-1B, check the quantities shown below and then request a quote for pricing and delivery. 4 Star Electronics, Inc. is the world's leading supplier of obsolete electronic components. We are ISO 9001:2008 certified and ANSI/ESD-S20.20 certified. Our 30,000 sqft. facility houses over 150,000 line items of obsolete, allocated and discontinued electronic components. Part number 940070-1B and our other In-Stock parts ship same day when ordered by 2pm PST. We look forward to doing business with you.
* In-stock inventory can ship today if ordered by 2:00pm (PST). Available inventory can ship in 1-5 days depending on warehouse location. Call today and a sales rep can provide you price and delivery on Part# 940070-1B
We can supply part# SN74LS258AJ, check the quantities shown below and then request a quote for pricing and delivery. 4 Star Electronics, Inc. is the world's leading supplier of obsolete electronic components. We are ISO 9001:2008 certified and ANSI/ESD-S20.20 certified. Our 30,000 sqft. facility houses over 150,000 line items of obsolete, allocated and discontinued electronic components. Part number SN74LS258AJ and our other In-Stock parts ship same day when ordered by 2pm PST. We look forward to doing business with you.
* In-stock inventory can ship today if ordered by 2:00pm (PST). Available inventory can ship in 1-5 days depending on warehouse location. Call today and a sales rep can provide you price and delivery on Part# SN74LS258AJ
We can supply part# VC10F151R, check the quantities shown below and then request a quote for pricing and delivery. 4 Star Electronics, Inc. is the world's leading supplier of obsolete electronic components. We are ISO 9001:2008 certified and ANSI/ESD-S20.20 certified. Our 30,000 sqft. facility houses over 150,000 line items of obsolete, allocated and discontinued electronic components. Part number VC10F151R and our other In-Stock parts ship same day when ordered by 2pm PST. We look forward to doing business with you.
* In-stock inventory can ship today if ordered by 2:00pm (PST). Available inventory can ship in 1-5 days depending on warehouse location. Call today and a sales rep can provide you price and delivery on Part# VC10F151R