Switched On Tech Design

Tag: tar

  • Ubuntu: How to use multiple cores with tar gzip compression

     

    One thing you may have noticed when using the z switch with tar is that the compression can take some time! If you look at your CPU usage, though, you’ll notice that only one core is being utilised to compress the files. In a modern system 4 or 8 cores are common, meaning that there is plenty of potential to speed up the process if you could utilise more cores. As the gzip package only supports one core, we need to look elsewhere.

     

    Fortunately, there is a gzip package which uses multiple cores available – it’s called pigz. To install it type:

     

    sudo apt-get install pigz


     

    Once that is installed we can tell the tar command to use it like so:

     

    tar -c --use-compress-program=pigz  -f [tar file] [directory or files]


     

    e.g.:

     

    tar -c --use-compress-program=pigz -f backupOfMovies.tar /opt/movies


     

    Note the double hyphen before use. Check your CPU usage while the command is running – you should be able to see all available cores being utilised!

  • Ubuntu: How to add or append a file to an existing tape backup

     

    Sometimes you may with to add files to an existing backup; if you issue a command like:

     

    tar -cvf /dev/st0 backupfiles

     

    …and the tape is not already set to the end of the previous archive you will over-write any data from the position on the tape. Use the “eom” command to move the tape to the end of the alread-recorded files like so:

     

    mt -f /dev/[path-to-tape] eom

     

    e.g.:

     

    mt -f /dev/st0 eom

     

    Now you can use tar to add a file to the tape without over-writing the existing data.

  • Ubuntu: Using compression on your tape backups

     

    If you have compressible data you may save space on you tapes by using compression; this comes at a cost of CPU cycles to do the compressing, which can often be a worthwhile tradeoff for a long-term backup. To do this is quite simple – add in the -z switch to your tar command.

     

    tar -cvzf /dev/[tape-device] [folder or files to back up]

     

    e.g.:

     

    tar -cvzf /dev/st0 /opt/movies

     

    For some file types – e.g. movies, mp3s, compressed picture files and the like you probably won’t see a great deal of space saved – though if it enough to save you from using two tapes instead of one, it may be worth it even so. Text and other file types may compress more easily and you may see more of a savings – it will vary greatly depending on your dataset. Try it and see!

     

    Sometimes you may see people using the -j switch instead – this uses the bzip2 algorithm rather than the gzip algorithm (the -z switch). You will probably find that gzip is slightly better supported and bzip2 sometimes provides slightly better compression but takes longer. If you are chasing better compression it may be worth replacing the z switch with j to see if it helps.

  • Ubuntu: Using tar to span a backup across multiple tapes

     

    Following on from the article on writing files to a tape backup, what happens if you want to back up more than what fits on a single tape? Here we want to use the –multi-volume switch, or -M. Use it so:

     

    tar -cvMf /dev/[path-to-tape] [backup folder or files]

     

    As a working example:

     

    tar -cvMf /dev/st0 /opt/movies

     

    Once the first tape is full, tar will prompt you to insert another:

     

    Prepare volume #2 for `/dev/st0′ and hit return:

     

    Make sure you replace the tape with another before hitting return or else it will over-write the just-finished tape!

     

    With the exception of a file which sits across multiple tapes (think of a file which starts at the end of one tapes and finishes at the start of the next) you don’t need a first tape to restore the files on the second. You also cannot use compression for multi-volume archives.