Never been to CodeSnippets before?

Snippets is a public source code repository. Easily build up your personal collection of code snippets, categorize them with tags / keywords, and share them with the world (or not, you can keep them private!)

Copying JUST mp3 files, flattening directory structure

// This is a simple find command that copies only mp3 files
// from an iTunes directory; it only copies the files into
// a directory called "mp3_flat"

find /[[yourpath]]/iTunes\ Music -iname "*.mp3" -exec cp {} /[[yourbackuppath]]/mp3_flat/ \;

Copying JUST mp3 files

I wanted to copy just the mp3 files from my iTunes to a backup folder.
First I made a list of all mp3 files, including the full path, in a text file.
Then I went through that file, deleting any files I didn't want backed up.
Then I used rsync to make the backup.

I went through this process to ensure that I only backed up the files I wanted backed up, and no others.

Substitute your paths and file names as appropriate...this was done in the bash shell.

find [[yourpath]]/iTunes\ Music -name "*.mp3" -print >> [[yourpath]]/mp3List.txt

rsync -avu --files-from=[[yourpath]]/mp3List.txt [[yourpath]]/iTunes/iTunes\ Music [[yourbackuppath]]/mp3_files

flexbackup perl script running as a cron job

// Not so much code here as a record of events and observations
// The summary of countless hours of 'fun' tracking down reasons and solutions for flexbackup not running happily as a cron job.

Flexbackup is a well tried script to automate backup to tape of fixed-disk. It will run in Linux, BSD and Solaris (using gnu findutils [tar and find]) .

It uses perl and shell scripting to elegantly backup data 'sets'.

It uses a configuration file flexbackup.conf stored in /etc

In certain circumstances this will not run cooperatively as a 'cron' job.

This note refers to dealings with an aging Sun system running Solaris. It is considered that these notes will apply equally to many other areas. This is the result of a lot of searching and testing to find a workable solution.

The configuration was tried and tested using the terminal as direct input for commands but failed when run as a 'cron' job.

Interesting elements in my flexbackup.conf are:

##################### /etc/flexbackup.conf ################
$type = 'afio';

$compress = 'gzip'; # one of false/gzip/bzip2/lzop/zip/compress/hardware
$compr_level = '4'; # compression level (1-9) (for gzip/bzip2/lzop/zip)

# Buffering program - to help streaming
$buffer = 'false'; # one of false/buffer/mbuffer

$device = '/mnt/backup'; #the backup raid system on another machine in another office

# I had trouble with this and on an error from cron (it reported that ssh wasn't in the path) I put in the full path...however flexbackup script mistreated that on a content test so I added it as a path below
$remoteshell = 'ssh'; # command for remote shell (rsh/ssh/ssh2)

# I kept this as recording dates as it is useful when they are moved or deleted
$staticfiles = 'false';

$logdir = '/mnt/backup/log/flexbackup_log'; # directory for log files
$comp_log = 'gzip'; # compress log? false/gzip/bzip2/lzop/compress/zip
$staticlogs = 'false'; # static log filenames w/ no date stamp
$prefix = ''; # log files will start with this prefix
$tmpdir = '/mnt/backup/tmp'; # used for temporary refdate files, etc
$stampdir = '/mnt/backup/stamps'; # directory for backup timestamps
$index = '/mnt/backup/log/flexbackup_index'; # DB filename for tape indexes
$keyfile = '00-index-key'; # filename for keyfile if archiving to dir
$sprefix = ''; # stamp files will start with this prefix

# Example: If GNU tar is called "gtar" on your system:
# $path{'sudo'} = '/usr/local/bin/sudo'; # see note below
$path{'tar'} = 'sudo /usr/local/bin/tar';
$path{'pax'} = 'sudo /bin/pax';
#
# Or can be used to "sudo" certain commands. Examples:
$path{'find'} = 'sudo /usr/local/bin/find';
# $path{'dump'} = 'sudo dump';
# $path{'afio'} = 'sudo -u nonrootuser afio';
$path{'afio'} = 'sudo /usr/local/bin/afio';
$path{'ssh'} = '/usr/local/bin/ssh';
#

############################ /etc/sudoers ##############

#additions

Cmnd_Alias FLEXBACKUP = /bin/flexbackup, /bin/pax, /usr/local/bin/find, /usr/local/bin/tar, /bin/gzip, /usr/local/bin/afio

flexbackupuser ALL = NOPASSWD: FLEXBACKUP

############################ end ###################

################### comments and notes ##############

Use of crontab

Check and/or amend the editor otherwise you will get good old ed ... aagh!

$export EDITOR=vi
$sudo crontab -e

By using this command the cron daemon will autmatically be restarted to pickup the amendments.

Initially, knowing that cron disliked output even though it 'might' send it by mail...I used the following command...(for a 1:45pm test after a successful full starting backup from the terminal.

In crontab...

45 13 * * 1-5 /usr/local/bin/sudo /bin/flexbackup -set test_set -level incremental > /dev/null 2>&1

Finally realising problems I redirected output to a file using:

50 13 * * 1-5 /usr/local/bin/sudo /bin/flexbackup -set test_set -level incremental 2>&1 >> /path/to/tester.log

Note that sudo lived in /usr/local/bin.

Long story cut short...

cron operates out of its own (very very limited environment. In other words amongst other things its PATH is strictly limited. For the Sun Solaris this default path is detailed here and usually for all root jobs is restricted to /usr/sbin and /usr/bin.

http://docs.sun.com/app/docs/doc/816-1055/6m7gh31f1?a=view

I tried amending many of the paths on flexbackup.conf including setting these (which failed variously):

DO NOT USE FOR INFORMATION ONLY - THESE DO NOT WORK
$path{'sudo'} = '/usr/local/bin/sudo'; # see note below
$path{'tar'} = '/usr/local/bin/sudo /usr/local/bin/tar';
$path{'find'} = '/usr/local/bin/sudo /usr/local/bin/find';
$path{'afio'} = '/usr/local/bin/sudo /usr/local/bin/afio';
$path{'ssh'} = '/usr/local/bin/ssh';

Ultimately, I was concerned about opening too many additional paths for cron so I simply copied sudo from /usr/local/bin to /usr/bin (ironically I used sudo to copy sudo).

I didn't want to copy over the other binaries as most of them were late additions to the system and in particular would conflict with tar and find. The binaries listed in /usr/local/bin for example were gnu tar and find etc.

I could then delete the path to sudo and successfully created a sequence of backups using cron.

The final cron instruction therefore can also change the location of sudo to /usr/bin/sudo but it exists in both directories now.

For further information.

To list contents (it creates its own output log file in the current directory and rushes a copy to stdout to test your speed reading skills!

flexbackup -list name_of_the_archive.afio-gz

Interestingly this log file is suffixed with the date-time of the enquiry to avoid confusion with other logs.

To extract a single known file:

flexbackup -extract name_of_the_archive.afio-gz -onefile name_of_the_requested_file

...and there it is in the current directory. At last!

Restore?
Not tried yet

full means full/all
differential means only the files changed/added since the last backup (of that set)
incremental means only the files changed/added since the last backup

So differential is ideal for weekly catch all backups and incremental for daily backups.

An idealised makeup is a monthly full, with incrementals everyday then a weekely differential, then delete the incrementals and start them again. Then do another full and rotate the last full to tape or another partition.

To check space used by directories use:

sudo du -sk dir_name

or in a current directory sud du -sk *

(Output is in kb)

I achieved about 50% reduction in size but that obviously depends on numbers of uncompressed items.

I am happy with afio for integrity of individual files. I wanted to try pax but it failed. I never examined the reason.

I hope this helps somebody. It took me absolutely ages to sort out but I really wanted to get flexbackup working completely.

Afio used is version 2.5

Make a file backup adding date and seconds to the filename and remove the original

// In the case where a report file exists and further data will be appened the user might forget to remove or rename it so this script segment queries the user and copies the file then deletes the original version. In this script the date appended includes seconds so that multiple runs during a day will create additional unique files

#!/bin/bash

#note no spaces around equals (=) sign
base_dir='/a_directory_path_name_with_trailing_slash/'
filename='report.txt'
answer='n'
#note lack of concatenation point
report=$base_dir$filename

if (test -f $report) then
    echo "############## ATTENTION ###################"
    echo "The report file exists and it will be overwritten!"
	echo "Would you like me to make a copy of it and delete the existing version?"
		read answer
		if [ "$answer" == y ]
		then
			#date with seconds in case more than one in a day
			date=`date +%Y%m%d_%s`
			#concatenate with a point in between
			backup="$filename.$date"
			
			cp $report $base_dir$backup
			echo "Created...$backup"
			rm -rf "$report" 
			echo "Original removed"
		fi
    #continue
else
    echo "Data will be appended to the existing report file!"
    exit
fi
#script ends

Postgresql dump SS archive script

Dump, bzip, and upload a postgres database to strongspace. Via Ubernostrum

#!/usr/bin/bash

FILENAME=b_list-`/usr/xpg4/bin/date +%Y%m%d`.sql.bz2
cd /home/myuser/dumps
/opt/local/bin/pg_dump -U db_username db_name | /usr/bin/bzip2 > $FILENAME
/usr/bin/scp $FILENAME me@strongspace:/my/backup/dir/

backup files

// create backup copy

#!/bin/bash

filename=$1
date=`date +%Y%m%d`

usage () {
        echo "Usage: `basename $0` filename"
}

if [ -z "$filename" -a ! -f "$filename" ]; then
        usage
        exit 1
fi

rev=0
backup="$filename.$date.$rev"

while [ -f $backup ]; do
        let rev+=1
        backup="$filename.$date.$rev"
done

cp $filename $backup
exit $?

mysql backup / restore

// description of your code here
exports given database to a textfile which can then be zipped and downloaded or sent somewhere (mailing pending)
mysqldump -u <username> -p -q --single-transaction <db_name> > <backup_filename>


To restore the database (be sure to create the target DB before running this)
mysql -u <username> -p <db_name> < <backup_filename>


To backup and restore data only, use -t, no table data
mysqldump -u <username> -p -q --single-transaction -t <db_name> > <backup_filename>


Then restore from command line as usual.
mysql -u <username> -p <db_name> < <backup_filename>

Tar up, excluding common media extensions

Just the code, please! Hop in the dir you want to back up (note: this means it'll expand without a base directory, so do a mkdir/cd first!)

find . ! -type d -print | egrep '/,|%$|~$|\.old$|\.mpg$|\.zip$|\.wmv$|\.mp3$|\.MP3$|\.mp4$|\.MP4$|\.av2$|\.ppt$|\.dir$|\.pps$|\.qt$|SCCS|/core$|\.o$|\.orig$|\.mpeg$|\.mov$|\.doc$|\.xls$|\.pdf$|\.swf$|\.fla$|\.wav$|\.aif$|\.aiff$|\.mp3$|\.jpg$|\.JPG$|\.jpeg$|\.JPEG$|\.gif$|\.GIF$|\.png$|\.PNG$|\.psd$|\.PSD$|\.tar.gz$|\.tgz$|\.TGZ$|\.tif$|\.TIF$|\.tiff$|\.TIFF$|\.tga$|\.TGA$|\.ram$|\.rm$|\.rma$|\.psd$|\.PSD$|\.ai$|\.AI$' > Exclude
tar czfvX ~/backup_text_back.tgz Exclude .

Web and database backup


mysqldump -u root -p —all-databases > /path/to/backups/YYYY-MM-DD-alldbs.sql
scp /path/to/backups/YYYY-MM-DD-alldbs.sql.gz user@remotehost.com:/path/to/backups
tar czvf /path/to/backups/YYYY-MM-DD-production.tgz /path/to/production
scp /path/to/backups/YYYY-MM-DD-production.tgz user@remotehost.com:/path/to/backups

Mount Strongspace to a folder in Ubuntu

1) Install the software
sudo apt-get install sshfs


2) Add fuse to /etc/modules
sudo nano /etc/modules


3) Add yourself to the 'fuse' group, then log out and log in again.
sudo adduser your-username fuse


4) Create a mountpoint and give yourself ownership
sudo mkdir /media/mount-name
sudo chown your-username /media/mount-name


5) Mount the filesystem
sshfs remote-system-name:/remote-folder /media/mount-name


6) Unmount the filesystem
fusermount -u /media/mount-name


Directions lifted from Ubuntu forums and here also.

For myself, I had better results running the following command in the same directory that the file that I mounted resides in...

sudo sshfs user_name@subdomain.strongspace.com: folder_name


Where folder_name is the name of the folder that you are mounting strongspace to.