Unix: Using 'find' and 'xargs' on filenames that have spaces within them

To use find and xargs effectively together, use -print0 to separate filenames with a null character instead of new lines. Along with that, use -0 with xargs.

$ touch "my file"
$ find | xargs ls
ls: ./my: No such file or directory
ls: file: No such file or directory
$ find -print0 | xargs -0 ls
./my file 

Unix: Script to compare a directory to a reference directory

Suppose there are two directories. We will call them as the reference and target directories. The requirement is to ensure that each and every entry - directory & file - in the reference directory is also available under a sub-tree of the target directory.

Illustration of requirement:

Given:
Reference Directory:   /tmp/a
Entry under reference directory:   /tmp/a/b/c

Then, if /home/mydir is considered as target directory, /home/mydir/b and /home/mydir/b/c should be present and are of the same type of entry i.e. directory or file.

It should not matter if there are additional directories or files under the target directory - this means as long as everything under the reference directory exists correspondingly (under the same subtree) in the target directory, the requirement is considered fulfilled.

On modern Unixes and Linux machines, we could start-off with:

diff -arq <reference_directory> <target_directory>

to further develop and refine a tool to check if our requirement is fulfilled. Alas many old Unix machines have diff versions that do not support the -a and -q options.

The following script can be a workaround if you have such a limitation.


#!/bin/sh

diff_dir_entry () {
   if ! [ -e $2 ]
   then
      echo "***ERROR: '$1' does not have corresponding '$2'."
   else
      if [ -d $1 ]
      then
         if ! [ -d $2 ]
         then
            echo "***ERROR: '$1' is a directory while '$2' is not."
         fi
      else
         if ! cmp $1 $2 >/dev/null 2>&1
         then
            echo "***ERROR: Files '$1' and '$2' differ."
         fi
      fi
   fi
}

dir_resolve()
{
   cd "$1" 2>/dev/null || return $?
   echo "`pwd -P`" # output full, link-resolved path
}

REF_DIR=`dir_resolve $1`
EXAM_DIR=`dir_resolve $2`

find $REF_DIR -mindepth 1 -print |
perl -sn -e'chomp; $examined_dir_entry=$_; $examined_dir_entry=~s{^$reference_rootdir}{$examined_rootdir}; print "$_ $examined_dir_entry\n"' -- -reference_rootdir=$REF_DIR -examined_rootdir=$EXAM_DIR |
while read DIRS; do diff_dir_entry $DIRS; done




Note:
1) Reference directory entries should not have white-space within their names.
2) Script has been tested with directories and regular files only; yet to test with soft-links.
3) Reference and target directories can be specified with absolute or relative pathnames.
4) If you have noticed, there will be a problem should the target directory be a sub-tree of the reference directory. I will have to include a constraint-check to ensure the user does not specify reference and target directories which are structured as such.


I would be very interested if someone could demonstrate a more compact way of solving this problem while still maintaining portability.

Unix: Embedding regex and external commands within sed

You have a file data.txt which has a field YEAR_MONTH:


YEAR_MONTH="201211"

And you wish to change the value to reflect the current year and month. You could use sed to automate this process with a one-line sed program:

sed -e "s/YEAR_MONTH=\"20[0-9][0-9]\(\(0[1-9]\)\|\(1[0-2]\)\)\"/YEAR_MONTH=\"`date +%Y%m`\"/g"

Note:
1. You need to use double-quotes "..." to encapsulate the entire sed program string so that the `date` command will be executed by the shell before it passes the sed program string to sed.
2. You will need to escape the inner double-quotes and the parentheses ( ) with backslashes to avoid them being interpreted by the shell.
3. 20[0-9][0-9]((0[1-9])|(1[0-2])) means all values from 200001 to 209912

Unix: How to ensure only one instance of a Unix script is running

There are times when you need to ensure only one instance of your program or script is running. The reason for this is usually related to maintaining data integrity.The common general mechanism employed by any program/script to ensure itself to be the single running instance is simply this:

1. Check if a specified condition exist
2. If the condition does not exist, create the condition and continue running; else exit

The mechanism appears straightforward but there is one problem - there are 2 distinct steps in the mechanism. In modern day multiprocessing operating systems, these 2 steps within a program cannot be guaranteed to be run in succession. To show why this is a concern, we look at 2 scenarios of two instances of a program started somewhat concurrently. We will assume the condition to be checked does not exist in both scenarios.

Scenario 1

In scenario 1, Instance-A checks for the condition and detects the condition does not exist yet, all within execution cycle-1. Before it could create the condition, the OS's process scheduler swaps it out (makes a context switch) and executes Instance-B in cycle-2. Instance-B also now thinks the condition is missing. At this point, each instance thinks it is the only one about to run - but both continue to run after creating the conditions (assuming repetitive creation of the condition is not flagged as a problem).

Scenario 2

In scenario 2, we have condition checking and the ensuing action executed as 2 continuous sub-steps within one complete (bigger) step. We term this as an atomic operation - an operation which is indivisible. The 2 sub-steps within the atomic step must be completed one after another or none at all. This way, there is a guarantee only either of the two instances will end up running.


Compiled languages in general usually have facilities built-in the language or as external API calls for the programmer to guarantee a single running instance of  her program. The shell script developer will have to resort to indirect methods to ensure a script will indeed be the sole running instance because of no innate facilities within the typical shell scripting language.


We have at least 3 (indirect) methods for the shell script developer to create a singleton process (i.e single instance process).


Method A - Lock by creating a directory - mkdir is atomic

/tmp/myprog$ ls
/tmp/myprog$ mkdir lock_dir
/tmp/myprog$ ls
lock_dir
/tmp/myprog$ mkdir lock_dir
mkdir: cannot create directory `lock_dir': File exists
/tmp/myprog$

• After creating the lock directory, you could create a file under this directory containing the PID of the locking process.

Method B - Lock by setting Bash/Korn shell's noclobber option

/tmp/myprog$ ls
/tmp/myprog$ set -o noclobber
/tmp/myprog$ echo "1" > lock_file
/tmp/myprog$ ls
lock_file
/tmp/myprog$ echo "2" > lock_file
bash: lock_file: cannot overwrite existing file
/tmp/myprog$

• It's good to have the PID of the locking process stored in the lock file.

Method C - Lock by creating a fifo - mkfifo is atomic

/tmp/myprog$ ls -l

total 0

/tmp/myprog$ mkfifo ./lock_fifo

/tmp/myprog$ ls -l

total 0

prw-r--r-- 1 openet openet 0 2012-05-04 16:41 lock_fifo

/tmp/myprog$ mkfifo ./lock_fifo

mkfifo: cannot create fifo `./lock_fifo': File exists

/tmp/myprog$

If allowed by your environment or requirements, /tmp will be a suitable location to create the lock directory, file or FIFO.

Reference:

• wiki.bash-hackers.org/howto/mutex

• UNIX Network Programming, Volume 2: Interprocess Communications (2nd Edition) - Richard Stevens

Unix: Shell (Bash) Variable Substitution

This feature offered by the shell with regards to its variables, comes in quite handy. I highlight a portion of this feature with which one can extract and use only a part of a string contained in a shell variable. The string gets truncated from either the left or the right end. The following command transcript should be self-explanatory.

$ VAR="[begin]:A_left_furthest:A_middle:A_right_furthest:[end]"
$ echo ${VAR#*A}
_left_furthest:A_middle:A_right_furthest:[end]
$ echo ${VAR##*A}
_right_furthest:[end]
$ echo ${VAR%A*}
[begin]:A_left_furthest:A_middle:
$ echo ${VAR%%A*}
[begin]:

The # or ## markers enable truncation from the left side; while the % and %% markers performs truncation from the right side of the string.

One situation in which I often find myself using variable substitution is when I need to convert a list of WAV files that were ripped from my CD, to MP3 or FLAC files.

Assuming I'm in the directory with a bunch of WAV files (with .wav extension in filename) and wish to convert them to MP3, I could do something like this:

for FILE in `ls  *.wav`

do

    lame -b 256 $FILE  $(FILE%\.wav).mp3

done


Note:

1. I converted all *.wav files in the current directory, to MP3 format, with the resulting files' .wav extension replaced by a .mp3 extension. A constant bit-rate of 256 kbps was selected for the mp3 encoding property.
2. Lame was used for the audio file format conversion.

Unix: Using ssh & tar to copy files/directories from or to remote hosts

Continuing the post Using Standard Input & Output With Tar: Copying File & Directory Trees, we cover the how-to of copying files over from or to a remote host using SSH and tar.

Suppose we wish to copy files/directories between these two hosts i.e. local & remote host. The remote host must have a SSH-server running while the local host should have a SSH-client.

There are two scenarios we will consider:

1. copying from remote host to local host - download
2. copying form local host to remote host - upload

In both scenarios, one typically employs scp to copy a bunch of files and directories (made into a tar-ball) remotely over an ssh connection. The disadvantages to this method are:
- If you interrupt the scp process of the tar file, you cannot access any of the files or portion of the tar-ball already copied.  All you get is a partially copied tar file.
- You have the additional step of creating the tar file prior to the actual copying process.

A better way of copying files between 2 machines is by exploiting ssh's ability to execute a remote command. This is much like what rsh is able to do.


To copy files/directories from remote host to local host, enter the command:

ssh -C user@host 'tar -cvf - <space_separated_list_of_files_&_directories> ' |  tar -xvf -

or

ssh -C user@host 'cd <directory>; tar -cvf - .' |  tar -xvf -

To copy everything from current directory in local host to remote host, use the command:

tar -cvf  -  .  |  ssh  -C  user@host  'cd <directory>; tar -xvf  -'

-C  option given for SSH, turns on compression for quicker transfer of data over the SSH tunnel

Unix: Difference between 'du' and 'ls'

The following exercise was done on Linux box, using a filesystem which has the storage block size of 4096 bytes i.e.  1 block = 4096 bytes
Within an empty directory, I create 5 files of the following sizes:


5-bytes                 < 1 block
6-bytes                 < 1 block
4096-bytes          = 1 block
8192-bytes          = 2 blocks
9000-bytes          > 2 blocks  &  < 3 blocks

See command transcript below.

ramesh@linux:/tmp/trial$ dd if=/dev/zero of=5_bytes.txt bs=5 count=1
1+0 records in
1+0 records out
5 bytes (5 B) copied, 0.000121224 s, 41.2 kB/s
ramesh@linux:/tmp/trial$ dd if=/dev/zero of=6_bytes.txt bs=6 count=1
1+0 records in
1+0 records out
6 bytes (6 B) copied, 2e-09 s, 3.0 GB/s
ramesh@linux:/tmp/trial$ dd if=/dev/zero of=4096_bytes.txt bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 0.000404249 s, 10.1 MB/s
ramesh@linux:/tmp/trial$ dd if=/dev/zero of=8192_bytes.txt bs=4096 count=2
2+0 records in
2+0 records out
8192 bytes (8.2 kB) copied, 0.000135646 s, 60.4 MB/s
ramesh@linux:/tmp/trial$ dd if=/dev/zero of=9000_bytes.txt bs=9000 count=1
1+0 records in
1+0 records out
9000 bytes (9.0 kB) copied, 0.000177566 s, 50.7 MB/s
 
ramesh@linux:/tmp/trial$ ls -ltr
total 32
-rw-r--r-- 1 ramesh users    5 2011-10-04 16:20 5_bytes.txt
-rw-r--r-- 1 ramesh users    6 2011-10-04 16:20 6_bytes.txt
-rw-r--r-- 1 ramesh users 4096 2011-10-04 16:20 4096_bytes.txt
-rw-r--r-- 1 ramesh users 8192 2011-10-04 16:21 8192_bytes.txt
-rw-r--r-- 1 ramesh users 9000 2011-10-04 16:25 9000_bytes.txt

The significance of the values in red from the the output of the command  ls -ltr  above is expounded below. The fifth column's values 5, 6, 4096, 8192 and 9000 indicate the total number of the bytes of the contents within the corresponding files.

The actual number of data blocks each file uses is:

File -- Block Count

5_bytes.txt -- 1 block

6_bytes.txt -- 1 block

4096_bytes.txt -- 1 block

8192_bytes.txt -- 2 blocks

9000_bytes.txt -- 3 blocks

The total count of blocks used by all the entries in the directory /tmp/trial is 8. As such, the total number of blocks used in terms of kilobytes is 8 x 4k = 32k. This is what the line total 32 makes known. Note that 1 kilo here refers to the value 1024 and not 1000.




ramesh@linux:/tmp/trial$ du *
4   4096_bytes.txt
4   5_bytes.txt
4   6_bytes.txt
8   8192_bytes.txt
12  9000_bytes.txt



du *  lists the directory entries along with their storage blocks (in terms of kilobytes). For instance, the file 6_bytes.txt actually takes up 4 kilobytes of actual storage space - as denoted in its first column. 

In short,  'ls -l' provides actual number of data bytes contained within a file while 'du' provides the block count consumed (= actually used) by the file from the filesystem.