Tuesday, December 13, 2011

Unix: Shell (Bash) Variable Substitution


This feature offered by the shell with regards to its variables, comes in quite handy. I highlight a portion of this feature with which one can extract and use only a part of a string contained in a shell variable. The string gets truncated from either the left or the right end. The following command transcript should be self-explanatory.

$ VAR="[begin]:A_left_furthest:A_middle:A_right_furthest:[end]"
$ echo ${VAR#*A}
_left_furthest:A_middle:A_right_furthest:[end]
$ echo ${VAR##*A}
_right_furthest:[end]
$ echo ${VAR%A*}
[begin]:A_left_furthest:A_middle:
$ echo ${VAR%%A*}
[begin]:


The # or ## markers enable truncation from the left side; while the % and %% markers performs truncation from the right side of the string.

One situation in which I often find myself using variable substitution is when I need to convert a list of WAV files that were ripped from my CD, to MP3 or FLAC files.

Assuming I'm in the directory with a bunch of WAV files (with .wav extension in filename) and wish to convert them to MP3, I could do something like this:


for FILE in `ls  *.wav`
do
    lame -b 256 $FILE  $(FILE%\.wav).mp3
done


Note:
  1. I converted all *.wav files in the current directory, to MP3 format, with the resulting files' .wav extension replaced by a .mp3 extension. A constant bit-rate of 256 kbps was selected for the mp3 encoding property.
  2. Lame was used for the audio file format conversion.

Monday, November 28, 2011

Unix: Using ssh & tar to copy files/directories from or to remote hosts

Continuing the post Using Standard Input & Output With Tar: Copying File & Directory Trees, we cover the how-to of copying files over from or to a remote host using SSH and tar.

Suppose we wish to copy files/directories between these two hosts i.e. local & remote host. The remote host must have a SSH-server running while the local host should have a SSH-client.

There are two scenarios we will consider:
  1. copying from remote host to local host - download
  2. copying form local host to remote host - upload
In both scenarios, one typically employs scp to copy a bunch of files and directories (made into a tar-ball) remotely over an ssh connection. The disadvantages to this method are:
- If you interrupt the scp process of the tar file, you cannot access any of the files or portion of the tar-ball already copied.  All you get is a partially copied tar file.
- You have the additional step of creating the tar file prior to the actual copying process.

A better way of copying files between 2 machines is by exploiting ssh's ability to execute a remote command. This is much like what rsh is able to do.


To copy files/directories from remote host to local host, enter the command:

ssh -C user@host 'tar -cvf - <space_separated_list_of_files_&_directories> ' |  tar -xvf -

or

ssh -C user@host 'cd <directory>; tar -cvf - .' |  tar -xvf -

To copy everything from current directory in local host to remote host, use the command:

tar -cvf  -  .  |  ssh  -C  user@host  'cd <directory>; tar -xvf  -'


-C  option given for SSH, turns on compression for quicker transfer of data over the SSH tunnel

Tuesday, October 4, 2011

Unix: Difference between 'du' and 'ls'

The following exercise was done on Linux box, using a filesystem which has the storage block size of 4096 bytes i.e.  1 block = 4096 bytes
Within an empty directory, I create 5 files of the following sizes:


5-bytes                 < 1 block
6-bytes                 < 1 block
4096-bytes          = 1 block
8192-bytes          = 2 blocks
9000-bytes          > 2 blocks  &  < 3 blocks

See command transcript below.

ramesh@linux:/tmp/trial$ dd if=/dev/zero of=5_bytes.txt bs=5 count=1
1+0 records in
1+0 records out
5 bytes (5 B) copied, 0.000121224 s, 41.2 kB/s
ramesh@linux:/tmp/trial$ dd if=/dev/zero of=6_bytes.txt bs=6 count=1
1+0 records in
1+0 records out
6 bytes (6 B) copied, 2e-09 s, 3.0 GB/s
ramesh@linux:/tmp/trial$ dd if=/dev/zero of=4096_bytes.txt bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 0.000404249 s, 10.1 MB/s
ramesh@linux:/tmp/trial$ dd if=/dev/zero of=8192_bytes.txt bs=4096 count=2
2+0 records in
2+0 records out
8192 bytes (8.2 kB) copied, 0.000135646 s, 60.4 MB/s
ramesh@linux:/tmp/trial$ dd if=/dev/zero of=9000_bytes.txt bs=9000 count=1
1+0 records in
1+0 records out
9000 bytes (9.0 kB) copied, 0.000177566 s, 50.7 MB/s

 

ramesh@linux:/tmp/trial$ ls -ltr
total 32
-rw-r--r-- 1 ramesh users    5 2011-10-04 16:20 5_bytes.txt
-rw-r--r-- 1 ramesh users    6 2011-10-04 16:20 6_bytes.txt
-rw-r--r-- 1 ramesh users 4096 2011-10-04 16:20 4096_bytes.txt
-rw-r--r-- 1 ramesh users 8192 2011-10-04 16:21 8192_bytes.txt
-rw-r--r-- 1 ramesh users 9000 2011-10-04 16:25 9000_bytes.txt


The significance of the values in red from the the output of the command  ls -ltr  above is expounded below. The fifth column's values 5, 6, 4096, 8192 and 9000 indicate the total number of the bytes of the contents within the corresponding files.

The actual number of data blocks each file uses is:

File -- Block Count

5_bytes.txt -- 1 block
6_bytes.txt -- 1 block
4096_bytes.txt -- 1 block
8192_bytes.txt -- 2 blocks
9000_bytes.txt -- 3 blocks

The total count of blocks used by all the entries in the directory /tmp/trial is 8. As such, the total number of blocks used in terms of kilobytes is 8 x 4k = 32k. This is what the line total 32 makes known. Note that 1 kilo here refers to the value 1024 and not 1000.




ramesh@linux:/tmp/trial$ du *
4   4096_bytes.txt
4   5_bytes.txt
4   6_bytes.txt
8   8192_bytes.txt
12  9000_bytes.txt



du *  lists the directory entries along with their storage blocks (in terms of kilobytes). For instance, the file 6_bytes.txt actually takes up 4 kilobytes of actual storage space - as denoted in its first column. 

In short,  'ls -l' provides actual number of data bytes contained within a file while 'du' provides the block count consumed (= actually used) by the file from the filesystem.

Tuesday, June 14, 2011

Unix: Finding files or directories based on their timestamp or age using a resolution finer than of days

When you use the –mtime option, the typical find command’s granularity is in terms of days i.e. how many days old. To effectively get finer granularity with the find command  i.e. to be able to specify hours, minutes or even seconds, warrants an indirect approach.

Suppose I have a directory within which I would like to search for files/directories that are older or newer than 2 hours from current time:



$ ls -ltr
total 0
-rw-r--r-- 1 ramesh users 0 2010-05-15 18:34 file_1
-rw-r--r-- 1 ramesh users 0 2011-06-15 03:00 file_2
-rw-r--r-- 1 ramesh users 0 2011-06-15 09:16 file_5
-rw-r--r-- 1 ramesh users 0 2011-06-15 09:16 file_4
-rw-r--r-- 1 ramesh users 0 2011-06-15 09:16 file_3

**  The command ls -ltr  lists our directory entries with the later ones at the bottom.


Current time is:

$ date
Wed Jun 15 09:19:23 MYT 2011


I create a (dummy) reference file with a time stamp of 2 hours earlier which is 15-June-2011  07:19:23:

$ touch -t 201106150719.23 ref_timestamp_file
$ ls -ltr
total 0
-rw-r--r-- 1 ramesh users 0 2010-05-15 18:34 file_1
-rw-r--r-- 1 ramesh users 0 2011-06-15 03:00 file_2
-rw-r--r-- 1 ramesh users 0 2011-06-15 07:19 ref_timestamp_file
-rw-r--r-- 1 ramesh users 0 2011-06-15 09:16 file_5
-rw-r--r-- 1 ramesh users 0 2011-06-15 09:16 file_4
-rw-r--r-- 1 ramesh users 0 2011-06-15 09:16 file_3


To find files and directories created after the reference time-stamp file i.e. created after 7:16 A.M. on 15-June-2011, use the newer option of find command:

$ find . -newer ref_timestamp_file
.
./file_5
./file_4
./file_3


And to get files which are older than the reference time-stamp file i.e. created before 7:16 A.M. on 15-June-2011, just negate the result of the find command:

$ find . ! -newer ref_timestamp_file
./file_2
./file_1
./ref_timestamp_file

Friday, March 18, 2011

Unix: Find PID of The Process Using A Port

You generally use lsof to list the files which are open in the system. However, with the option -i, lsof lists all of the processes with the sockets/ports which they are using.

lsof -i

To find the PID of the process holding up a port:

lsof -i | grep  port_number

This command comes in handy when you want to start-up a process that needs to bind to a particular port, but another running process is holding up that port. You can use lsof to find the PID of that process and kill it, should you have the permissions to do so.

Tuesday, March 8, 2011

Unix: Using Standard Input & Output With Tar: Copying File & Directory Trees


Aside from its use to archive files/directories into archive media like tapes, magneto-optical disks etc., tar is also used to bundle-up multiple files and directories with their contents into a single tarfile.. The tarfile is also referred to as a tar-bundle or more popularly, a tarball. The way we usually run tar to create a tarball is simply like so:

tar -cvf tar_filename/tarball_name list_of_files_&_directories

Note:
* The -v option is ....well, optional  :-) It's just to make tar give you updates on what it is doing as it progresses.
* It is customary to suffix the tarball name with ".tar"

Instead of specifying a file name as a target to create a tarball as in the above command, you could specify a dash, "-",  in place of the tarball name to output directly to standard output, rather than to a file. With the archiving instruction/switch, -c, tar interprets "-"  as standard output.You can try this command on the shell to see what I mean:

tar -cvf - /etc/passwd /etc/hosts /tmp

Running the above command, you get a fast scrolling output on the screen (default destination of standard output) of the files /etc/passwd, /etc/hosts along with the directory /tmp and all of its contents and their contents recursively. A tarball 'stream' is being created and directed to standard output, at the same time. Assuming you have not redirected the standard-output, the net end-effect of the above command on what you see on the screen, is similar to what this command will produce:

tar -tvf  /tmp/temptarfile /etc/passwd /etc/hosts /tmp  && cat /tmp/temptarfile && rm /tmp/temptarfile

It is unchangeable reality - what gets tarballed, gets un-tarred. :-) To un-tar a tarball file, we usually do this: 

tar -xvf tarball

When we used '-' with the -c switch in tar it is regarded as standard output. Conversely, if you specify '-' with the -x switch to un-tar a tarball, the '-' is taken to be as standard input. Knowing this, the above command's result can also be obtained with the following:

cat tarball | tar -xvf  -

The first process which runs cat, reads out the tarball file to standard output. By virtue of the pipe "|", that process' standard output is chained to the standard input of the second process in which runs the "tar -xvf" command. Because of the "-", tar here reads its input from standard input, which is the cat command's output. And that output is a tarball stream.

We have seen the archive and un-tar operations separately. Combining them, and with the "-", we could use the tar utility to copy files/directories from disparate locations to another location within a host. Working with the original example, we could do the following:

tar -cvf - /etc/passwd /etc/hosts /tmp | tar -xvf - 

The above command, copies the files /etc/passwd & /etc/hosts and the directory tree /tmp into the current directory. Execute the command while being in your home directory.

Note:
* You can get an almost similar result to the above, with the 'cp' command used like so:

   cp -r /etc/passwd /etc/hosts /tmp .

There is a difference between copying with the tar and and with the cp command. Run them both and compare the results.

All the preceding examples, un-tars in the current directory. Should you wish to un-tar in another directory, you could do either:

tar -cvf - /etc/passwd /etc/hosts /tmp | tar -C my_directory -xvf -

or this:

tar -cvf - /etc/passwd /etc/hosts /tmp | { cd my_directory; tar -xvf -; }

Note:
* In relation to the above command, if your current working directory is not 'my_directory' and you wish it to remain so, you contain the 'cd' and 'tar -xvf' commands within curly braces { }. This ensures the 'cd my_directory' and 'tar -xvf  -'  commands get executed in a sub-shell. As a result, when the command is completed, your current working directory remains unchanged in the shell from where you invoked the commands.

Tuesday, February 15, 2011

Perl: Killing Multiple Processes

This is done by getting a list of the user owned process' IDs and piping it into a perl one-line program as follows:  

ps -u username perl -ane 'kill 9, $F[0]'

What the above command string does is quite simple. We obtain the list of processes owned by username with: 

ps -u username

A sample output from my Linux host is as below:

  PID TTY          TIME CMD
 1414 ?        00:00:00 gnome-keyring-d
 1432 ?        00:00:00 gnome-session
 1466 ?        00:00:00 ssh-agent
 1469 ?        00:00:00 dbus-launch
 1470 ?        00:00:04 dbus-daemon
 1473 ?        00:00:03 gconfd-2
 1480 ?        00:00:05 gnome-settings-
 1482 ?        00:00:00 gvfsd

The entire list of lines is then piped into the perl program:

perl -ane 'kill 9, $F[0]'

What the 'short-form' Perl command above does is functionally equivalent to the 'expanded' Perl program below: 

while (<>) {
  @F = split; 
  kill 9,$F[0]; 
}

The options and their contribution to the code construct above, is explained in matching colour text here:

-e The Perl executable statement
-n Envelopes the entire Perl executable in a while-loop 
-a Ensures each line read from standard input is split on the spaces, into individual fields, on each iteration of the while-loop. The fields are stored in the array @F. If we consider the line #2 from the sample out of 'ps -u', we will have: 
                                                                    $F[0] = 1414
                                                                    $F[1] = ?
                                                                    $F[2] = 00:00:00
                                                                    $F[3] =  gnome-keyring-d
Since the process ID always gets stored in $F[0], we use that within the 'kill' command.


N.B.  
  • You would have noticed the first line of the output from the ps command is the header. As such, $F[0] would be 'PID' in this case. Perl's kill command will throw a warning for this, which can be safely ignored. Should you wish to avoid the warning altogether with a 'cleaner' run of the command, you ensure only numbers are extracted with the if construct:
ps -u username | perl -ane 'kill 9,$F[0] if $F[0]=~/^\d+/'

Perl: Command Options

Perl is a very-high-level programming language. Although famous for its expressiveness in big programming tasks, it is equally powerful within the limited confines of the Unix command line. With just a few command invocation options of Perl, we could perform tasks in similar succinct ways as provided by the traditional Unix shell utilities much like SED, AWK and so on. We explore the various ways of calling Perl on the command line.

Presented here is a side-by-side mapping of each of several Perl command line constructs to its expanded form. The colours match-up a command line option to its construct within the expanded Perl code form.


Command line Perl                                                                                         Expanded Perl Form

perl -n -e 'perl statements'                                      while (<>) {
                                                 perl statement(s)
                                               }
 
perl -p -e 'perl statements'                   while (<>) {
                                                 perl statement(s)
                                                 print;

                                               }
perl -a -n -e 'perl statements'                while (<>) { 
                                                 @F = split;
                                                 perl statement(s)
                                               } 


perl -a -p -e 'perl statements'                while (<>) {
                                                 @F = split;                                                  perl statement(s)
                                                 print;

                                               }

perl -a -F: -p -e 'perl statements'            while (<>) {
                                                 @F = split ':', $_;                                                  perl statement(s)
                                                 print;

                                               }

Legend:

-e The Perl executable statement
-n Envelopes the entire Perl executable in a while-loop 
-a Ensures each line read from standard input is split on spaces into individual fields, within each iteration of the while-loop. The fields are stored in the array @F. We access each field using Perl's array element access notation $F[0], $F[1] etc.
 -F   Specify a character at which split is made, when used with the -a option.
-p The effect here is identical to that of the  '-n' option but a 'print' statement, which prints out $_ at the end of each pass of the while-loop, is included.