Inlab 5 unix bourne Shell Scripting(1)

Unix lab 9 UNIX Bourne Shell Scripting

Shell scripting involves chaining several UNIX commands together to accomplish a task. For example, you might run the 'date' command and then use today's date as part of a file name. I'll show you how to do this below.

Some of the tools of the trade are variables, backquotes and pipes. First we'll study these topics and also quickly review a few other UNIX topics.

Variables
Topics covered: storing strings in variables

Utilities covered: echo, expr

After we log in to remote1.unl.csi.cuny.edu, a Bourne shell is started up automatically. (If shell is not started, you can use command /bin/sh to start it.)

A variable stores a string (try running these commands in a Bourne shell)

(1) Step 1: type the following command:

name="John"

echo $name
What do you get?
Now type the following command:

echo name

What do you get?
Now type the following command:

name="John Doe"

echo $name

What do you get?

Now type the following command:

name=John Doe

What do you get?

The quotes are required in the example above because the string contains a special character (the space)

(2) Step 2:

A variable may store a number

num=137

The shell stores this as a string even though it appears to be a number

A few UNIX utilities will convert this string into a number to perform arithmetic

Now type the following command:

expr $num + 3 (make sure you have space before + and 3)

What do you get?

Try defining num as '7m8' and try the expr command again.What happens when num is not a valid number?

I/O Redirection

Topics covered: specifying the input or capturing the output of a command in a file

Utilities covered: wc, sort

(3) Step 3:

The wc command counts the number of lines, words, and characters in a file

wc /etc/passwd

wc -l /etc/passwd

What do you get from these two commands?

You can save the output of wc (or any other command) with output redirection

wc /etc/passwd > wc.file
then type the command: cat wc.file

What is the result?

(4) Step 4:

You can specify the input with input redirection

wc < /etc/passwd

What is the output?

Many UNIX commands allow you to specify the input file by name or by input redirection

sort /etc/passwd

sort < /etc/passwd

You can also append lines to the end of an existing file with output redirection

wc -l /etc/passwd >> wc.file

then type the command cat wc.file, what do you get?

Backquotes
Topics covered: capturing output of a command in a variable

Utilities covered: date

The backquote character looks like the single quote or apostrophe, but slants the other way

It is used to capture the output of a UNIX utility

A command in backquotes is executed and then replaced by the output of the command

(5) Step 5:

Execute these commands

date

save_date=`date`

echo The date is $save_date

What is the output?

Notice how echo prints the output of 'date', and gives the time when you defined the save_date variable

(6) Step 6:

Store the following in a file named backquotes.sh and execute it (right click and save in a file)

#!/bin/sh

# Illustrates using backquotes

# Output of 'date' stored in a variable

Today="`date`"

echo Today is $Today

Execute the script with

bash backquotes.sh

What is the output?

The example above shows you how you can write commands into a file and execute the file with a Bourne Again shell

Backquotes are very useful, but be aware that they slow down a script if you use them hundreds of times

(7) Step 7:

You can save the output of any command with backquotes. Try this:

LS=`ls -l`

echo $LS
Pipes

Topics covered: using UNIX pipes

Utilities covered: sort, cat, head

Pipes are used for post-processing data

One UNIX command output is introduced as input to another command

(8) Step 8: type the following command:

sort /etc/passwd | head -5

What is the output?

Notice that this pipe can be simplified

cat /etc/passwd | head -5

What is the output for this?

You could accomplish the same thing more efficiently with either of the two commands:

head -5 /etc/passwd

head -5 < /etc/passwd

(9) Step 9:

The command displays all the files in the current directory sorted by file size

ls -al | sort -n -r +4

What is the output? What are the outputs sorting according to? Which column is this?

The command ls -al writes the file size in the fifth column, which is why we skip the first four columns using +4.

The options -n and -r request a numeric sort (which is different than the normal alphabetic sort) in reverse order

awk
Topics covered: processing columnar data

Utilities covered: awk

The awk utility is used for processing columns of data

(10) Step 10:

The following example shows how to extract column 5 (the file size) from the output of ls -l

ls -l | awk '{print $5}'

What is the output?

Cut and paste this line into a Bourne shell and you should see a column of file sizes, one per file in your current directory.

(11) Step 11:

The following example shows how to sum the file sizes and print the result at the end of the awk run

ls -al | awk '{sum = sum + $5} END {print sum}'

In this example you should see printed just one number, which is the sum of the file sizes in the current directory.

Shell Scripts
Topics covered: storing commands in a file and executing the file

Utilities covered: date, cal, last (shows who has logged in recently)

(12) Step 12:

Store the following in a file named simple.sh and execute it

#!/bin/sh

# Show some useful info at the start of the day

date

echo Good morning $USER

cal

last | head -6

What is the result after you execute it?(Shows current date, calendar, and a six of previous logins Notice that the commands themselves are not displayed, only the results )

last command - display login and logout information about users and

terminals

(13) Step 13:

To display the commands verbatim as they run, execute with

sh -v simple.sh

Another way to display the commands as they run is with -x

sh -x simple.sh

What is the difference between -v and -x? Notice that with -v you see '$USER' but with -x you see your login name

Run the command 'echo $USER' at your terminal prompt and see that the variable $USER stores your login name

With -v or -x (or both) you can easily relate any error message that may appear to the command that generated it

(14) Step 14:

When an error occurs in a script, the script continues executing at the next command

Verify this by changing 'cal' to 'caal' to cause an error, and then run the script again

Run the 'caal' script with 'sh -v simple.sh' and with 'sh -x simple.sh' and verify the error message comes from cal

Other standard variable names include: $HOME, $PATH, $PRINTER. Use echo to examine the values of these variables

Storing File Names in Variables
Topics covered: variables store strings such as file names, more on creating and using variables

Utilities covered: echo, ls, wc

A variable is a name that stores a string

It's often convenient to store a filename in a variable

Step 1: Store the following in a file named variables.sh and execute it

#!/bin/sh

# An example with variables

filename="/etc/passwd"

echo "Check the permissions on $filename"

ls -l $filename

echo "Find out how many accounts there are on this system"

wc -l $filename
What is the output?
Now change the value of $filename to /etc/link, the change is automatically propagated throughout the entire script

What is the output?

Scripting With sed
Topics covered: global search and replace, input and output redirection

Utilities covered: sed

Step 2: Here's how you can use sed to modify the contents of a variable:

echo "Hello Jim" | sed -e 's/Hello/Bye/'

Step 3: create a file named nlanr.txt with the following contents:

The National Laboratory for Applied Network Research (NLANR) is a

collaboration among NSF-supported supercomputer sites. NLANR was

created in 1995 to provide technical and engineering support and

overall coordination of the vBNS connections at the five NSF-supported

supercomputer centers. The vBNS has evolved to become a "leading edge

but stable" platform to enable the development and use of high

performance applications by the broader academic research community.

Consequently, NLANR's focus has expanded.

notice how the word 'vBNS' appears in it several times

Step 4: Change 'vBNS' to 'NETWORK' with the following command:

sed -e 's/vBNS/NETWORK/g' < nlanr.txt

open the file nlanr.txt, what do you find?

Step 5: open file nlanr.txt, and change NETWORK back to vBNS, then type the following command:

sed -e 's/vBNS/NETWORK/g' < nlanr.txt > nlanr.new

open the file nlanr.new, what do you find?

Sed can be used for many complex editing tasks, we have only scratched the surface here

Performing Arithmetic
Topics covered: integer arithmetic, preceding '*' with backslash to avoid file name wildcard expansion

Utilities covered: expr

Arithmetic is done with expr

Step 6: type the following command:

expr 5 + 7

expr 5 \* 7

Backslash required in front of '*' since it is a filename wildcard and would be translated by the shell into a list of file names

You can save arithmetic result in a variable

Step 7: Store the following in a file named arith.sh and execute it

#!/bin/sh

# Perform some arithmetic

x=24

y=4

Result=`expr $x \* $y`

echo "$x times $y is $Result"

What is the output?
Translating Characters
Topics covered: converting one character to another, translating and saving string stored in a variable

Utilities covered: tr

Step 8: create a file with the name of sdsc.txt with the following contents:

San Diego Supercomputer Center
A National Laboratory for Computational Science and Engineering
The San Diego Supercomputer Center (SDSC) -- a campus research unit of

the University of California, San Diego -- is the focus of

computational activities within NPACI. In operation since 1986, SDSC

has provided the national research community with access to the

highest performance computers available; conducted applications and

technology research/development projects; trained researchers,

educators, and students; and facilitated technology transfer to

industry and government. Within NPACI, SDSC will broaden its focus to

include partnership with 36 other academic and research institutions.

Step 9: type the following command(The utility tr translates characters):

tr 'a' 'Z' < sdsc.txt

This example shows how to translate the contents of a variable and display the result on the screen with tr

Step 10: Store the following in a file named tr1.sh and execute it

#!/bin/sh

# Translate the contents of a variable

Cat_name="Piewacket"

echo $Cat_name | tr 'a' 'i'
what is the output?

This example shows how to change the contents of a variable

Step 11: Store the following in a file named tr2.sh and execute it

#!/bin/sh

# Illustrates how to change the contents of a variable with tr

Cat_name="Piewacket"

echo "Cat_name is $Cat_name"

Cat_name=`echo $Cat_name | tr 'a' 'i'`

echo "Cat_name has changed to $Cat_name"

You can also specify ranges of characters.

Step 12: Type the following commands and see what happened?(This example converts upper case to lower case)

tr 'A-Z' 'a-z' < file

Now you can change the value of the variable and your script has access to the new value

Processing Multiple Files

Topics covered: executing a sequence of commands on each of several files with for loops

Utilities covered: no new utilities

Step 13: Store the following in a file named loop1.sh and execute it

#!/bin/sh

# Execute ls and wc on each of several files

# File names listed explicitly

for filename in simple.sh variables.sh loop1.sh

do

echo "Variable filename is set to $filename..."

ls -l $filename

wc -l $filename

done

This executes the three commands echo, ls and wc for each of the three file names

You should see three lines of output for each file name

filename is a variable, set by "for" statement and referenced as $filename

Now we know how to execute a series of commands on each of several files

Using File Name Wildcards in For Loops

Topics covered: looping over files specified with wildcards

Utilities covered: no new utilities

Step 14: Store the following in a file named loop2.sh and execute it

#!/bin/sh

# Execute ls and wc on each of several files

# File names listed using file name wildcards

for filename in *.sh

do

echo "Variable filename is set to $filename..."

ls -l $filename

wc -l $filename

done

You should see three lines of output for each file name ending in '.sh'

The file name wildcard pattern *.sh gets replaced by the list of filenames that exist in the current directory

Step 15: For another example with filename wildcards try this command

echo *.sh
Search and Replace in Multiple Files
Topics covered: combining for loops with utilities for global search and replace in several files

Utilities covered: mv

Step 16: type the following command (Sed performs global search and replace on a single file):

sed -e 's/application/APPLICATION/g' sdsc.txt > sdsc.txt.new

open file sdsc.txt.new and see what are changed?

The original file sdsc.txt is unchanged

How can we arrange to have the original file over-written by the new version?

Step 17: Store the following in a file named s-and-r.sh and execute it

#!/bin/sh

# Perform a global search and replace on each of several files

# File names listed explicitly

for text_file in sdsc.txt nlanr.txt

do

echo "Editing file $text_file"

sed -e 's/application/APPLICATION/g' $text_file > temp

mv -f temp $text_file

done

First, sed saves new version in file 'temp'

Then, use mv to overwrite original file with new version

Accessing Command-line Arguments

Topics covered: accessing command-line arguments

Utilities covered: no new utilities

Step 18: Store the following in a file named args1.sh

#!/bin/sh

# Illustrates using command-line arguments

# Execute with

# sh args1.sh On the Waterfront

echo "First command-line argument is: $1"

echo "Third argument is: $3"

echo "Number of arguments is: $#"

echo "The entire list of arguments is: $*"

Execute the script with

sh args1.sh -x On the Waterfront

Words after the script name are command-line arguments

Arguments are usually options like -l or file names

Looping Over the Command-line Arguments
Topics covered: using command-line arguments in a for loop

Utilities covered: no new utilities

Step 19: Store the following in a file named args2.sh and execute it

#!/bin/sh

# Loop over the command-line arguments

# Execute with

# sh args2.sh simple.sh variables.sh

for filename in "$@"

do

echo "Examining file $filename"

wc -l $filename

done

This script runs properly with any number of arguments, including zero

The shorter form of the for statement shown below does exactly the same thing

for filename

do

...

Don't use

for filename in $*

Fails if any arguments include spaces

Also, don't forget the double quotes around $@

The read Command
Topics covered: reading a line from the standard input

Utilities covered: no new utilities

stdin is the keyboard unless input redirection used

Step 20: type the following command(Read one line from stdin, store line in a variable):

read variable_name

Step 20: Store the following in a file named read.sh and execute it (This file is to ask the user if he wants to exit the script):

#!/bin/sh

# Shows how to read a line from stdin

echo "Would you like to exit this script now?"

read answer

if [ "$answer" = y ]

then

echo "Exiting..."

exit 0

fi