Skip to main content Skip to navigation

5. Shell scripting

WARNING: You may find that some commands that 'shouldn't work' under bash DO ACTUALLY WORK for your shell in the linux labs. Unfortunately there are some variations in support for some bash features. You should abide by the material in the following section to pass questions about bash scripting in the online test, NOT run the commands yourself to see if you get the correct answer! The idea here is that you will be able to write shell-scripts that will safely run on ANY machine.

1. Introduction

It's quite likely during in the course of your scientific computing career that you will type many commands at the command line that are pretty much the same - e.g. set environment variables, run job, copy output to a special location, extract data from output....and so on. Rather than actually type these commands in every time you want to go through this procedure, there is a way to collect these commands into something called a shell script. Then, each time you can just type one command (that basically equates to 'run shell-script').

This shell-script will run each of the commands typed into the script, just like you had typed them onto the command line yourself. This tutorial is based on bash.

The first line that should always appear at the very top of your script is:

 #!/bin/bash

This line ensures that the script is run in the bash shell, regardless of what shell you actually prefer to use at the command line. All subsequent lines should be commands that you would like to be issued to the shell. You can include comment statements in shell scripts by using the hash (#) symbol - anything to the right of a hash symbol on a line is interpreted as a comment.

Putting it into Practice:

Here is a trivial first script, that includes some echo commands and the ls command. Using emacs, open a file called myscript.sh, and type in the following:
 
#!/bin/bash

echo "hello, $USER. Here is a list of your files"
echo "The files are contained in $PWD"
#now list the files
ls -lrt
Save the file and quit out of emacs. You now have a file called myscript.sh, but it is not ready to be run yet. You must first change the permissions (see Section 1.7 to refresh your memory about file permissions) such that the file is executable.
Do this by typing on the command line
 chmod +x myscript.sh
Now run your script by typing the following at the command line and hitting return
./myscript.sh


Back to Top

2. Shell variables

The shell variables used in your shell script need not be environment variables. You can define variables in your script. For example, let's set the variable CST to be equivalent to castep. To do this, you would include a line in your script as follows:

 CST="castep"

Note that there is no whitespace on either side of the equals sign - this is important, as your variable assignment won't work if the whitespace is there. It's a good idea to use double quotes, since in general this will protect whitespace in your variable (even though we don't need it in this case). Now every time you wish to refer to this variable in your script, you must refer to it as

 $CST

Putting it into Practice:

Modify your existing script, myscript.sh, so that you declare this variable and echo it to your x-term . In your script, modify it to look like the source below:
 
#!/bin/bash

CST="castep"

echo "hello, $USER. Here is a list of your files"
echo "The files are contained in $PWD"
#now list the files
ls -lrt 
echo $CST
Run the script to see if your variable expansion works. It should print castep to your xterm, not $CST.
NB: You don't need to run chmod again, - the file stays executable upon modification.

Now suppose that you wish to refer to several files in your script that begin with castep. such as castep_cell and castep_param. Taking the CST variable from the last exercise, you can refer to these files by putting braces {} around your variable. Otherwise 'bash' interprets the whole part after the $ as a variable name (when it is not separated by, say, a space or a dot).

Putting it into Practice:

Modify your existing script, myscript.sh, so that you can refer to the file castep.cell and castep.param:
 
#!/bin/bash

CST="castep"

echo "hello, $USER. Here is a list of your files"
echo "The files are contained in $PWD"
#now list the files
ls -lrt 
echo "${CST}_cell"
echo "${CST}_param"
Run the script to see if your variable expansion works. It should print castep_cell and castep_param to your xterm
You should change these lines to see what happens if you try to refer to, say
 echo "$CST_cell"
What happens when you run a script with a line like the one above?
This will not work for all variants of bash - so be careful if running this on an unfamiliar machine!


Back to Top

3. Special characters and quotation marks

As we saw before we can use wildcards to look for certain files. For example, the command ls a*b shows all files in the current directory that start with an a and end with an b. This is often very convenient. The only drawback is that * now becomes a special character. Likewise for '?'. If you do echo ? then bash will look for all files that have a single-character filename, such as 'f' and outputs all those filenames to the screen. If it did not find any of such files, it will just prints the question mark to the screen. There are more characters with a special meaning in bash, such as ?, |, &, #, ;, ", ', \, and /.

There are three options to still be able to use these characters in a different setting:

  • Use a backslash \ in front of the character. This is called escaping the character. For example, to escape * in an echo statement one can use:
    echo \*
    This will suppress the special behaviour of the character and instead it will be used literally -- the * is displayed on the screen, instead of all filenames.
  • Enclose the whole expression with double quotes. This will only negate the effect of some special characters. Whitespace, for example, is not considered any more as a mean to separate commands. But variable substitution will still take place. An example is
    echo "Hello $USER, 3 * 3 = 9"
    This will show the name of the current user, but it will not expand the *
  • Enclose the whole expression with single quotes. This will treat all characters in between as non-special. Even variable substitution does not take place any more. An example is
    echo 'A literal text with non-working variable substitution $USER'

Putting it into Practice:

Type in the following two lines in the bash shell
touch awkward!
echo This is awkward?

What do you see on the screen? Is this what you expect? Fix it in three different ways, if your aim was to display the text literally.


4. Logical operations

You can do logical operations in your shell script. One such construct is if then fi which follows the syntax

if condition
then
      execute line 1
      execute line 2
      ... etc ...
fi

The lines 1 and 2 will only execute if the condition is satisfied. The condition to be satisfied is usually evaluated with a test operation. Test operations should be enclosed in square brackets, with the following syntax

 [ operand1 operator operand2 ] 

notice the whitespace between the operands and brackets at either end. These must be present for the test operation to work. there also needs to be whitespace between each operand and operator.

Some very useful test operators are the following:

= the operands are equal and they are strings
!= the opposite of =, for strings
-eq the operands are integers and they are equal
-ne the opposite of -eq, for integers
-lt the operand1 is less than operand2 and they are integers
-gt the operand1 is greater than operand2 and they are integers
-le the operand1 is less than or equal to operand2 and they are integers
-ge the operand1 is greater than or equal to operand2 and they are integers

Putting it into Practice:

Using emacs create a new file called anything you like that ends in .sh, e.g junk.sh. Below is some trivial example code that makes use of logical operators. Try typing this in and running the script.
#!/bin/bash

X=3
Y=4

if [ $X -lt $Y ]
then

echo "$X is less than $Y"

fi
You will have to change the permissions to make this an executable file, if you didn't copy it from an existing executable shell script.


Back to Top

5. Loop structures

Loop structures can also be used in shell scripts. In bash, you have the choice of for loops and while loops.

In general, for loop constructs follow the syntax

for loop-items
do
      execute line 1
      execute line 2
      ... etc ...
done

The loop-items is over a list of space-separated items, see the example below for usage:

Example:

In this example shell-script, the loop will run over 3 items, printing out the items one by one
#!/bin/bash

echo 'Here are some characters from the Clangers'
echo

c1="major clanger"
c2="soup dragon"
c3="tiny clanger"
for X in "$c1" "$c2" "$c3"
do
	echo $X
done
after hitting return, the following would be printed at your x-term.
Here are some characters from the Clangers

major clanger
soup dragon
tiny clanger
 

The second loop construct is the while loop, which follows the syntax:

while test-condition
do
      execute line 1
      execute line 2
      ... etc ...
done

It is possible for your final execute line in the loop to be an iteration line - although it is a little clumsy to implement. The test condition should follow the syntax as observed for the if/fi conditionals, ie be enclosed in square brackets, have whitespace between brackets and operands, have whitespace between operands and operators, and use the same logical operators. See the example below for usage:

Example:

In this example shell-script, the loop will iterate from X=1 to X=10, printing out the value of X each time
#!/bin/bash
X=1
while [ $X -le 10 ]
do
	echo $X
        # below is the increment line
	X=$((X+1))
done
after hitting return, the follwing would be printed at your xterm.
1
2
3
4
5
6
7
8
9
10
 

Putting it into Practice:

Try creating and running the example scripts above, and try making your own changes to these files. Don't forget to ensure the permissions are set correctly.
 


Back to Top

6. Command substitution

One final useful feature of the bash shell is command substitution. With this feature, the output of a shell command can be set to a variable. There are two ways to implement command substitution, parentheses expansion or the older backtick expansion, as in the example below.

Here is an example of parentheses expansion:

X=$(ls -lrt)

Here is an example of backtick expansion

X=`ls -lrt`

The parentheses expansion has the benefit that it can be nested, e.g., x=$(grep hi $(ls))

See below for example usage

Example:

In this example shell-script, the output of two types of ls command will be set to variables and then echoed to the x-term
#!/bin/bash

files="$(ls -lrt)"
dot_files=`ls -lrt .*`
echo "$files"
echo $dot_files
The double quotation marks "..." in the first 'echo' statement ensures that the newlines are preserved. Without it 'echo' will remove them by default, as is the case in the second 'echo' statement.

When using integers as shell variables, it is also useful to sometimes combine these integers in various ways. This can be done using the expr command in bash. This command can evaluate an arithmetic expression involving integers. You should refer to a book on the bash shell for more information. See the example below for basic usage

Example:

In this example, the expression "3 times 2 plus 4" will be evaluated, set to a shell variable, and echoed to the x-term.
#!/bin/bash

X=`expr 3 \* 2 + 4`
echo $X
Notice the backslash before the *, this is because * is a special character. A backslash preceeding a special character means that the shell will interpret * as a real asterisk, and not as a wildcard substitution.


Back to Top