[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

22.2 Implementation

By their very nature, a sizeable part of the functionality of shell scripts, is provided by the many utility programs that they routinely call to perform important subsidiary tasks. Addressing the portability of the script involves issues of portability in the host operating system environment, and portability of the utility programs as well as the portability of the shell implementation itself.

This section discusses differences between shell implementations to which you must cater when writing a portable script. It is broken into several subsections, each covering a single aspect of shell programming that needs to be approached carefully to avoid pitfalls with unexpected behaviour in some shell implementations. The following section discusses how to cope with the host environment in a portable fashion. The last section in this chapter addresses the portability of common shell utilities.


[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

22.2.1 Size Limitations

Quite a lot of the Unix vendor implementations of the Bourne shell have a fixed buffer for storing command lines, as small as 512 characters in the worst cases. You may have an error akin to this:

 
$ ls -d /usr/bin/* | wc -l
sh: error: line too long

Notice that the limit applies to the expanded command line, not just the characters typed in for the line. A portable way to write this would be:

 
$ ( cd /usr/bin && ls | wc -l )
   1556

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

22.2.2 #!

When the kernel executes a program from the file system, it checks the first few bytes of the file, and compares them with its internal list of known magic numbers, which encode how the file can be executed. This is a similar, but distinct, system to the ‘/etc/magic’ magic number list used by user space programs.

Having determined that the file is a script by examining its magic number, the kernel finds the path of the interpreter by removing the ‘#!’ and any intervening space from the first line of the script. One optional argument is allowed (additional arguments are not ignored, they constitute a syntax error), and the resulting command line is executed. There is a 32 character limit to the significant part of the ‘#!’ line, so you must ensure that the full path to the interpreter plus any switches you need to pass to it do not exceed this limit. Also, the interpreter must be a real binary program, it cannot be a ‘#!’ file itself.

It used to be thought, that the semantics between different kernels’ idea of the magic number for the start of an interpreted script varied slightly between implementations. In actual fact, all look for ‘#!’ in the first two bytes – in spite of commonly held beliefs, there is no evidence that there are others which require ‘#! /’.

A portable script must give an absolute path to the interpreter, which causes problems when, say, some machines have a better version of Bourne shell in an unusual directory – say ‘/usr/sysv/bin/sh’. See () for a way to re-execute the script with a better interpreter.

For example, imagine a script file called ‘/tmp/foo.pl’ with the following first line:

 
#! /usr/local/bin/perl

Now, the script can be executed from the ‘tmp’ directory, with the following sequence of commands:

 
$ cd /tmp
$ ./foo.pl

When executing these commands, the kernel will actually execute the following from the ‘/tmp’ directory directory:

 
/usr/local/bin/perl ./foo.pl

This can pose problems of its own though. A script such as the one described above will not work on a machine where the perl interpreter is installed as ‘/usr/bin/perl’. There is a way to circumvent this problem, by using the env program to find the interpreter by looking in the user’s ‘PATH’ environment variable. Change the first line of the ‘foo.pl’ to read as follows:

 
#! /usr/bin/env perl

This idiom does rely on the env command being installed as ‘/usr/bin/env’, and that, in this example, perl can be found in the user’s ‘PATH’. But that is indeed the case on the great majority of machines. In contrast, perl is installed in ‘usr/local/bin’ as often as ‘/usr/bin’, so using env like this is a net win overall. You can also use this method to get around the 32 character limit if the path to the interpreter is too long.

Unfortunately, you lose the ability to pass an option flag to the interpreter if you choose to use env. For example, you can’t do the following, since it requires two arguments:

 
#! /usr/bin/env guile -s

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

22.2.3 :

In the beginning, the magic number for Bourne shell scripts used to be a colon followed by a newline. Most Unices still support this, and will correctly pass a file with a single colon as its first line to ‘/bin/sh’ for interpretation. Nobody uses this any more and I suspect some very new Unices may have forgotten about it entirely, so you should stick to the more usual ‘#! /bin/sh’ syntax for your own scripts. You may occasionally come across a very old script that starts with a ‘:’ though, and it is nice to know why!

In addition, all known Bourne compatible shells have a builtin command, ‘:’ which always returns success. It is equivalent to the system command /bin/true, but can be used from a script without the overhead of starting another process. When setting a shell variable as a flag, it is good practice to use the commands, : and false as values, and choose the sense of the variable to be ‘:’ in the common case: When you come to test the value of the variable, you will avoid the overhead of additional processes most of the time.

 
var=:
if $var; then
  foo
fi

The : command described above can take any number of arguments, which it will fastidiously ignore. This allows the ‘:’ character to double up as a comment leader of sorts. Be aware that the characters that follow are not discarded, they are still interpreted by the shell, so metacharacters can have unexpected effects:

 
$ cat foo
:
: echo foo
: `echo bar`
: `echo baz >&2'
$ ./foo
baz

You may find very old shell scripts that are commented using ‘:’, or new scripts that exploit this behavior in some esoteric fashion. My advice is, don’t: It will bite you later.


[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

22.2.4 ()

There are still a great number of shells that, like Steve Bourne’s original implementation, do not have functions! So, strictly speaking, you can’t use shell functions in your scripts. Luckily, in this day and age, even though ‘/bin/sh’ itself may not support shell functions, it is not too far from the truth to say that almost every machine will have some shell that does.

Taking this assumption to its logical conclusion, it is a simple matter of writing your script to find a suitable shell, and then feed itself to that shell so that the rest of the script can use functions with impunity:

 
#! /bin/sh

# Zsh is not Bourne compatible without the following:
if test -n "$ZSH_VERSION"; then
  emulate sh
  NULLCMD=:
fi

# Bash is not POSIX compliant without the following:
test -n "$BASH_VERSION" && set -o posix

SHELL="${SHELL-/bin/sh}"
if test x"$1" = x--re-executed; then
  # Functional shell was found.  Remove option and continue
  shift
elif "$SHELL" -c 'foo () { exit 0; }; foo' 2>/dev/null; then
  # The current shell works already!
  :
else
  # Try alternative shells that (sometimes) support functions
  for cmd in sh bash ash bsh ksh zsh sh5; do
    set IFS=:; X="$PATH:/bin:/usr/bin:/usr/afsws/bin:/usr/ucb"; echo $X`
    for dir
      shell="$dir/$cmd"
      if (test -f "$shell" || test -f "$shell.exe") &&
        "$shell" -c 'foo () { exit 0; }; foo 2>/dev/null
      then
        # Re-execute with discovered functional shell
        SHELL="$shell" exec "$shell" "$0" --re-executed ${1+"$@"}
      fi
    done
  done
  echo "Unable to locate a shell interpreter with function support" >&2
  exit 1
fi

foo () {
    echo "$SHELL: ta da!"
}

foo

exit 0

Note that this script finds a shell that supports functions of the following syntax, since the use of the function keyword is much less widely supported:

 
foo () { ... }

A notable exception to the assertion that all machines have a shell that can handle functions is 4.3BSD, which has only a single shell: a shell function deprived Bourne shell. There are two ways you can deal with this:

  1. Ask 4.3BSD users of your script to install a more featureful shell such as bash, so that the technique above will work.
  2. Have your script run itself through sed, chopping itself into pieces, with each function written to it’s own script file, and then feed what’s left into the original shell. Whenever a function call is encountered, one of the fragments from the original script will be executed in a subshell.

If you decide to split the script with sed, you will need to be careful not to rely on shell variables to communicate between functions, since each ‘function’ will be executed in its own subshell.


[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

22.2.5 .

The semantics of ‘.’ are rather peculiar to say the least. Here is a simple script – it just displays its positional parameters:

 
#! /bin/sh
echo "$0" ${1+"$@"}

Put this in a file, ‘foo’. Here is another simple script – it calls the first script. Put this in another file, ‘wrapper’:

 
#! /bin/sh
. ./foo
. ./foo bar baz

Observe what happens when you run this from the command line:

 
$ ./wrapper
./wrapper
./wrapper bar baz

So ‘$0’ is inherited from the calling script, and the positional parameters are as passed to the command. Observe what happens when you call the wrapper script with arguments:

 
$ ./wrapper 1 2 3
./wrapper 1 2 3
./wrapper bar baz

So the sourced script has access to the calling scripts positional parameters, unless you override them in the ‘.’ command.

This can cause no end of trouble if you are not expecting it, so you must either be careful to omit all parameters to any ‘.’ command, or else don’t reference the parameters inside the sourced script. If you are reexecuting your script with a shell that understands functions, the best use for the ‘.’ command is to load libraries of functions which can subsequently be used in the calling script.

Most importantly, don’t forget that, if you call the exit command in a script that you load with ‘.’, it will cause the calling script to exit too!


[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

22.2.6 [

Although technically equivalent, test is preferable to [ in shell code written in conjunction with Autoconf, since ‘[’ is also used for M4 quoting in Autoconf. Your code will be much easier to read (and write) if you abstain from the use of ‘[’.

Except in the most degenerate shells, test is a shell builtin to save the overhead of starting another process, and is no slower than ‘[’. It does mean, however, that there is a huge range of features which are not implemented often enough that you can use them freely within a truly portable script. The less obvious ones to avoid are ‘-a’ and ‘-o’ – the logical ‘and’ and ‘or’ operations. A good litmus test for the portability of any shell feature is to see whether that feature is used in the source of Autoconf, and it turns out that ‘-a’ and ‘-oare used here and there, but never more than once in a single command. All the same, to avoid any confusion, I always avoid them entirely. I would not use the following, for example:

 
test foo -a bar

Instead I would run test twice, like this:

 
test foo && test bar

The negation operator of test is quite portable and can be used in portable shell scripts. For example:

 
if test ! foo; then bar; fi

The negation operator of if is not at all portable and should be avoided. The following would generate a syntax error on some shell implementations:

 
if ! test foo; then bar; fi

An implication of this axiom is that when you need to branch if a command fails, and that command is not test, you cannot use the negation operator. The easiest way to work around this is to use the ‘else’ clause of the un-negated if, like this:

 
if foo; then :; else bar; fi

Notice the use of the : builtin as a null operation when foo doesn’t fail.

The test command does not cope with missing or additional arguments, so you must take care to ensure that the shell does not remove arguments or introduce new ones during variable and quote expansions. The best way to do that is to enclose any variables in double quotes. You should also add a single character prefix to both sides in case the value of the expansion is a valid option to test:

 
$ for foo in "" "!" "bar" "baz quux"; do
>   test x"$foo" = x"bar" && echo 1 || echo 0
> done
0
0
1
0

Here, you can see that using the ‘x’ prefix for the first operand saves test from interpreting the ‘!’ argument as a real option, or from choking on an empty string – something you must always be aware of, or else the following behaviour will ensue:

 
$ foo=!
$ test "$foo" = "bar" && echo 1 || echo 0
test: argument expected
0
$ foo=""
$ test "$foo" = "bar" && echo 1 || echo 0
test: argument expected
0

Also, the double quote marks help test cope with strings that contain whitespace. Without the double quotes, you will see this errors:

 
$ foo="baz quux"
$ test x$foo = "bar" && echo 1 || echo 0
test: too many arguments
0

You shouldn’t rely on the default behaviour of test (to return ‘true’ if its single argument has non-zero length), use the ‘-n’ option to force that behaviour if it is what you want. Beyond that, the other thing you need to know about test, is that if you use operators other than those below, you are reducing the portability of your code:

-nstring

string is non-empty.

-zstring

string is empty.

string1 = string2

Both strings are identical.

string1 != string2

The strings are not the same.

-dfile

file exists and is a directory.

-ffile

file exists and is a regular file.

You can also use the following, provided that you don’t mix them within a single invocation of test:

expression-aexpression

Both expressions evaluate to ‘true’.

expression-oexpression

Neither expression evaluates to ‘false’.


[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

22.2.7 $

When using shell variables in your portable scripts, you need to write them in a somewhat stylised fashion to maximise the number of shell implementations that will interpret your code as expected:


[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

22.2.8 * versus .*

This section compares file globbing with regular expression matching. There are many Unix commands which are regularly used from shell scripts, and which provide some sort of pattern matching mechanism: expr, egrep and sed, to name a few. Unfortunately they each have different quoting rules regarding whether particular meta-characters must be backslash escaped to revert to their literal meaning and vice-versa. There is no real logic to the particular dialect of regular expressions accepted by these commands. To confirm the correctness of each regular expression, you should always check them from the shell prompt with the relevant tool before committing to a script, so I won’t belabour the specifics.

Shell globbing however is much more regular (no pun intended), and provides a reasonable and sometimes more cpu efficient solution to many shell matching problems. The key is to make good use of the case command, which is easier to use (because it uses globbing rules) and doesn’t require additional processes to be spawned. Unfortunately, GNU Bash doesn’t handle backslashes correctly in glob character classes – the backslash must be the first character in the class, or else it will never match. For example, if you want to detect absolute directory paths on Unix and Windows using case, you should write the code like this:

 
case $dir in
  [\\/]* | ?:[\\/]* ) echo absolute ;;
  * )                 echo relative ;;
esac

Even though expr uses regular expressions rather than shell globbing, it is often(52) a shell builtin, so using it to extract sections of strings can be faster than spawning a sed process to do the same. As with echo and set, for example, you must be careful that variable or command expansions for the first argument to expr are not accidentally interpreted as reserved keywords. As with echo, you can work around this problem by prefixing any expansions with a literal ‘x’, as follows:

 
$ foo=substr
$ expr $foo : '.*\(str\)'
expr: syntax error
$ expr x$foo : '.*\(str\)'
str

[ < ] [ > ]   [ << ] [ Up ] [ >> ]

This document was generated by Ben Elliston on July 10, 2015 using texi2html 1.82.