Dashed to Shreds

Every once and a while, I write something that makes me feel like a bad person. I’m not talking about an angry text message, or a comment on someting online, or even some sort of opinionated blog post. No, when this happens, it’s most likely because I’ve been writing something in perl or as it was today, in POSIX-Compliant Shell Script.

Specifically, today I wrote this gem (context and comments removed for emphasis):

_ARGS=
_ARGI=0
for arg in "$@"; do
  eval "_A$_ARGI=\$arg"
  _ARGS="$_ARGS \\"\$_A$_ARGI\\""
  _ARGI=`expr $_ARGI + 1`
done
eval exec $_ARGS

This is part of the new catkin_tools package which provides a new top-level interface to the Catkin buildsystem that we all love to hate. It provides a new multi-command program simply called catkin which will supercede catkin_make and catkin_make_isolated, but that’s a subject for another time.

This tool needs to so some environment manipulation, and in order to play well with the standard Catkin infrastructure, it needed to accept a command with arguments, inject some environment, and then execute the given command.

Just accept that this is what they want to do. If you can’t accept this, then please read all of the catkin documentation until you no longer care any more, and just want your code to build without surprises.

Previously, the full script in catkin_tools looked like the following:

build_env.sh

#!/usr/bin/env sh
# generated from within catkin_tools/verbs/catkin_build/common.py

if [ $# -eq 0 ] ; then
  /bin/echo "Usage: build_env.sh COMMANDS"
  /bin/echo "Calling build_env.sh without arguments is not supported anymore."
  /bin/echo "Instead spawn a subshell and source a setup file manually."
  exit 1
fi

# save original args for later
_ARGS=$@
# remove all passed in args, resetting $@, $*, $#, $n
shift $#
# set the args for the sourced scripts
set -- $@ "--extend"
# source setup.sh with implicit --extend argument for each direct build depend in the workspace
{sources}

# execute given args
exec $_ARGS

Where the {sources} but was replaced with “dot” commands to source some other shell scripts.

The problem with this is that pure POSIX shells like dash (the default on Ubuntu Linux) don’t support array types in shell scripts. This means that the $_ARGS variable ends up being one long string variable. Unfortunately, this means that no matter what you do to escape or quote the commands, they get passed to the exec directive at the bottom of the script as a set of whitespace-delimited arguments.

This is fine, until you want to have any arguments with whitespace in them, even if the whitespace is escaped. This problem is documented in catkin_tools issues #23 and #74. No matter what was done in the rest of the codebase, this fundamental limitation of the build_env.sh script prevented any arguments with whitespace in them to be executed with the injected environment (as noted in catkin_tools pull request #29.

Interestingly enough, when iterating through the initial list of arguments $@, the tokens are separated properly. There’s just no way to store them that way in an array without using non-POSIX features.

After a bunch of searching, I came across a similar problem where GrapefruiTgirl posted a tutorial on linuxquestions.org about How to do arrays, without using real arrays. This tutorial gives an example of using the eval directive to declare variables programmatically with POSIX-compliant features only. This means that to define the variable $_Ai where i is some number, you can do the following (for i=42 for example):

i=42
eval "_A$i='Hello world.'"
echo "$_A42" # output: Hello world.

This means that we can create a variable for each argument in $@ and keep it separated. This can be done like so, storing each argument in a meta-variable named $_ARGi where i is the index of the argument:

_ARGI=0
for arg in "$@"; do
  eval "_A$_ARGI=\$arg"
  _ARGI=`expr $_ARGI + 1`
done

The only remaining issue is how to extract them out and pass them into the call to exec. Unfortunately, you can’t just compose the arguments into a string, because we’d encounter the same problem as before.

Enter eval again. However, we can’t just call eval on the concatenation of the meta-variables. In order for exec to treat arguments as atomic tokens and not separate them into smaller whitespace-delimited chunks, the strings need to be double-quoted and properly escaped, which gives the following:

_ARGS=
_ARGI=0
for arg in "$@"; do
  eval "_A$_ARGI=\$arg"
  _ARGS="$_ARGS \\"\$_A$_ARGI\\""
  _ARGI=`expr $_ARGI + 1`
done
# ...
# $_ARGS="\"$_A0\" \"$_A1\" \"$_A2\" \"$_A3\" ..."
eval exec $_ARGS

The whole resulting script (which was part of catkin_tools pull request #109) ends up looking like this (with some commented-out debug lines removed):

build_env.sh

#!/usr/bin/env sh
# generated from within catkin_tools/verbs/catkin_build/common.py

if [ $# -eq 0 ] ; then
  /bin/echo "Usage: build_env.sh COMMANDS"
  /bin/echo "Calling build_env.sh without arguments is not supported anymore."
  /bin/echo "Instead spawn a subshell and source a setup file manually."
  exit 1
fi

# save original args for later
_ARGS=
_ARGI=0
for arg in "$@"; do
  # Define placeholder variable
  eval "_A$_ARGI=\$arg"
  # Add placeholder variable to arg list
  _ARGS="$_ARGS \\"\$_A$_ARGI\\""
  # Increment arg index
  _ARGI=`expr $_ARGI + 1`
done

# remove all passed in args, resetting $@, $*, $#, $n
shift $#
# set the args for the sourced scripts
set -- $@ "--extend"
# source setup.sh with implicit --extend argument for each direct build depend in the workspace
{sources}

# execute given args
eval exec $_ARGS

I hope you enjoyed the details of the birth of the travesty that is build_env.sh.