Skip to main content
7 ton shark

Parallelism with Semaphores in Bash

I was reviewing the build scripts for an old project and unearthed one of my favorite bash scripts, written back when I was learning new bash tricks every day. Nowadays I would write a lot of this kind of thing in TypeScript or some other more structured language, but there's still a place in my heart for scripting right in the terminal... so: parallelism in bash!

The goal is to run a series of many "testing scripts", all written in bash, and to do so in parallel -- but a controlled parallelism, with some configurable number (let's say 5) running at a time. One way to do this is to build a semaphore: a lock which can be locked N times, but when you try to lock it N+1 times, you have to wait until someone holding one of the N locks releases their lock.

The main loop #

abortParallelSpecs() {
    kill $CHILD_PIDS >/dev/null 2>&1
    wait $CHILD_PIDS >/dev/null 2>&1
    sleep 1
    exit 1
}

MAX_PARALLEL="$(getconf _NPROCESSORS_ONLN)"
let "MAX_PARALLEL=MAX_PARALLEL-2" "MAX_PARALLEL=(MAX_PARALLEL<2?2:MAX_PARALLEL)"

echo "Running unit tests across $MAX_PARALLEL cores..."

trap "abortParallelSpecs" EXIT
openSemaphore $MAX_PARALLEL

while read TEST_FILE; do
  runWithLock bash <<EOF
    "$TEST_FILE" >> $LOG_DIR/$TEST_FILE.log
EOF
  CHILD_PIDS="$CHILD_PIDS $!"
done < "$SPECS_FILE"

wait $CHILD_PIDS

This code skeleton sets the scene. We start by determining some number of parallel threads to run, ideally based on the available cores. Then, we set up a trap, so that exiting (Ctrl+C) the script will kill all of the running test processes. We create a semaphore with N slots, using the not-yet-defined openSemaphore function; then we loop through a big file listing our test scripts, calling the not-yet-defined runWithLock function on each one. As we trigger each script, we append it to a list of all child processes we've started, so at the very end we can wait for any stragglers to finish.

I've cut out some other details about printing the logged output, handling script arguments, etc., but this is the script in a nutshell. What's left is for us to define are these mysterious functions -- what exactly do openSemaphore and runWithLock do?.

Opening the semaphore #

openSemaphore() {
  mkfifo pipe-$$
  exec 3<>pipe-$$
  rm pipe-$$

  local i=$1
  for ((;i>0;i--)); do
    printf %s 000 >&3
  done
}

Now it gets interesting.

To recap: we call openSemaphore N, which uses file descriptor 3 to create a blocking queue, initially filled with N*3 zeroes (0).

Using the semaphore #

runWithLock() {
  local x

  read -u 3 -n 3 x && ((0==x)) || exit $x
  (
    # Execute the passed command
    ( "$@"; )

    printf "%.3d" $? >&3
  ) &
}

More reading #

This design was inspired by the answers to StackOverflow question Parallelize a Bash FOR Loop -- if curious, please check it out to see many different takes on this problem!