# Advanced Bash Scripting Techniques

URL: https://www.devopsness.com/blog/advanced-bash-scripting-techniques
Author: Kiril Urbonas
Published: 2023-12-25
Updated: 2023-12-25
Categories: Linux
Tags: Bash, Linux, Sysadmin, Automation

Summary: Bash patterns beyond the basics: arrays, traps, process substitution, parameter expansion. The features that earn their place when scripts grow.

---
# Bash Patterns Beyond the Basics

Most bash content stops at "use shellcheck and quote your variables." That's important but a narrow slice of what bash can do. After enough years of writing operational scripts, I keep reaching for a specific set of intermediate features. This post is those features, with examples of when each pays off and when it's the wrong tool.

## When bash earns its place

Before any pattern: bash is the right tool when the script is mostly orchestrating other commands. Pure logic, complex data structures, anything that would benefit from types — Python is a better choice. The patterns below are for when bash is genuinely the right call.

The line for me is around 150 lines. Above that, the script's complexity usually justifies a real language.

## Arrays

Bash arrays let you handle lists of items cleanly:

```bash
files=(
  "/etc/passwd"
  "/etc/hosts"
  "/etc/resolv.conf"
)

for f in "${files[@]}"; do
  echo "Checking $f"
  [ -f "$f" ] || echo "  Missing!"
done
```

The `"${files[@]}"` is the right way to expand an array — quoted, with `@`. Using `*` joins with the first IFS character; using unquoted forms breaks on filenames with spaces.

Reading lines from a file into an array:

```bash
mapfile -t lines < input.txt
echo "Read ${#lines[@]} lines"
```

`mapfile -t` (also called `readarray`) reads each line as an array element, stripping the trailing newline. Much better than the old "while read" pattern for whole-file ingestion.

Associative arrays (bash 4+):

```bash
declare -A counts
counts["error"]=0
counts["warning"]=0

while read line; do
  if [[ "$line" =~ ERROR ]]; then
    counts["error"]=$((counts["error"]+1))
  elif [[ "$line" =~ WARN ]]; then
    counts["warning"]=$((counts["warning"]+1))
  fi
done < log.txt

for key in "${!counts[@]}"; do
  echo "$key: ${counts[$key]}"
done
```

Hash maps in bash. Useful for counting, grouping, lookups. Mostly for shorter scripts; for anything substantial, Python is cleaner.

## Trap for cleanup

`trap` runs a command when a signal arrives or when the script exits:

```bash
TMPDIR=$(mktemp -d)
trap 'rm -rf "$TMPDIR"' EXIT
trap 'echo "Interrupted; cleaning up..."; rm -rf "$TMPDIR"; exit 130' INT TERM

# Use $TMPDIR confidently — it'll get cleaned up
```

The `EXIT` trap runs whether the script succeeds or fails — perfect for cleanup. `INT` (Ctrl-C) and `TERM` (kill) traps catch user-initiated interruption.

Common cleanup uses:
- Remove temp files / directories
- Reset terminal state (e.g., re-enable echo if you turned it off)
- Send a "I died" notification
- Release locks

I add a trap to nearly every non-trivial script that creates state. Without it, interrupting the script leaves cruft.

## Process substitution

`<(...)` and `>(...)` are bash-specific (not POSIX) features for treating command output as a file:

```bash
diff <(sort file1) <(sort file2)
```

This sorts both files and diffs them, without writing temporary files. Each `<(...)` is a virtual file (technically `/dev/fd/N`).

Common uses:
- Comparing command outputs (`diff`, `comm`)
- Reading from a command into a `while read` loop without subshell scoping issues
- Feeding a command's output to a tool that wants a filename

```bash
# Avoid the subshell trap of "while read | piped output"
while read -r line; do
  count=$((count + 1))
done < <(some_command_that_emits_lines)
echo "$count"
```

The `< <(...)` (note the space) feeds the output of a command as the script's stdin to the while loop. Unlike `... | while read`, this doesn't run the loop in a subshell, so variables modified inside persist.

## Parameter expansion: more than just $var

Bash has rich parameter expansion — string manipulation without forking subprocesses:

```bash
# Default if unset
name="${USER:-unknown}"

# Default and assign if unset
: "${LOG_LEVEL:=INFO}"

# Error if unset
: "${REQUIRED_VAR:?must be set}"

# Substring
str="hello world"
echo "${str:0:5}"        # hello
echo "${str:6}"          # world

# Length
echo "${#str}"           # 11

# Pattern removal (greedy with ##, %% — non-greedy with #, %)
filename="/path/to/file.txt"
echo "${filename##*/}"   # file.txt (basename)
echo "${filename%/*}"    # /path/to (dirname)
echo "${filename%.*}"    # /path/to/file (without extension)
echo "${filename##*.}"   # txt (extension)

# Replacement
str="foo bar foo"
echo "${str/foo/baz}"    # baz bar foo (first occurrence)
echo "${str//foo/baz}"   # baz bar baz (all)

# Case modification
echo "${str^^}"          # FOO BAR FOO (upper)
echo "${str,,}"          # foo bar foo (lower, lowercase variant)
```

These are faster than spawning sed/cut/awk for simple string ops. For anything complex, those tools are still right.

## Brace expansion

Bash expands brace patterns before execution:

```bash
echo {1..5}              # 1 2 3 4 5
echo {a..f}              # a b c d e f
echo file_{1..3}.txt     # file_1.txt file_2.txt file_3.txt
echo dir/{prod,staging,dev}/config.yml
```

Useful for terse iteration:

```bash
for env in {dev,staging,prod}; do
  ./deploy.sh "$env"
done

# Backup files atomically
cp /etc/nginx/nginx.conf{,.bak}
# Equivalent to: cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.bak
```

## Conditional patterns: `[[` over `[`

`[[ ... ]]` is bash's enhanced test (`[ ... ]` is the POSIX original). `[[` doesn't word-split, supports regex, and has clearer syntax:

```bash
# Better
if [[ "$var" =~ ^[a-z]+$ ]]; then ...
if [[ -n "$var" && "$var" != "default" ]]; then ...

# Worse / fragile
if [ "$var" != "" ] && [ "$var" != "default" ]; then ...
```

For new scripts, prefer `[[` unless you specifically need POSIX compatibility (in which case, POSIX shell isn't bash anyway).

## Heredocs and herestrings

Heredoc: multi-line string literal:

```bash
cat <<EOF > /etc/myconfig
listen = 8080
host = ${HOSTNAME}
log_level = INFO
EOF
```

Variables expand inside (use `<<'EOF'` with single quotes to disable expansion).

Herestring: short single-line input:

```bash
read -r var <<< "input string"
```

Both pass content to a command's stdin without a temporary file or echo.

## Functions with proper return values

Functions in bash can't return arbitrary values; they return exit status (0-255). For other return values, use stdout:

```bash
get_user_id() {
  local username="$1"
  id -u "$username" 2>/dev/null
}

# Usage
if user_id=$(get_user_id "myuser"); then
  echo "User ID: $user_id"
else
  echo "User not found"
fi
```

The function writes to stdout; the caller captures with `$(...)`. The function's exit status (whether `id` succeeded) is checkable via `if`.

For functions that need multiple outputs, multi-value packing via `printf`:

```bash
get_user_info() {
  local username="$1"
  printf "%s|%s|%s" "$(id -u "$username")" "$(id -g "$username")" "$(id -gn "$username")"
}

IFS='|' read -r uid gid gname <<< "$(get_user_info "myuser")"
```

Awkward but functional. If you're doing this a lot, that's a sign to move to a real language.

## Command-line option parsing

For more than 2-3 options, use `getopts` or a hand-rolled parser:

```bash
verbose=0
output=""

while getopts ":vo:" opt; do
  case $opt in
    v) verbose=1 ;;
    o) output="$OPTARG" ;;
    \?) echo "Invalid option: -$OPTARG" >&2; exit 1 ;;
    :) echo "Option -$OPTARG requires argument" >&2; exit 1 ;;
  esac
done
shift $((OPTIND-1))
```

For long options (`--verbose`), getopts doesn't help; you need hand-rolled or `getopt` (the external command, different from bash's `getopts`).

Honestly, for non-trivial CLI parsing, Python with `argparse` is much nicer.

## Locking to prevent concurrent runs

For scripts that shouldn't run concurrently:

```bash
LOCKFILE="/var/run/myscript.lock"
exec 200>"$LOCKFILE"
flock -n 200 || { echo "Already running"; exit 1; }

# Now we have the lock; do work
```

`flock` on a file descriptor; non-blocking (`-n`) so we exit if locked. The lock is released when the script exits (file descriptor closes).

Useful for cron jobs that should not overlap themselves.

## Strict mode patterns

The strict mode preamble:

```bash
#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'
```

Plus extras for debugging:

```bash
# Trace execution (verbose)
set -x

# Run with `bash -x script.sh` instead if you don't want to modify the script

# Or, more selective:
PS4='+ ${BASH_SOURCE}:${LINENO}: ${FUNCNAME[0]:-main}() '
```

The `PS4` makes the trace show source location, which is invaluable for debugging.

## What I avoid

A few patterns that look clever but cause problems:

**Multiline `eval`**. Already covered: `eval` is dangerous. If you need dynamic execution, use a real language.

**Function return values via global variables.** Named "out parameters." Works but obscures data flow. Functions returning via stdout are cleaner.

**Heavy use of `bash -c "$dynamic_string"`**. Same risks as eval.

**Custom argument-parsing libraries in bash.** They exist but they're awkward. Use Python at this point.

**`printf` for complex formatting**. Useful for simple cases; for anything elaborate, a templating language or real string-formatting code is better.

## Testing bash scripts

For scripts beyond a certain complexity, tests:

```bash
# tests/test_helper.bash
test_get_user_id() {
  source ../script.sh
  result=$(get_user_id "root")
  [ "$result" = "0" ] || { echo "FAIL"; return 1; }
}

test_get_user_id
```

For more elaborate tests, `bats` (Bash Automated Testing System) is the standard tool. Each `@test` block is a test case; pass/fail per block.

`shellcheck` for static analysis (mandatory in our CI).

## What I'd tell a team

**Quote everything. `set -euo pipefail`. shellcheck in CI.** The boring fundamentals matter most.

**Use arrays for lists.** Don't pretend space-separated strings are arrays.

**`trap` for cleanup.** Every script that creates state should clean up on exit.

**Process substitution to avoid temp files.** `<(cmd)` is your friend.

**Parameter expansion for simple string ops.** Faster and more readable than calling sed/cut/awk.

**Move to Python at ~150 lines.** Bash for short orchestration; Python for anything substantive.

The intermediate bash patterns above are the ones that earn their place in operational scripts. Knowing them keeps your bash readable and your scripts maintainable. They're not a substitute for moving to a real language when the task outgrows shell — but for the work that genuinely belongs in bash, they're the difference between scripts that survive and scripts that need rewriting in 6 months.

