Categories
Code Linux

Locking critical sections in shell scripts

A critical section is some piece of code which, due to its nature and effects, should be executed by at most one thread or process at a time. If such code is executed concurrently, the results often become undefined and arbitrary. Atomically locking a shared resource is a common pattern to synchronize execution of critical sections and ensure mutually exclusive access.

Shell scripts mostly deal in processes and files, and there are several common scenarios where code is actually a critical section. If such code is run in several processes concurrently, it could introduce race conditions and arbitrary results. Consider a script that starts a background process if it is not already running – a typical pattern to start a singleton daemon process. Such code is a critical section, because you can end up with two running daemons if the code runs concurrently (and the daemon does not check for other instances of itself). Another good example is multiple scripts writing to a shared file, or even a shared directory structure.

There are a few strategies to implement locking in shell scripts, some are better than others. In this post, I will focus on one of the most robust ways: using flock(1). This nice tool gives you access to kernel level file locking from you shell. It has some clear advantages over traditional existence based file locking:

  1. It is truly atomic.
  2. The kernel manages the locks and releases them automatically when lock owning processes die. So no more stale lock files to clean up.
  3. You can block and wait for lock, indefinitely or with a timeout, and instantly get it when another process frees it. (Avoid lock polling loops with sleeping.)

It is important to understand that file locks are tied to both a file on the file system and running processes with open file descriptors to it. Even if a file used for locking exists on the file system, it does not mean the lock is taken ! The file system acts as a namespace of shared resources on which we can attach locks. Also note that the locks are advisory only – if a process does not care to check for locks, it will not participate in any synchronization and can do whatever it pleases.

Shell script with locking functions

We will look at a script which needs to protect a critical section with locking. The locking shall be done on a common file job.lock, which means any process with access to that file can obtain or check for a lock. To code along, you can copy the script to your own file and run the examples.

job.sh

#!/bin/bash

lock_acquire() {
    # Open a file descriptor to lock file
    exec {LOCKFD}>job.lock || return 1

    # Block until an exclusive lock can be obtained on the file descriptor
    flock -x $LOCKFD
}

lock_release() {
    test "$LOCKFD" || return 1
    
    # Close lock file descriptor, thereby releasing exclusive lock
    exec {LOCKFD}>&- && unset LOCKFD
}

lock_acquire || { echo >&2 "Error: failed to acquire lock"; exit 1; }

# --- Begin critical section ---

if [ -f job.dat ]; then
    value=$(<job.dat)
else
    value=0
fi
value=$((value + 1))

echo $value >job.dat

# --- End critical section ---

lock_release

The lock_acquire function uses flock -x N to obtain an exlusive lock on a file descriptor N. Since the file descriptor is opened by the script process itself, it will be the owner of the lock after flock exits. Flock is able to lock the descriptor because it is inherited from the shell process that started it. The critical section reads a number from a file if it exists, increments it by one, and writes the updated number back to the file.

Testing

First we’ll run the job script once:

$ bash job.sh 
$ ls
job.dat  job.lock  job.sh
$ cat job.dat 
1

A job.dat file is produced with a value of 1, which is entirely expected and not very interesting.

Next we’ll start 100 job processes asynchronously as fast as possible in the background, which means that many of them will run concurrently. We do this two times:

$ rm job.dat 
$ (for i in {1..100}; do bash job.sh & done; wait)
$ cat job.dat 
100
$ (for i in {1..100}; do bash job.sh & done; wait)
$ cat job.dat 
200

The for loop is started in a sub shell to avoid job control messages. The data has been incremented exactly 100 times after the first run, and incremented again by 100 after the second. If you look at the code in the critical section, it both reads, updates and then writes to the shared file, and doing this without locking would not work consistently.

Actually, let us try that, by commenting out the lock_acquire call in the script:

[...]

#lock_acquire || { echo >&2 "Error: failed to acquire lock"; exit 1; }

# --- Begin critical section ---

Then we run the test again:

$ rm job.dat
$ (for i in {1..100}; do bash job.sh & done; wait)
$ cat job.dat
3
$ (for i in {1..100}; do bash job.sh & done; wait)
$ cat job.dat
14

Which ends with a final result of 14, clearly incorrect and arbitrary. The results will vary with each run and depend on things like the speed of your computer.

Releasing the lock ?

In this case the script actually does not need to release the lock right before it exits, because the kernel will automatically do that when the process exits anyway. We will try it by re-enabling the locking call, and commenting out the lock_release call:

[...]

lock_acquire || { echo >&2 "Error: failed to acquire lock"; exit 1; }

# --- Begin critical section ---
[...]
# --- End critical section ---

#lock_release

And run the test:

$ rm job.dat
$ (for i in {1..100}; do bash job.sh & done; wait)
$ (for i in {1..100}; do bash job.sh & done; wait)
$ cat job.dat
200

It still works fine.

Starting a daemon process from your shell init scripts

A common use case is the need to start a single daemon process from your shell init scripts, unless one is already running. So you only ever want one such thing running. Consider the following:

if ! ps -ef|grep some-daemon|grep -qv grep; then
    some-daemon & pid=$!
    echo Started some-daemon with pid $pid
fi

This code is racy unless it is protected by locking. If you were to start two terminals more or less simultaneously, both executing your shell init scripts, you could possibly end up with two running daemon processes.

To protect this with flock, you could do the following:

if exec {bashrc_fd}<~/.bashrc && flock -nx $bashrc_fd; then
    # --- Begin critical section ---
    if ! ps -ef|grep some-daemon|grep -qv grep; then
        some-daemon & pid=$!
        echo Started some-daemon with pid $pid
    fi
    # --- End critical section ---

    flock -u $bashrc_fd && exec {bashrc_fd}>&-
fi

Here we open a read-only file descriptor to ~/.bashrc and then try to grab an exclusive lock on it, but we do it non-blocking with option -n. If some other bash process is already executing that part of the init file, flock will not succeed and immediately exit with a non-zero code, so the block is skipped. It has the effect that only one bash process will execute the code, and others running at the same time will skip it.

You may notice that we explicitly release the lock using flock -u $bashrc_fd after the critical section. Normally it is enough to close the file descriptor used for locking, but when starting child processes, those may inherit and keep such descriptors open. So the parent process closing its copy of the descriptor may not be enough actually release lock. Therefore we do it explicitly.

Closing notes

The manual page for the flock command lists a few good examples of how you can use it your scripts. However, none of those examples show how you can make the current shell process own and control the locks without using sub shells or flock invocations to wrap critical sections/commands.

The manual page for the flock(2) system call is a good read if you are interested in more details about how it works.

Read more about handling file descriptors with Bash in this part of the bash(1) manual.

Categories
Linux

Extract media file tags with ffprobe

The ffprobe tool, which is part of ffmpeg, can be used to extract all sorts of technical information from media streams. It is very capable, but is perhaps not the most intuitive tool with regards to its options. Therefore, examples are handy to get started. This post will collect examples focusing on metadata tags and the JSON output format.

Extract format section to JSON

The format section is a global part of the media container and provides generic information about the container streams. It also contains the metadata tags, like title, artist, year, etc. In this example, the file is an MP4 audio file.

$ ffprobe -v quiet -of json -show_entries format audio.m4a
{
    "format": {
        "filename": "audio.m4a",
        "nb_streams": 1,
        "nb_programs": 0,
        "format_name": "mov,mp4,m4a,3gp,3g2,mj2",
        "format_long_name": "QuickTime / MOV",
        "start_time": "0.000000",
        "duration": "3.024000",
        "size": "27730",
        "bit_rate": "73359",
        "probe_score": 100,
        "tags": {
            "major_brand": "M4A ",
            "minor_version": "512",
            "compatible_brands": "isomiso2",
            "genre": "Tone",
            "artist": "A",
            "title": "A title",
            "album": "X",
            "date": "2018",
            "encoder": "Lavf58.13.100"
        }
    }
}
-v quietBe quiet with diagnostics to stderr.
-of jsonSelect JSON output format.
-show_entries formatSelect the format section.
Options breakdown

Extract only the metadata tags

$ ffprobe -v quiet -of json -show_entries format_tags Movie.mkv 
{
    "format": {
        "tags": {
            "title": "The Movie",
            "DATE_RELEASED": "2008",
            "creation_time": "2022-02-11T18:26:58.000000Z",
            "ENCODER": "Lavf58.29.100"
        }
    }
}

Here we select the format_tags section from a video file in Matroska format, omitting the other bits of information under format. Finally, we can use jq to select parts of the JSON data:

$ ffprobe -v quiet -of json -show_entries format_tags Movie.mkv|\
          jq -r .format.tags.title
The Movie

Other output formats

ffprobe supports several output formats. Personally I like JSON, since it is very well defined with regard to syntax and escaping of special characters, etc. However, depending on use case, other forms may be more suitable for extraction of data in contexts like shell scripts.

Categories
Hardware Linux

Capture images from a webcam using ffmpeg

The examples are for Linux and access the web camera through the Video4Linux2 interface. To control web camera settings, use the tool v4l2-ctl. To list connected camera devices, you can use the command: v4l2-ctl --list-devices. On a typical Debian-ish Linux distro, you will also want to add your user to the video and audio groups, so that you can easily access the webcam from a non-desktop session.

Capture to an image file, continually overwriting it with new contents

ffmpeg -y -f v4l2 -video_size 1280x720 -i /dev/video0 \
       -r 0.2 -qscale:v 2 -update 1 /tmp/webcam.jpg
-f v4l2specify input format explicitly as capture from a Video4Linux2 device
-video_size 1280x720specify video frame size from webcam
-i /dev/video0select input device (a UVC-compatible webcam in my case)
-r 0.2set output frame rate to one per 5 seconds
-qscale:v 2set video quality [JPEG quality in this case], 2 is highest quality.
-update 1Image2 muxer option, enable in place update of image file for each video output frame
Options breakdown

Point the output file to a place served by your web server to make your camera image available on the web. The ffmpeg command will run until interrupted or killed.

Add a timestamp to captured images

ffmpeg -y -f v4l2 -video_size 1280x720 -i /dev/video0 \
       -r 0.2 \
       -vf "drawtext=text=%{localtime}:fontcolor=white@1.0:fontsize=26:borderw=1:x=980:y=25" \
       -qscale:v 2 -update 1 /tmp/webcam.jpg

Here we have inserted the drawtext video filter into the processing pipeline. We use its text expansion facilities to simply render the local time onto each video frame with filter-argument text=%{localtime}. It is placed in the top right corner of the image using the x and y arguments.

Running as background job

You can ssh to the host which has the web camera connected, and start the ffmpeg capture process as a background job:

ffmpeg -y -loglevel fatal \
       -f v4l2 -video_size 1280x720 -i /dev/video0 \
       -r 0.2 \
       -vf "drawtext=text=%{localtime}:fontcolor=white@1.0:fontsize=26:borderw=1:x=980:y=25" \
       -qscale:v 2 -update 1 /tmp/webcam.jpg \
       </dev/null &>/tmp/webcam-ffmpeg.log & disown $!

This silences ffmpeg to log only fatal errors, runs it in the background and finally detaches the process from your [bash] shell’s job control, to avoid it being killed if you log out. A more polished solution would be to create a systemd service which controls the ffmpeg webcam capture process, running as a dedicated low privilege system user.

Creating a time lapse video from a bunch of image files

As a sort of bonus chapter on this post, here is how to create a time lapse video from a bunch of captured image files. Assuming you have a directory with JPEG images named in such a way that they sort chronologically by their filenames (padded sequence numbers or timestamps), here’s how you can transform them into a video.

VP9 video in WebM container:

ffmpeg -y -f image2 -pattern_type glob -framerate 30 -i webcam-images/\*.jpg \
       -pix_fmt yuv420p -b 1500k timelapsevid.webm

H264 video in MP4 container:

ffmpeg -y -f image2 -pattern_type glob -framerate 30 -i webcam-images/\*.jpg \
       -pix_fmt yuv420p -b 1500k timelapsevid.mp4
-f image2Input demuxer is Image2, which can read image files.
-pattern_type globInstructs Image2 demuxer to treat input pattern as file name glob.
-framerate 30Set desired framerate; how many images to display per second in the resulting video.
-i webcam-images/\*.jpgSet input to a glob pattern matching the images files you would like to include in the video. Note that we do not want the shell to expand the glob, but rather pass the asterisk verbatim to ffmpeg.
-pix_fmt yuv420pSet video codec pixel format. YUV420p is selected to ensure compatibility with a broad range of decoders/players.
-b 1500kSet desired bitrate of video file.
Options breakdown

Note that all input images should have the same dimensions. Otherwise, you will likely have to add more options to ffmpeg to transform everything to a single suitable video size.

The resulting video files will be suitable for publishing on the web using the <video> tag.