Category: Linux

Linux related posts.

Categories
Hardware Linux

Capture images from a webcam using ffmpeg

The examples are for Linux and access the web camera through the Video4Linux2 interface. To control web camera settings, use the tool v4l2-ctl. To list connected camera devices, you can use the command: v4l2-ctl --list-devices. On a typical Debian-ish Linux distro, you will also want to add your user to the video and audio groups, so that you can easily access the webcam from a non-desktop session.

Capture to an image file, continually overwriting it with new contents

ffmpeg -y -f v4l2 -video_size 1280x720 -i /dev/video0 \
       -r 0.2 -qscale:v 2 -update 1 /tmp/webcam.jpg
-f v4l2specify input format explicitly as capture from a Video4Linux2 device
-video_size 1280x720specify video frame size from webcam
-i /dev/video0select input device (a UVC-compatible webcam in my case)
-r 0.2set output frame rate to one per 5 seconds
-qscale:v 2set video quality [JPEG quality in this case], 2 is highest quality.
-update 1Image2 muxer option, enable in place update of image file for each video output frame
Options breakdown

Point the output file to a place served by your web server to make your camera image available on the web. The ffmpeg command will run until interrupted or killed.

Add a timestamp to captured images

ffmpeg -y -f v4l2 -video_size 1280x720 -i /dev/video0 \
       -r 0.2 \
       -vf "drawtext=text=%{localtime}:fontcolor=white@1.0:fontsize=26:borderw=1:x=980:y=25" \
       -qscale:v 2 -update 1 /tmp/webcam.jpg

Here we have inserted the drawtext video filter into the processing pipeline. We use its text expansion facilities to simply render the local time onto each video frame with filter-argument text=%{localtime}. It is placed in the top right corner of the image using the x and y arguments.

Running as background job

You can ssh to the host which has the web camera connected, and start the ffmpeg capture process as a background job:

ffmpeg -y -loglevel fatal \
       -f v4l2 -video_size 1280x720 -i /dev/video0 \
       -r 0.2 \
       -vf "drawtext=text=%{localtime}:fontcolor=white@1.0:fontsize=26:borderw=1:x=980:y=25" \
       -qscale:v 2 -update 1 /tmp/webcam.jpg \
       </dev/null &>/tmp/webcam-ffmpeg.log & disown $!

This silences ffmpeg to log only fatal errors, runs it in the background and finally detaches the process from your [bash] shell’s job control, to avoid it being killed if you log out. A more polished solution would be to create a systemd service which controls the ffmpeg webcam capture process, running as a dedicated low privilege system user.

Creating a time lapse video from a bunch of image files

As a sort of bonus chapter on this post, here is how to create a time lapse video from a bunch of captured image files. Assuming you have a directory with JPEG images named in such a way that they sort chronologically by their filenames (padded sequence numbers or timestamps), here’s how you can transform them into a video.

VP9 video in WebM container:

ffmpeg -y -f image2 -pattern_type glob -framerate 30 -i webcam-images/\*.jpg \
       -pix_fmt yuv420p -b 1500k timelapsevid.webm

H264 video in MP4 container:

ffmpeg -y -f image2 -pattern_type glob -framerate 30 -i webcam-images/\*.jpg \
       -pix_fmt yuv420p -b 1500k timelapsevid.mp4
-f image2Input demuxer is Image2, which can read image files.
-pattern_type globInstructs Image2 demuxer to treat input pattern as file name glob.
-framerate 30Set desired framerate; how many images to display per second in the resulting video.
-i webcam-images/\*.jpgSet input to a glob pattern matching the images files you would like to include in the video. Note that we do not want the shell to expand the glob, but rather pass the asterisk verbatim to ffmpeg.
-pix_fmt yuv420pSet video codec pixel format. YUV420p is selected to ensure compatibility with a broad range of decoders/players.
-b 1500kSet desired bitrate of video file.
Options breakdown

Note that all input images should have the same dimensions. Otherwise, you will likely have to add more options to ffmpeg to transform everything to a single suitable video size.

The resulting video files will be suitable for publishing on the web using the <video> tag.

Categories
Code Linux

command line jq snippets

Jq(1) is a surprisingly powerful command line JSON stream processing tool. I have not used it much, so this post will be a growing collection of random and useful snippets to help me remember.

Create a JSON array of a list of string values, with or without pretty printing
$ echo '"foo" "bar" "b a z"'|jq -s .
[
  "foo",
  "bar",
  "b a z"
]
$ echo '"foo" "bar" "b a z"'|jq -cs .
["foo","bar","b a z"]

The option -s/--slurp is what makes jq pack the values in an array here. The option -c/--compact-output toggles pretty printing.

Extract two particular values from the first of multiple deeply nested JSON-objects
$ cat data.json
{
  "data": [
    {
      "id": 1,
      "results": [
        {
          "type": "result",
          "objs": [
            {
              "a": "foo",
              "b": "bar"
            }
          ]
        }
      ]
    },
    {
      "id": 2,
      "results": [
        {
          "type": "result",
          "objs": [
            {
              "a": "foo2",
              "b": "bar2"
            }
          ]
        }
      ]
    }
  ]
}
$ cat data.json|jq -r '.data[0].results[0].objs[0].a, .data[0].results[0].objs[0].b'
foo
bar

The option -r/--raw-output causes jq not to quote the extracted string values. Makes use of the values in shell scripts easier.

Filter stream of JSON objects based on some condition
$ cat data.ndjson
{ "id": 1, "color": "red" }
{ "id": 2, "color": "green" }
{ "id": 3, "color": "blue" }

$ cat data.ndjson | jq -s 'map(select(.color == "blue"))'
[
  {
    "id": 3,
    "color": "blue"
  }
]
Create JSON from command line arguments
$ jq -n --arg a 1 --arg b 2 '{"a":$a, "b":$b}'
{
  "a": "1",
  "b": "2"
}

You can declare variables which jq makes available when constructing JSON, as can be seen in the example. The option -n/--null-input tells jq to not read anything on stdin, just output stuff. Notice that --arg by default treats values as JSON strings. If you need to encode values of a and b as real numbers in the above example, use --argjson instead:

$ jq -n --argjson a 1 --argjson b 2 '{"a":$a, "b":$b}'
{
  "a": 1,
  "b": 2
}

Categories
Linux

Backups now and in the future

I have never lost a single piece of data valuable to me in 20 plus years of computing. That is a very boring fact. But anyway. Deep in some archives, I even have copies of the very first web pages I made, back in 1996 (ugly and embarrassing, but funny stuff). These days, internet services (or “cloud” if you like) is all the rage for backups. There are the huge players, like Dropbox, Google Drive, OneDrive, iCloud and so forth. They all have client agents able to synchronize local data off to remote cloud storage. For mobile devices, everything’s automated to the point where the user doesn’t have to worry at all. (Except for their own privacy.) There are basically tons of services where you can send off your data and not worry about the details of how it is kept safe. I want more control over how and when my personal data is shipped off, so I only use such services for one thing related to backups: store gigabytes of locally encrypted backup archives that I upload manually about once a month.

If I was to make a list of 10 general requirements to a site backup system, well, this would probably be it:

  1. No human involvement: Backups must be performed automatically and regularly.
  2. Low maintenance: Backup must require little effort to setup on client machines (very little configuration).
  3. Availability: Backups must not interfere with regular usage of client devices.
  4. Storage efficiency: Backup snapshots must be stored incrementally (using some delta-scheme).
  5. Versioning: Older versions of files must be kept, and the most recent backup snapshot should be easily accessible.
  6. Privacy: Backups must always be encrypted at the source end before being sent off to a remote storage location or saved onto external media.
  7. Redundancy: Backup archives should be kept at multiple physical sites, to avoid complete loss of everything in case one site burns down to the ground.
  8. Redundancy: Encrypted backup archives of the latest snapshots should be transferred to an offsite storage location regularly, but not necessarily as often as automated local site backups are performed.
  9. Redundancy: The encrypted backup archives should also be kept at the local site.
  10. Monitoring: an automated backup system must warn promptly if problems arise, but otherwise stay silent and do its job. For details, a log of all operations must be available somewhere locally.

This list is of course not a random selection of 10 good points about data safety best practices. I have had a custom system in place for several years, which satisfies most of these criteria. It automates all steps except for number 8; I manually upload encrypted archives to offsite cloud storage (currently using Google drive). The archives themselves are automatically generated at the end of each month, so all I need to do is open a browser, access a local network share and initiate the upload procedure. This requires very little of my time.

My setup is based around a central backup server model, where the server pulls data from client hosts that are online on the local network. It’s a rather substantial shell script solution with support for configuration as code, pattern based exclusion rules, pluggable hooks and it uses rdiff-backup as the engine internally. Backup snapshots are saved to backup server local storage. The most recent (current) host snapshots are directly available at all times on the backup server (an advantage of rdiff-backup). The backup job runs at regular intervals and retries hosts missed in previous attempts within the course of a day. The snapshots are complete, in that all operating system files are backed up, in addition to personal data. The ssh protocol is used for transport across network.

In general this setup has worked very well over the years. Client setup is very lightweight, only requiring the installation of OpenSSH server, rdiff-backup and ssh key setup for root user. That procedure is automated using Ansible. An obvious weakness is that there is no support for Windows hosts or mobile devices. I don’t regularly use Windows-clients except for work-related things, so it is not a big deal. But a simple solution I’ve used in the past is to simply [shadow] copy from Windows host to a network share on a Linux host that is backed up.

So, in this modern age, is my trusty but crusty backup regime still a good solution ? All in all, yes, since data still boils down to files with valuable bytes in them, and my solution gives me absolute control and privacy. It has some disadvantages, though. Laptop clients will not be backed up when they are not present on my local network, since it is a pull model. So I am considering some options like dejadup, but it needs to support complete system backups. I have not investigated how well that works with dejadup. (rdiff-backup has proven itself to be excellent in this respect and happily creates full file based host system snapshots without issue.)