Category: Linux

Linux related posts.

Categories
Linux

Backups now and in the future

I have never lost a single piece of data valuable to me in 20 plus years of computing. That is a very boring fact. But anyway. Deep in some archives, I even have copies of the very first web pages I made, back in 1996 (ugly and embarrassing, but funny stuff). These days, internet services (or “cloud” if you like) is all the rage for backups. There are the huge players, like Dropbox, Google Drive, OneDrive, iCloud and so forth. They all have client agents able to synchronize local data off to remote cloud storage. For mobile devices, everything’s automated to the point where the user doesn’t have to worry at all. (Except for their own privacy.) There are basically tons of services where you can send off your data and not worry about the details of how it is kept safe. I want more control over how and when my personal data is shipped off, so I only use such services for one thing related to backups: store gigabytes of locally encrypted backup archives that I upload manually about once a month.

If I was to make a list of 10 general requirements to a site backup system, well, this would probably be it:

  1. No human involvement: Backups must be performed automatically and regularly.
  2. Low maintenance: Backup must require little effort to setup on client machines (very little configuration).
  3. Availability: Backups must not interfere with regular usage of client devices.
  4. Storage efficiency: Backup snapshots must be stored incrementally (using some delta-scheme).
  5. Versioning: Older versions of files must be kept, and the most recent backup snapshot should be easily accessible.
  6. Privacy: Backups must always be encrypted at the source end before being sent off to a remote storage location or saved onto external media.
  7. Redundancy: Backup archives should be kept at multiple physical sites, to avoid complete loss of everything in case one site burns down to the ground.
  8. Redundancy: Encrypted backup archives of the latest snapshots should be transferred to an offsite storage location regularly, but not necessarily as often as automated local site backups are performed.
  9. Redundancy: The encrypted backup archives should also be kept at the local site.
  10. Monitoring: an automated backup system must warn promptly if problems arise, but otherwise stay silent and do its job. For details, a log of all operations must be available somewhere locally.

This list is of course not a random selection of 10 good points about data safety best practices. I have had a custom system in place for several years, which satisfies most of these criteria. It automates all steps except for number 8; I manually upload encrypted archives to offsite cloud storage (currently using Google drive). The archives themselves are automatically generated at the end of each month, so all I need to do is open a browser, access a local network share and initiate the upload procedure. This requires very little of my time.

My setup is based around a central backup server model, where the server pulls data from client hosts that are online on the local network. It’s a rather substantial shell script solution with support for configuration as code, pattern based exclusion rules, pluggable hooks and it uses rdiff-backup as the engine internally. Backup snapshots are saved to backup server local storage. The most recent (current) host snapshots are directly available at all times on the backup server (an advantage of rdiff-backup). The backup job runs at regular intervals and retries hosts missed in previous attempts within the course of a day. The snapshots are complete, in that all operating system files are backed up, in addition to personal data. The ssh protocol is used for transport across network.

In general this setup has worked very well over the years. Client setup is very lightweight, only requiring the installation of OpenSSH server, rdiff-backup and ssh key setup for root user. That procedure is automated using Ansible. An obvious weakness is that there is no support for Windows hosts or mobile devices. I don’t regularly use Windows-clients except for work-related things, so it is not a big deal. But a simple solution I’ve used in the past is to simply [shadow] copy from Windows host to a network share on a Linux host that is backed up.

So, in this modern age, is my trusty but crusty backup regime still a good solution ? All in all, yes, since data still boils down to files with valuable bytes in them, and my solution gives me absolute control and privacy. It has some disadvantages, though. Laptop clients will not be backed up when they are not present on my local network, since it is a pull model. So I am considering some options like dejadup, but it needs to support complete system backups. I have not investigated how well that works with dejadup. (rdiff-backup has proven itself to be excellent in this respect and happily creates full file based host system snapshots without issue.)

Categories
Linux

Inspector Pom on Github

I’ve brushed up a handy tool I wrote a few years ago. It allows quick inspection of Maven pom.xml files on the command line, producing human readable plain text output. If you’ve ever found yourself grepping pom.xml files for version numbers or other bits of information, you might find it useful. Check it out on Github: https://github.com/oyvindstegard/pom

Inspector Pom command line
Example of output
Categories
Code Linux

Run shell script as different user with proper argument handling

So you have a shell script which needs to drop or modify its privileges by switching to an appropriate system user before continuing its execution.

Here are some alternatives to accomplish this while preserving all original command line arguments properly. Privileges are dropped by switching to the nobody user, adapt the RUN_AS variable as desired.

Alternative 1 – using only su, requires being root (uid 0) or the target user to run
#!/bin/sh

# This script must run as
RUN_AS=nobody

if [ `id -nu` != $RUN_AS ]; then
    if [ `id -u` -ne 0 ]; then
        echo >&2 "Sorry, you must be either root or $RUN_AS to run me."
        exit 1
    fi

    # This environment variable is just a safe guard for endless re-exec loop
    # and something the script can use to test up to this point if it has
    # dropped privileges by re-executing itself
    if [ "$EXEC_SU" ]; then
        echo >&2 "Re-exec loop circuit breaker engaged, something is wrong"
        exit 1
    fi

    exec su $RUN_AS -s /bin/sh -c "EXEC_SU=1 \"$0\" \"\$@\"" -- "$0" "$@"
fi

# At this point, we can be sure we are running as the desired user.
echo Running as `id -nu`
for arg in "$@"; do
    echo Argument $((n=n+1)): $arg
done

This alternative requires being root or the target user when invoking the script. Command line arguments are preserved after switching.

Alternative 2 – using sudo
#!/bin/sh

# This script must run as
RUN_AS=nobody

if [ `id -nu` != $RUN_AS ]; then
    if [ -z "$SUDO_EXEC" ]; then
        exec sudo -u $RUN_AS SUDO_EXEC=1 "$0" "$@"
    else
        echo >&2 'Re-exec loop circuit breaker engaged, something is wrong'
        exit 1
    fi
fi

# At this point, we can be sure we are running as the desired user.
echo Running as `id -nu`
for arg in "$@"; do
    echo Argument $((n=n+1)): $arg
done

Unlike alternative 1, this method does not require the script to be invoked by root or the target user directly, if switching to a system user that typically has no password set on its own. (Su will also prompt for password automatically, but that requires the target user password.)

Alternative 3 – when a clean login environment is required

If you require the script to run in a clean login environment for the target user, then things become slightly more complicated. It can be accomplished by combining sudo with su:

#!/bin/sh

# This script must run as
RUN_AS=nobody

if [ `id -nu` != $RUN_AS ]; then
    if [ -z "$SUDO_SU_EXEC" ]; then
        exec sudo su -s /bin/sh - $RUN_AS -c "SUDO_SU_EXEC=1 \"$0\" \"\$@\"" -- "$0" "$@"
    else
        echo >&2 'Re-exec loop circuit breaker engaged, something is wrong'
        exit 1
    fi
fi

# At this point, we can be sure we are running as the desired user.
echo Running as `id -nu`
echo USER=$USER HOME=$HOME 
for arg in "$@"; do
    echo Argument $((n=n+1)): $arg
done

Note that since the example above switches to nobody, which is a system user that by default does not have a shell configured, we explicitly set the shell using su argument "-s /bin/sh".