Categories
Code Linux

command line jq snippets

Jq(1) is a surprisingly powerful command line JSON stream processing tool. I have not used it much, so this post will be a growing collection of random and useful snippets to help me remember.


Create a JSON array of a list of string values, with or without pretty printing
$ echo '"foo" "bar" "b a z"'|jq -s .
[
  "foo",
  "bar",
  "b a z"
]
$ echo '"foo" "bar" "b a z"'|jq -cs .
["foo","bar","b a z"]

The option -s/--slurp is what makes jq pack the values in an array here. The option -c/--compact-output toggles pretty printing.

Extract two particular values from the first of multiple deeply nested JSON-objects
$ cat data.json
{
  "data": [
    {
      "id": 1,
      "results": [
        {
          "type": "result",
          "objs": [
            {
              "a": "foo",
              "b": "bar"
            }
          ]
        }
      ]
    },
    {
      "id": 2,
      "results": [
        {
          "type": "result",
          "objs": [
            {
              "a": "foo2",
              "b": "bar2"
            }
          ]
        }
      ]
    }
  ]
}
$ cat data.json|jq -r '.data[0].results[0].objs[0].a, .data[0].results[0].objs[0].b'
foo
bar

The option -r/--raw-output causes jq not to quote the extracted string values. Makes use of the values in shell scripts easier.

Filter stream of JSON objects based on some condition
$ cat data.ndjson
{ "id": 1, "color": "red" }
{ "id": 2, "color": "green" }
{ "id": 3, "color": "blue" }

$ cat data.ndjson | jq -s 'map(select(.color == "blue"))'
[
  {
    "id": 3,
    "color": "blue"
  }
]
Create JSON from command line arguments
$ jq -n --arg a 1 --arg b 2 '{"a":$a, "b":$b}'
{
  "a": "1",
  "b": "2"
}

You can declare variables which jq makes available when constructing JSON, as can be seen in the example. The option -n/--null-input tells jq to not read anything on stdin, just output stuff. Notice that --arg by default treats values as JSON strings. If you need to encode values of a and b as real numbers in the above example, use --argjson instead:

$ jq -n --argjson a 1 --argjson b 2 '{"a":$a, "b":$b}'
{
  "a": 1,
  "b": 2
}

Categories
Code

Spring Boot http access logging in three steps

Sometimes it can be handy to have a Spring boot web app log http requests. Here is a simple way to achieve this, with minimal dependencies. The examples apply only when using default Apache Tomcat embedded web server and Logback as logging implementation.

1. Add dependency logback-access to your project:

<dependency>
  <groupId>ch.qos.logback</groupId>
  <artifactId>logback-access</artifactId>
</dependency>

2. Add Logback access log configuration file to root of classpath:
src/main/resources/logback-access.xml

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <appender name="access_stdout" 
              class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>common</pattern>
        </encoder>
    </appender>

    <appender-ref ref="access_stdout"/>
</configuration>

3. Add a @Configuration class which integrates Logback into Tomcat (Kotlin in this example):

import ch.qos.logback.access.tomcat.LogbackValve
import org.springframework.boot.web.embedded.tomcat.TomcatContextCustomizer
import org.springframework.context.annotation.Bean
import org.springframework.context.annotation.Configuration
import java.nio.file.Files

@Configuration
open class AccessLogConfiguration {

    @Bean
    fun addLogbackAccessValve() = TomcatContextCustomizer { context ->

        javaClass.getResourceAsStream("/logback-access.xml").use {
            Files.createDirectories((context.catalinaBase.toPath()
                .resolve(LogbackValve.DEFAULT_CONFIG_FILE)).parent)

            Files.copy(it, context.catalinaBase.toPath()
                .resolve(LogbackValve.DEFAULT_CONFIG_FILE))
        }

        LogbackValve().let {
            it.isQuiet = true
            context.pipeline.addValve(it)
        }
    }

}

Here we copy the Logback access configuration file into Tomcat embedded web server runtime dir. This circumvents an issue with LogbackValve being initialized in a different class loader context by Tomcat, where it is not able to resolve its configuration file in the Spring Boot app main classpath. If it’s not working, set isQuiet to false to debug.

Now your Spring Boot app logs http requests through the console appender in a typical format:

127.0.0.1 - - [20/feb./2021:22:16:54 +0100] "GET / HTTP/1.1" 200 332

Getting Logstash-compatible JSON-output

In case you use Kibana and need structured log formatting, you can easily achieve this now. First ensure you have the dependency
net.logstash.logback:logstash-logback-encoder added to your build. Then simply use a slightly different
logback-access.xml configuration file:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <appender name="access_stdout" 
              class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="net.logstash.logback.encoder.LogstashAccessEncoder"/>
    </appender>

    <appender-ref ref="access_stdout"/>
</configuration>

Now your access log messages will be structured like this:

{
  "@timestamp": "2021-02-21T17:38:30.813+01:00",
  "@version": "1",
  "message": "127.0.0.1 - - [2021-02-21T17:38:30.813+01:00] \"GET / HTTP/1.1\" 200 332",
  "method": "GET",
  "protocol": "HTTP/1.1",
  "status_code": 200,
  "requested_url": "GET / HTTP/1.1",
  "requested_uri": "/",
  "remote_host": "127.0.0.1",
  "content_length": 332,
  "elapsed_time": 47
}
Categories
Code Linux

Run shell script as different user with proper argument handling

So you have a shell script which needs to drop or modify its privileges by switching to an appropriate system user before continuing its execution.

Here are some alternatives to accomplish this while preserving all original command line arguments properly. Privileges are dropped by switching to the nobody user, adapt the RUN_AS variable as desired.

Alternative 1 – using only su, requires being root (uid 0) or the target user to run
#!/bin/sh

# This script must run as
RUN_AS=nobody

if [ `id -nu` != $RUN_AS ]; then
    if [ `id -u` -ne 0 ]; then
        echo >&2 "Sorry, you must be either root or $RUN_AS to run me."
        exit 1
    fi

    # This environment variable is just a safe guard for endless re-exec loop
    # and something the script can use to test up to this point if it has
    # dropped privileges by re-executing itself
    if [ "$EXEC_SU" ]; then
        echo >&2 "Re-exec loop circuit breaker engaged, something is wrong"
        exit 1
    fi

    exec su $RUN_AS -s /bin/sh -c "EXEC_SU=1 \"$0\" \"\$@\"" -- "$0" "$@"
fi

# At this point, we can be sure we are running as the desired user.
echo Running as `id -nu`
for arg in "$@"; do
    echo Argument $((n=n+1)): $arg
done

This alternative requires being root or the target user when invoking the script. Command line arguments are preserved after switching.

Alternative 2 – using sudo
#!/bin/sh

# This script must run as
RUN_AS=nobody

if [ `id -nu` != $RUN_AS ]; then
    if [ -z "$SUDO_EXEC" ]; then
        exec sudo -u $RUN_AS SUDO_EXEC=1 "$0" "$@"
    else
        echo >&2 'Re-exec loop circuit breaker engaged, something is wrong'
        exit 1
    fi
fi

# At this point, we can be sure we are running as the desired user.
echo Running as `id -nu`
for arg in "$@"; do
    echo Argument $((n=n+1)): $arg
done

Unlike alternative 1, this method does not require the script to be invoked by root or the target user directly, if switching to a system user that typically has no password set on its own. (Su will also prompt for password automatically, but that requires the target user password.)

Alternative 3 – when a clean login environment is required

If you require the script to run in a clean login environment for the target user, then things become slightly more complicated. It can be accomplished by combining sudo with su:

#!/bin/sh

# This script must run as
RUN_AS=nobody

if [ `id -nu` != $RUN_AS ]; then
    if [ -z "$SUDO_SU_EXEC" ]; then
        exec sudo su -s /bin/sh - $RUN_AS -c "SUDO_SU_EXEC=1 \"$0\" \"\$@\"" -- "$0" "$@"
    else
        echo >&2 'Re-exec loop circuit breaker engaged, something is wrong'
        exit 1
    fi
fi

# At this point, we can be sure we are running as the desired user.
echo Running as `id -nu`
echo USER=$USER HOME=$HOME 
for arg in "$@"; do
    echo Argument $((n=n+1)): $arg
done

Note that since the example above switches to nobody, which is a system user that by default does not have a shell configured, we explicitly set the shell using su argument "-s /bin/sh".