Categories
Code Linux

command line jq snippets

Jq(1) is a surprisingly powerful command line JSON stream processing tool. I have not used it much, so this post will be a growing collection of random and useful snippets to help me remember.


Create a JSON array of a list of string values, with or without pretty printing
$ echo '"foo" "bar" "b a z"'|jq -s .
[
  "foo",
  "bar",
  "b a z"
]
$ echo '"foo" "bar" "b a z"'|jq -cs .
["foo","bar","b a z"]

The option -s/--slurp is what makes jq pack the values in an array here. The option -c/--compact-output toggles pretty printing.

Extract two particular values from the first of multiple deeply nested JSON-objects
$ cat data.json
{
  "data": [
    {
      "id": 1,
      "results": [
        {
          "type": "result",
          "objs": [
            {
              "a": "foo",
              "b": "bar"
            }
          ]
        }
      ]
    },
    {
      "id": 2,
      "results": [
        {
          "type": "result",
          "objs": [
            {
              "a": "foo2",
              "b": "bar2"
            }
          ]
        }
      ]
    }
  ]
}
$ cat data.json|jq -r '.data[0].results[0].objs[0].a, .data[0].results[0].objs[0].b'
foo
bar

The option -r/--raw-output causes jq not to quote the extracted string values. Makes use of the values in shell scripts easier.

Filter stream of JSON objects based on some condition
$ cat data.ndjson
{ "id": 1, "color": "red" }
{ "id": 2, "color": "green" }
{ "id": 3, "color": "blue" }

$ cat data.ndjson | jq -s 'map(select(.color == "blue"))'
[
  {
    "id": 3,
    "color": "blue"
  }
]
Create JSON from command line arguments
$ jq -n --arg a 1 --arg b 2 '{"a":$a, "b":$b}'
{
  "a": "1",
  "b": "2"
}

You can declare variables which jq makes available when constructing JSON, as can be seen in the example. The option -n/--null-input tells jq to not read anything on stdin, just output stuff. Notice that --arg by default treats values as JSON strings. If you need to encode values of a and b as real numbers in the above example, use --argjson instead:

$ jq -n --argjson a 1 --argjson b 2 '{"a":$a, "b":$b}'
{
  "a": 1,
  "b": 2
}

Categories
Code

Spring Boot http access logging in three steps

Sometimes it can be handy to have a Spring boot web app log http requests. Here is a simple way to achieve this, with minimal dependencies. The examples apply only when using default Apache Tomcat embedded web server and Logback as logging implementation.

1. Add dependency logback-access to your project:

<dependency>
  <groupId>ch.qos.logback</groupId>
  <artifactId>logback-access</artifactId>
</dependency>

2. Add Logback access log configuration file to root of classpath:
src/main/resources/logback-access.xml

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <appender name="access_stdout" 
              class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>common</pattern>
        </encoder>
    </appender>

    <appender-ref ref="access_stdout"/>
</configuration>

3. Add a @Configuration class which integrates Logback into Tomcat (Kotlin in this example):

import ch.qos.logback.access.tomcat.LogbackValve
import org.springframework.boot.web.embedded.tomcat.TomcatContextCustomizer
import org.springframework.context.annotation.Bean
import org.springframework.context.annotation.Configuration
import java.nio.file.Files

@Configuration
open class AccessLogConfiguration {

    @Bean
    fun addLogbackAccessValve() = TomcatContextCustomizer { context ->

        javaClass.getResourceAsStream("/logback-access.xml").use {
            Files.createDirectories((context.catalinaBase.toPath()
                .resolve(LogbackValve.DEFAULT_CONFIG_FILE)).parent)

            Files.copy(it, context.catalinaBase.toPath()
                .resolve(LogbackValve.DEFAULT_CONFIG_FILE))
        }

        LogbackValve().let {
            it.isQuiet = true
            context.pipeline.addValve(it)
        }
    }

}

Here we copy the Logback access configuration file into Tomcat embedded web server runtime dir. This circumvents an issue with LogbackValve being initialized in a different class loader context by Tomcat, where it is not able to resolve its configuration file in the Spring Boot app main classpath. If it’s not working, set isQuiet to false to debug.

Now your Spring Boot app logs http requests through the console appender in a typical format:

127.0.0.1 - - [20/feb./2021:22:16:54 +0100] "GET / HTTP/1.1" 200 332

Getting Logstash-compatible JSON-output

In case you use Kibana and need structured log formatting, you can easily achieve this now. First ensure you have the dependency
net.logstash.logback:logstash-logback-encoder added to your build. Then simply use a slightly different
logback-access.xml configuration file:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <appender name="access_stdout" 
              class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="net.logstash.logback.encoder.LogstashAccessEncoder"/>
    </appender>

    <appender-ref ref="access_stdout"/>
</configuration>

Now your access log messages will be structured like this:

{
  "@timestamp": "2021-02-21T17:38:30.813+01:00",
  "@version": "1",
  "message": "127.0.0.1 - - [2021-02-21T17:38:30.813+01:00] \"GET / HTTP/1.1\" 200 332",
  "method": "GET",
  "protocol": "HTTP/1.1",
  "status_code": 200,
  "requested_url": "GET / HTTP/1.1",
  "requested_uri": "/",
  "remote_host": "127.0.0.1",
  "content_length": 332,
  "elapsed_time": 47
}
Categories
Hardware

Sony+Android is a security failure

My last couple of smart phones have been Sony Xperia compacts, the latest being an Xperia Compact XZ1. It is a robust piece of hardware, and it has a perfect size, with its 4.6 inch screen. I dislike big smart phones, and since they are just getting bigger every year, I plan on keeping my current phone for as long as possible. This might prove difficult, due to Sony’s lack of product support. It is already well known that Android is a failure when it comes to phone manufacturers not supporting their products with software updates for a reasonable amount of time, especially within the category of security fixes. This is just another example of the sad state of affairs, I guess.

The phone currently runs Android 9 using the latest Sony firmware version 47.2.A.11.228. Security patch level is dated September of 2019, which means my phone is missing a year’s worth of upstream security fixes to the Android operating system. For example high severity Bluetooth vulnerabilities listed on the February 2020 security bulletin like CVE-2020-0022 (aka “BlueFrag”):

On Android 8.0 to 9.0, a remote attacker within proximity can silently execute arbitrary code with the privileges of the Bluetooth daemon as long as Bluetooth is enabled. No user interaction is required and only the Bluetooth MAC address of the target devices has to be known. For some devices, the Bluetooth MAC address can be deduced from the WiFi MAC address. This vulnerability can lead to theft of personal data and could potentially be used to spread malware (Short-Distance Worm).

https://insinuator.net/2020/02/critical-bluetooth-vulnerability-in-android-cve-2020-0022/

My phone is a critical device in my every day life, and having it hacked like this would certainly lead to a miserable few days. So now I must rigidly remember to keep Bluetooth off, which is inconvenient at best. And I must assume that many of the vulnerabilities listed from October 2019 and going forward, apply to my phone.

Sony customer support

I had picked up information that Sony was cutting support for my phone model early in 2020. Normally I would not bother interacting with large corporation customer support channels, but this time I was annoyed enough to fill out the form. As expected, it was a pain, and the form post failed a couple of times with server side errors. But eventually I got it through and then didn’t hope for anything. After all, attempting to communicate with big corporations is like screaming directly at /dev/null.

My question was regarding the security patch level of my phone and when Sony was planning to fix it, pointing out the Bluetooth vulnerabilities. Very simple. This is the first response I got, translated from Norwegian:

Thank you for contacting Sony Support. It worries us that you are worried about the update. I have confirmed your IMEI number. Your phone is running the latest version of Android. I will forward your question to a specialist to get the best possible information, and for this I need you to answer a couple of questions: 1) which mobile network operator are you using ? 2) regarding your question about security, are you using any Bluetooth accessories ?

Response #1, 22 May 2020, from Sony customer support Norway.

I followed up by providing the information requested. The second response came about a week later:

Thanks for your patience regarding the response time for your question. We understand that you are worried about the mentioned [Bluetooth] vulnerability. Today I have received a reply from a [person] responsible for this at our place, and we have no more information regarding the [firmware] update in your question.

Response #2, 1 June 2020, from Sony customer support Norway.

I am not surprised by this response, nor am I amused. The answer was too vague, so I decided to push them a little harder, by simply asking “does this mean Sony will not fix the mentioned security issues on this phone model ?”. About a week later, the third response came:

Thank you for your email. This is a question related to product development. Unfortunately we cannot answer questions directly related to product development.

Response #3, 30 June 2020, from Sony customer support Norway

At this point, customer support stops providing any references to my actual case and responds with some rubbish about product development information policies. While they could have simply said “no more soup for your phone model” in the very first response, this instead looks more like an attempt to hide behind hazy replies and using stalling tactics.

After some time, I decided to have another go, since I have an allocated case number, and it only requires me to send off an email. This time I state that I am not asking Sony to divulge any internal product development information, along with the following question: “How should I as and end-user deal with this situation, considering the security issues at hand ? Do you recommend continued use of the phone ?”. A day later another response arrives:

We cannot at this time confirm when the next update will happen for this model.

Response #4, 25 August 2020, from Sony customer support Norway

There is really no point in continuing this conversation. I replied with a simple statement of disappointment. Sony later replied with a confirmation on support period, finally:

We would like to inform you that updates on our phones stop after two years, since that is the end of the Sony product warranty period. Of course you still have the customer warranty for this product.

Response #5, 25 August 2020, from Sony customer support Norway

Ok, apparently I have a useless customer warranty on a product with security issues that will never be fixed ? Case closed.

There needs to be a distinction between product warranty period, and product lifetime and security support. A smart phone needs software updates beyond a measly two year period after product launch, just like a regular computer operating system does. Otherwise users are put at risk due to lack of security fixes.

Wasted hardware

I purchased the phone brand new in February 2018. Sony ended its software support for the model with the last update in September 2019. That’s less than two years of support since time of purchase. And my phone is in excellent condition. The built-in battery still has great capacity, the screen is mostly free from scratches, and everything works fine. The hardware is easily capable of running Android 10, and I’m willing to bet also Android 11, to be released later this year.

The hardware has a lifespan that greatly extends beyond the period that Sony is willing to provide security updates for the device. So the end user is forced to either buy a new device prematurely, or risk the consequences of continued use with an increasing number of security vulnerabilities appearing every month. Manufacturers of Android based devices really need to wake up and take responsibility for their products, their customers and the environment. Because that is certainly a guarantee today with Android: you will be left behind if you take good care of your phone and want to keep it for several years.

What are my options ?

In order of likelihood.

  1. Install a custom Android ROM like Lineage OS. But it requires work and time and comes with no guarantees that things will function properly. I am especially concerned about hardware quirks and driver issues. (Still thinking about it, though.)
  2. Continue using the outdated Sony firmware, while limiting risk by keeping Bluetooth off and taking other precations. This is where I am currently at.
  3. Buy a new Android phone. But they are all too big these days, and I am tired of the bad support.
  4. Buy a “dumb” feature-phone and leave oversized smart phones behind. The options are limited. But I suspect I would manage just fine without phone apps in my life.
  5. Buy an iPhone. I’ll give credit to Apple for their device support with software updates over time, and also selling smaller form factors. But I can never be part of Apple’s walled garden, so this is not a realistic option.