Jq(1) is a surprisingly powerful command line JSON stream processing tool. I have not used it much, so this post will be a growing collection of random and useful examples to help me remember.
Table of contents
- Flatten an array of JSON objects into lines (tabular form), selecting only particular fields as columns
- Transform an array of values to lines with one element per line.
- Create a JSON array of a list of string values, with or without pretty printing
- Extract two particular values from the first of multiple deeply nested JSON-objects
- Filter stream of JSON objects based on some condition
- Create JSON from command line arguments
- Drill into, transform and select specific objects from structure based on matching
- Count the number of elements in an array
Flatten an array of JSON objects into lines (tabular form), selecting only particular fields as columns
# Given JSON data:
$ cat data.json
{
"items": [
{
"id": "obj-1",
"info": {
"desc": "Description of Object 1",
"type": "sometype"
}
},
{
"id": "obj-2",
"info": {
"desc": "Description of Object 2",
"type": "sometype"
}
}
]
}
# Transform this into a table,
# where each line is "<id> <TAB> <description>":
$ jq -r '.items[]
| .id + "\t" + .info.desc' data.json
obj-1 Description of Object 1
obj-2 Description of Object 2Use option -r to avoid quoting the output strings, making them more suitable for parsing by for instance shell scripts.
Extract values from an array to lines with one element per line
# Given a file data.json with the following contents:
{
"nftables": [
{
"metainfo": {
"version": "1.0.9",
"release_name": "Old Doc Yak #3",
"json_schema_version": 1
}
},
{
"set": {
"family": "inet",
"name": "addr-set-sshd",
"table": "f2b-table",
"type": "ipv4_addr",
"handle": 13,
"elem": [
"10.0.0.1", "10.0.0.2", "10.0.0.3"
]
}
}
]
}
# we can extract all string values from an array
# and print one value per line:
$ jq -r '.nftables[1].set.elem[]' data.json
10.0.0.1
10.0.0.2
10.0.0.3
The jq expression drills down to the array we are interested in and then uses the iteration operator to explode the array into separate values, which are printed one per line. Uses option -r to avoid quoting the output strings.
Create a JSON array of a list of string values, with or without pretty printing
$ echo '"foo" "bar" "b a z"'|jq -s .
[
"foo",
"bar",
"b a z"
]$ echo '"foo" "bar" "b a z"'|jq -cs .
["foo","bar","b a z"]The option -s/--slurp is what makes jq pack the values in an array here. The option -c/--compact-output toggles pretty printing.
Extract two particular values from the first of multiple deeply nested JSON-objects
$ cat data.json
{
"data": [
{
"id": 1,
"results": [
{
"type": "result",
"objs": [
{
"a": "foo",
"b": "bar"
}
]
}
]
},
{
"id": 2,
"results": [
{
"type": "result",
"objs": [
{
"a": "foo2",
"b": "bar2"
}
]
}
]
}
]
}
$ cat data.json|\
jq -r '.data[0].results[0].objs[0].a,
.data[0].results[0].objs[0].b'
foo
barThe option -r/--raw-output causes jq not to quote the extracted string values. Makes use of the values in shell scripts easier.
Filter stream of JSON objects based on some condition
$ cat data.ndjson
{ "id": 1, "color": "red" }
{ "id": 2, "color": "green" }
{ "id": 3, "color": "blue" }
$ cat data.ndjson | jq -s 'map(select(.color == "blue"))'
[
{
"id": 3,
"color": "blue"
}
]
Again we see the -s/--slurp option to read all objects into an array before applying our filtering code, causing the result to be an array of matching objects.
Create JSON from command line arguments
$ jq -n --arg a 1 --arg b 2 '{"a":$a, "b":$b}'
{
"a": "1",
"b": "2"
}You can declare variables which jq makes available when constructing JSON, as can be seen in the example. The option -n/--null-input tells jq to not read anything on stdin, just output stuff. Notice that --arg by default treats values as JSON strings. If you need to encode values of a and b as real numbers in the above example, use --argjson instead:
$ jq -n --argjson a 1 --argjson b 2 '{"a":$a, "b":$b}'
{
"a": 1,
"b": 2
}Drill into, transform and select specific objects from structure based on matching
Given the following input structure:
$ cat structure.json
{
"items": [
{
"id": 1,
"name": "The first item",
"subitems": [
{
"name": "foo", "value": 0
},
{
"name": "bar", "value": 1
},
{
"name": "baz", "value": 2
}
]
},
{
"id": 2,
"name": "The second item",
"subitems": [
{
"name": "a", "value": 0
},
{
"name": "b", "value": 1
},
{
"name": "c", "value": 2
}
]
},
{
"id": 3,
"name": "The third item",
"subitems": [
{
"name": "Alpha α", "value": 0
},
{
"name": "Beta β", "value": 1
},
{
"name": "Gamma γ", "value": 2
}
]
}
]
}We can extract particular sub items from a particular item in the items list by using select and array iterator in a filter pipeline:
$ jq '.items[]
| select(.name == "The first item")
| .subitems[]
| select(.value > 1)' structure.json
{
"name": "baz",
"value": 2
}Filtering explained in steps:
.items[]– selects theitemskey of the outer object and applies the iteration operator[]to the array. This turns the array into a stream of item values which can be processed further, one by one.select(.name == "The first item")– filters stream of item objects, selecting only those with a keynamehaving the value"The first item"..subitems[]– selects the key.subitemsof input values and explodes the arrays into separate values for further processing, using the iteration operator.select(.value > 1)– selects only subitems with a value greather than 1.- Resulting in a single matching subitem object, which becomes the output.
Build the filtering pipeline up from the first to the fourth step, each time adding the next |...., to better understand how the data is transformed and flows.
If we alter the last |...select(.value > 1) matching filter, we can see that multiple JSON objects will be output by jq:
$ jq '.items[] | select(.name == "The first item") | .subitems[] | select(.value > 0)' structure.json
{
"name": "bar",
"value": 1
}
{
"name": "baz",
"value": 2
}And if wrapping this output in an outer array is desired:
$ jq '.items[] | select(.name == "The first item") | .subitems[] | select(.value > 0)' structure.json | jq -s
[
{
"name": "bar",
"value": 1
},
{
"name": "baz",
"value": 2
}
]Notice how we actually use a shell pipe to send the output of the first jq invocation to a new one using slurp mode to wrap input objects in an array. (Don’t confuse shell pipes with jq filtering pipes when mixing the two !)
Count the number of elements in an array
$ cat array.json
[1, { "a":"b" }, true]
$ jq length < array.json
3The length function can count the number of elements in an array. It can also count some other datatypes, like strings (number of characters) and objects (number of keys).
