Deep JSON merge

Question

I have multiple JSON files that I'd like to merge into one. Some have the same root element but different children. I don't want to overwrite the children but too extend them if they have the same parent element. I've tried this answer, but it doesn't work:

jq: error (at file2.json:0): array ([{"title":"...) and array ([{"title":"...) cannot be multiplied

Sample files and wanted result (Gist)

Thank you in advance.

warning, the sample files are invalid json as they have some dangling commas before }. (for instance after "key": "12345610") — user803422, May 04 '17 at 21:04
Thanks. It should be fixed now, at least JSONLint says so now :) — Leo, May 06 '17 at 08:50
Your question has the potential to be quite interesting but because the data has a recursive structure, it would be helpful if you could elucidate the nature of the merge that you are expecting. Also, it would be helpful if you could clarify what you mean by "root element" in relation to the files and the array(s) they contain. Does each of the input files contain an array with exactly one JSON object in it? — peak, May 08 '17 at 22:18
@peak I suspect that by "root element", OP simply means relative nesting level in respect to merging. Unfortunately my answer at this stage has only reproduced the desired output based on the structure of the test files as you can see. — hmedia1, May 11 '17 at 18:54

score 1 · Answer 1 · answered Aug 18 '17 at 00:37

Here is a recursive solution which uses group_by(.key) to decide which objects to combine. This could be a little simpler if .children were more uniform. Sometimes it's absent in the sample data and sometimes it's the unusual value [{}].

def merge:

    def kids:
        map(
            .children
          | if length<1 then empty else .[] end
        )
      | if length<1 then {} else {children:merge} end
    ; 

    def mergegroup:
      {
          title: .[0].title
        , key:   .[0].key
      } + kids
    ;

    if   .==[{}] then .
    else group_by(.key) | map(mergegroup)
    end
;

[ .[] | .[] ] | merge

When run with the -s option as follows

jq -M -s -f filter.jq file1.json file2.json

It produces the following output.

[
  {
    "title": "Title1",
    "key": "12345678",
    "children": [
      {
        "title": "SubTitle2",
        "key": "123456713",
        "children": [
          {}
        ]
      },
      {
        "title": "SubTitle1",
        "key": "12345679",
        "children": [
          {
            "title": "SubSubTitle1",
            "key": "12345610"
          },
          {
            "title": "SubSubTitle2",
            "key": "12345611"
          },
          {
            "title": "DifferentSubSubTitle1",
            "key": "12345612"
          }
        ]
      }
    ]
  }
]

If the ordering of the objects within the .children matters then an a sort_by can be added to the {children:merge} expression, e.g. {children:merge|sort_by(.key)}

score 0 · Answer 2 · edited May 23 '17 at 12:10

Here is something that will reproduce your desired result. It's by no means automatic, It's really a proof of concept at this stage.

One liner:

jq -s '. as $in | ($in[0][].children[].children + $in[1][].children[0].children | unique) as $a1 | $in[1][].children[1] as $s1 | $in[0] | .[0].children[0].children = ($a1) | .[0].children += [$s1]' file1.json file2.json

Multi line breakdown (Copy/Paste):

jq -s '. as $in 
| ($in[0][].children[].children + $in[1][].children[0].children
| unique) as $a1
| $in[1][].children[1] as $s1
| $in[0]
| .[0].children[0].children = ($a1)
| .[0].children += [$s1]' file1.json file2.json

Where:

$in : file1.json and file2.json combined input
$a1: merged "SubSubTitle" array
$s1: second subtitle object

I suspect the reason this didn't work was because your schema is different and has nested arrays.

I find it quite hypnotic looking at this, it would be good if you could elaborate a bit on how fixed the structure is and what the requirements are.

Deep JSON merge

2 Answers2