Using Argparse and Json together

Question

I am a beginner to Python.

I wanted to know if Argparse and JSON could be used together. Say, I have variables p,q,r

I could add them to argparse as -

parser.add_argument('-p','--param1',help='x variable', required=True)
parser.add_argument('-q','--param2',help='y variable', required=True)
parser.add_argument('-r','--param3',help='z variable', required=True)

Now suppose I wanted to read the same variables from JSON file, is it possible to do it? So I could input the values either from command line or a JSON file.

JSON input file -

{
    "testOwner": "my name",
    "tests": [
        "test1",
        "test2",
        "test3"
    ],

    "testParameters": {
        "test1": {
            "param1": "0",
            "param2": "20",
            "param3" : "True"
        },

        "test2": {
            "param1": "cc"
        }
    }   
}

Answer 1

The args Namespace from parse_args can be transformed into a dictionary with:

argparse_dict = vars(args)

The JSON values are also in a dictionary, say json_dict. You can copy selected values from one dictionary to the other, or do a whole scale update:

argparse_dict.update(json_dict)

This way the json_dict values over write the argparse ones.

If you want to preserve both, you either need to have different argument (key) names, or the values have to be lists, which you can append or extend. That takes a bit more work, starting with using the correct nargs value in argparse.

---

The revised parser produces, with a test input:

In [292]: args=parser.parse_args('-p one -q two -r three'.split())
In [293]: args
Out[293]: Namespace(param1='one', param2='two', param3='three')
In [295]: args_dict = vars(args)    
In [296]: args_dict
Out[296]: {'param1': 'one', 'param2': 'two', 'param3': 'three'}

The JSON string, when parsed (json.loads?) produces a dictionary like:

In [317]: json_dict
Out[317]: 
{'testOwner': 'my name',
 'testParameters': {'test1': {'param1': '0', 'param2': '20', 'param3': 'True'},
  'test2': {'param1': 'cc'}},
 'tests': ['test1', 'test2', 'test3']}

I produced this by pasting your display into my Ipython session, but I think the JSON loader produces the same thing

The argparse values could be added with:

In [318]: json_dict['testParameters']['test3']=args_dict
In [319]: json_dict
Out[319]: 
{'testOwner': 'my name',
 'testParameters': {'test1': {'param1': '0', 'param2': '20', 'param3': 'True'},
  'test2': {'param1': 'cc'},
  'test3': {'param1': 'one', 'param2': 'two', 'param3': 'three'}},
 'tests': ['test1', 'test2', 'test3']}

Here I added it as a 3rd test set, taking (by conincidence) a name from the tests list. json_dict['testParameters']['test2']=args_dict would replace the values of test2.

One way to add the args values to the undefined values of 'test2' is:

In [320]: args_dict1=args_dict.copy()    
In [322]: args_dict1.update(json_dict['testParameters']['test2'])
In [324]: json_dict['testParameters']['test2']=args_dict1
In [325]: json_dict
Out[325]: 
{'testOwner': 'my name',
 'testParameters': {'test1': {'param1': '0', 'param2': '20', 'param3': 'True'},
  'test2': {'param1': 'cc', 'param2': 'two', 'param3': 'three'},
  'test3': {'param1': 'one', 'param2': 'two', 'param3': 'three'}},
 'tests': ['test1', 'test2', 'test3']}

I used this version of update to give priority to the 'cc' value in the JSON dictionary.

Answer 2

Turns out to be pretty easy with the following caveats

1. The setup overrides values in config files with values on the command line
2. It only uses default values if options have not been set on the command line nor the settings file
3. It does not check that the settings in the config file are valid

import argparse
import json

parser = argparse.ArgumentParser()

parser.add_argument('--save_json',
    help='Save settings to file in json format. Ignored in json file')
parser.add_argument('--load_json',
    help='Load settings from file in json format. Command line options override values in file.')

args = parser.parse_args()

if args.load_json:
    with open(args.load_json, 'rt') as f:
        t_args = argparse.Namespace()
        t_args.__dict__.update(json.load(f))
        args = parser.parse_args(namespace=t_args)

# Optional: support for saving settings into a json file
if args.save_json:
    with open(args.save_json, 'wt') as f:
        json.dump(vars(args), f, indent=4)

Answer 3

Given that your JSON file contains a dict of the form:

d = {"name": ["-x", "--xvar"], "help": "Help message", "required": True}

After creating the parser you could "unpack" the dict like so:

parser = argparse.ArgumentParser()
parser.add_argument(*(d.pop("name")), **d) 
# Put the 'name' as name/flag and then unpack the rest of
# the dict as the rest of the arguments
parser.parse_args("--xvar 12".split())
>>> Namespace(xvar='12')

However this forces you to maintain the dict keys to fit the arguments name of the method add_arguments. You also do not have a simple/straight forward way of using more advance behaviors like using the action, type, choices arguments.

Also you would have to change the form of your dict to contain the various arguments you want to use. One solution would be to have the name/flag as the key of the dict in a tuple and the arguments would be a dict:

d = {("-x", "--xvar"): {"help": "Help message for x", "required": True}, 
     ("-y", "--yvar"): {"help": "Help message for y", "required": True}}
for names, args in d.iteritems():
    parser.add_argument(*names, **args) # Use a similar unpacking 'magic' as the first example
parser.parse_args("-x 12 --yvar 42".split())
>>> Namespace(xvar='12', yvar='42')

EDIT Given the comments from the OP it looks like he wants to parse values taken from a JSON file.

d = {"-x": "12", "-y": "42"}
args = []
for item in d.items():
    args.extend(item)
parser.parse_args(args)
>>> Namespace(xvar='12', yvar='42')

EDIT 2

Looking at the argparse documentation this paragraph maybe somewhat relevant.

Answer 4

Here is defaults.json

{
    "param1": "from json",
    "param2": "from json"
}

and here is args.py

import argparse
from pathlib import Path
import json

json_text = Path('defaults.json').read_text()
args = argparse.Namespace(**json.loads(json_text))

parser = argparse.ArgumentParser()
parser.add_argument('--param1', default='from default')
parser.add_argument('--param2', default='from default')
parser.add_argument('--param3', default='from default')
args = parser.parse_args(namespace=args)

print(args)

running it gives the following output

python args.py --param2 'from par'
Namespace(param1='from json', param2='from par', param3='from default')

Answer 5

Some of the answers here are limited in that they neither validate the inputs, nor convert them to the expected types. A simple solution is to construct a list of strings for argparse to [re-]parse.

If your config file is simple (consisting of flags and single-value options) you can do the following:

import argparse
import functools
import json
import operator


js = '{ "class": 10, "no_checksum": "True", "my_string": "Jesus is Lord"}'
da = json.loads(js)
parser = argparse.ArgumentParser()
parser.add_argument('--no_checksum', action='store_true')
parser.add_argument('--class', type=int)
parser.add_argument('--my_string')
pairs = [ [f"--{k}", str(v)] if not v=='True' else [f"--{k}"] for k,v in da.items()]
argv = functools.reduce(operator.iadd, pairs, [])
parser.parse_args(argv)

This uses a list comprehension to build up a list of options and stringified (if necessary) values from the read JSON dictionary. Any option set to "True" is passed without an argument (this is for flags); note that this code does not handle "False" values. (If you want this, use v in ('True', 'False') instead of v=='True'.) The resulting pairs value is a list of lists (either pairs or single flags); this must be flattened (i.e. nesting removed) for argparse, which is what functools.reduce(operator.iadd, pairs, []) is for -- it iteratively and cumulatively applies an incremental add operation to the list, concatenating all sublists into one big list. (The [] initial value is there in case pairs turns out to be empty, which would otherwise break reduce.)

The result is:

Namespace(class=10, my_string='Jesus is Lord', no_checksum=True, test=None)

If your config file contains lists, the code is a bit more complicated:

js = '{ "class": 10, "no_checksum": "True", "evangelists": [ "Matthew", "Mark", "Luke", "John"], "my_string": "Jesus is Lord"}'
da = json.loads(js)
parser.add_argument('--evangelists', nargs='*')
pairs = [ functools.reduce(operator.iadd, [[f"--{k}"], [str(v)] if not isinstance(v,list) else list(map(str,v))]) if not v=='True' else [f"--{k}"] for k,v in da.items()]
argv = functools.reduce(operator.iadd, pairs, [])
parser.parse_args(argv)

This extends the previous code to 1) convert list items to strings, if they are not strings already (list(map(str,v)), which applies the str built-in function to all elements of v); 2) flatten inner list values.

The result:

Namespace(class=10, my_string='Jesus is Lord', no_checksum=True, evangelists=['Matthew', 'Mark', 'Luke', 'John'])

If your argument file is more complicated than this, you probably shouldn't be using argparse, I argue. This solution does have a limitation in that it may not correctly validate certain corner cases.