Passing nested parameters in the extra_params of the load_context in Kedro

Question

I am trying to load a Kedro context with some extra parameters. My intention is to update the configs in parameters.yml with only the ones passed in extra_params (so rest of the configs should remain same). I will then use this instance of context to run some nodes/pipelines (Note: I don't want to modify the parameters.yml file, just update it for this run). I am able to do so for every configs defined at the top/root level in parameters.yml but only the passed configs are persisted in case of nested parameters.

To replicate the issue,

parameters.yml

config_root1: "name"
config_root2: "surname"
config_root3:
   config_leaf1: 10
   config_leaf2: 20
   config_leaf3: 30

Extra params defined as,

extra_params = {'config_root1': 'new_name', 'config_root3': {'config_leaf1': 11}}

Loading the context with extra_params by,

from kedro.framework.context import load_context    
context = load_context(proj_path, extra_params=extra_params)

The parameters in context is updated to,

config_root1: "new_name"
config_root2: "surname"
config_root3:
   config_leaf1: 11

Points to note,

1. config_root1 was at root level and value was passed in extra_params, hence it was updated.
2. config_root2 was at root level and no value was passed, hence it remains the same.
3. config_leaf1 was at leaf level, value passed hence updated.
4. config_leaf{2, 3} were at leaf level, no value passed hence dropped.

It seems the issue has to do with how dictionary and nested dictionary are merged.

Any workarounds other than always passing all of the parameters in extra_params (even if only few needs to be changed)?

Thank you.

Answer 1

The issue in the replacement of key config_root3 to new value, and it cannot be omited easily.

I guess the best way to do it would be creation of two separate configs which override each other with Additional configuration environment, but it requires changes in the way you work with load_context.

Additionally, you can use config loader and merge parameters manually, then provide merge result as extra_params.

Answer 2

extra_params or the kedro run --params cli now accepts nested dict update.

With CLI:

kedro run --params=nested_key.nested_subkey:<value>

With Python API:

params = {'nested_level1': 
     {'nested_level2' : 'some_value'}
}

with KedroSession.create(extra_params=params
                        ) as session:
    session.run()