Serialization routines to/from a meta.json based directory

Module Contents


load_meta_based_dir(root_dir[, auxfile_types_member, ...])

Load the contents of a root_dir into a dict.

write_meta_based_dir(root_dir, valuedict[, ...])

Write a dictionary of quantities to a directory.

write_obj_to_meta_based_dir(obj, dirname, ...[, ...])

Write the contents of obj to dirname using a 'meta.json' file and an auxfile-types dictionary.

write_dict_to_json_or_pkl_files(d, dirname)

Write each element of d into a separate file in dirname.


QUICK_LOAD_MAX_SIZE, auxfile_types_member='auxfile_types', ignore_meta=('type',), separate_auxfiletypes=False, quick_load=False)

Load the contents of a root_dir into a dict.

The de-serialization uses the ‘meta.json’ file within root_dir to describe how the directory was serialized.



The directory name.

auxfile_types_memberstr, optional

The key within meta.json that is used to describe how other members have been serialized into files. Unless you know what you’re doing, leave this as the default.

ignore_metatuple, optional

Keys within meta.json that should be ignored, i.e. not loaded into elements of the returned dict. By default, “type” is in this category because it describes a class name to be built and is used in a separate first-pass processing to construct a object. Unless you know what you’re doing, leave this as the default.

separate_auxfiletypesbool, optional

If True, then return the auxfile_types_member element (a dict describing how quantities that aren’t in ‘meta.json’ have been serialized) as a separate return value, instead of placing it within the returned dict.

quick_loadbool, optional

Setting this to True skips the loading of members that may take a long time to load, namely those in separate files whose files are large. When the loading of an attribute is skipped, it is set to None.



A dictionary of the quantities in ‘meta.json’ plus any loaded from the auxiliary files.


Only returned as a separate value when separate_auxfiletypes=True. A dict describing how members of loaded_qtys that weren’t loaded directly from ‘meta.json’ were serialized., valuedict, auxfile_types=None, init_meta=None)

Write a dictionary of quantities to a directory.

Write the dictionary by placing everything in a ‘meta.json’ file except for special key/value pairs (“special” usually because they lend themselves to an non-JSON format or they simply cannot be rendered as JSON) which are placed in “auxiliary” files formatted according to auxfile_types (which itself is saved in meta.json).



The directory to write to (will be created if needed).


The dictionary of values to serialize to disk.

auxfile_typesdict, optional

A dictionary whose keys are a subset of the keys of valuedict, and whose values are known “aux-file” types. auxfile_types[key] says that valuedict[key] should be serialized into a separate file (whose name is usually key + an appropriate extension) of the given format rather than be included directly in ‘meta.json`. If None, this dictionary is assumed to be valuedict[‘auxfile_types’].

init_metadict, optional

A dictionary of “initial” meta-data to be included in the ‘meta.json’ (but that isn’t in valuedict). For example, the class name of an object is often stored as in the “type” field of meta.json when the_model objects .__dict__ is used as valuedict.


None, dirname, auxfile_types_member, omit_attributes=(), include_attributes=None, additional_meta=None)

Write the contents of obj to dirname using a ‘meta.json’ file and an auxfile-types dictionary.

This is similar to write_meta_based_dir(), except it takes an object (obj) whose .__dict__, minus omitted attributes, is used as the dictionary to write and whose auxfile-types comes from another object attribute.



the object to serialize


the directory name

auxfile_types_memberstr or None

the name of the attribute within obj that holds the dictionary mapping of attributes to auxiliary-file types. Usually this is “auxfile_types”.

omit_attributeslist or tuple

List of (string-valued) names of attributes to omit when serializing this object. Usually you should just leave this empty.

include_attributeslist or tuple or None

A list of (string-valued) names of attributs to specifically include when serializing this object. If None, then all attributes are included except those specifically omitted via omit_attributes. If include_attributes is not None then omit_attributes is ignored.

additional_metadict, optional

A dictionary of additional meta-data to be included in the ‘meta.json’ file (but that isn’t an attribute of obj).


None, dirname)

Write each element of d into a separate file in dirname.

If the element is json-able, it is JSON-serialized and the “.json” extension is used. If not, pickle is used to serialize the element, and the “.pkl” extension is used. This is the reverse of _read_json_or_pkl_files_to_dict().



the dictionary of elements to serialize.


the directory name.