pygsti.io.mongodb
Serialization routines to/from a MongoDB database
Module Contents
Functions
|
Read a document containing links to auxiliary documents from a MongoDB database. |
|
Load the contents of a MongoDB document into a dict. |
|
Write the attributes of an object to a MongoDB database, potentially as multiple documents. |
|
Similar to write_obj_to_mongodb_auxtree, but just collect write operations and update a main-doc dictionary. |
|
Write a dictionary of quantities to a MongoDB database, potentially as multiple documents. |
|
Similar to write_auxtree_to_mongodb, but just collect write operations and update a main-doc dictionary. |
|
Remove some or all of the MongoDB documents written by write_auxtree_to_mongodb |
|
Read a dictionary serialized via |
|
Write each element of d as a separate document in a MongoDB collection |
|
Similar to write_dict_to_mongodb, but just collect write operations and update a main-doc dictionary. |
|
Remove elements of (separate documents) of a dictionary stored in a MongoDB collection |
Create, if not existing already, indices useful for speeding up pyGSTi MongoDB operations. |
- pygsti.io.mongodb.read_auxtree_from_mongodb(mongodb, collection_name, doc_id, auxfile_types_member='auxfile_types', ignore_meta=('_id', 'type'), separate_auxfiletypes=False, quick_load=False)
Read a document containing links to auxiliary documents from a MongoDB database.
Parameters
- mongodbpymongo.database.Database
The MongoDB instance to load data from.
- collection_namestr
the MongoDB collection within mongodb to read from.
- doc_idbson.objectid.ObjectId
The identifier, of the root document to load from the database.
- auxfile_types_memberstr
the name of the attribute within the document that holds the dictionary mapping of attributes to auxiliary-file (document) types. Usually this is “auxfile_types”.
- ignore_metatuple, optional
Keys within the root document that should be ignored, i.e. not loaded into elements of the returned dict. By default, “_id” and “type” are in this category because they give the database id and a class name to be built, respectively, and are not needed in the constructed dictionary. Unless you know what you’re doing, leave this as the default.
- separate_auxfiletypesbool, optional
If True, then return the auxfile_types_member element (a dict describing how quantities that aren’t in the main document have been serialized) as a separate return value, instead of placing it within the returned dict.
- quick_loadbool, optional
Setting this to True skips the loading of members that may take a long time to load, namely those in separate documents that are large. When the loading of an attribute is skipped, it is set to None.
Returns
- loaded_qtysdict
A dictionary of the quantities in the main document plus any loaded from the auxiliary documents.
- auxfile_typesdict
Only returned as a separate value when separate_auxfiletypes=True. A dict describing how members of loaded_qtys that weren’t loaded directly from the main document were serialized.
- pygsti.io.mongodb.read_auxtree_from_mongodb_doc(mongodb, doc, auxfile_types_member='auxfile_types', ignore_meta=('_id', 'type'), separate_auxfiletypes=False, quick_load=False)
Load the contents of a MongoDB document into a dict.
The de-serialization possibly uses metadata within to root document to describe how associated data is stored in other collections.
Parameters
- mongodbpymongo.database.Database
The MongoDB instance to load data from.
- docdict
The already-retrieved main document being read in.
- auxfile_types_memberstr, optional
The key within the root document that is used to describe how other members have been serialized into documents. Unless you know what you’re doing, leave this as the default.
- ignore_metatuple, optional
Keys within the root document that should be ignored, i.e. not loaded into elements of the returned dict. By default, “_id” and “type” are in this category because they give the database id and a class name to be built, respectively, and are not needed in the constructed dictionary. Unless you know what you’re doing, leave this as the default.
- separate_auxfiletypesbool, optional
If True, then return the auxfile_types_member element (a dict describing how quantities that aren’t in root document have been serialized) as a separate return value, instead of placing it within the returned dict.
- quick_loadbool, optional
Setting this to True skips the loading of members that may take a long time to load, namely those in separate files whose files are large. When the loading of an attribute is skipped, it is set to None.
Returns
- loaded_qtysdict
A dictionary of the quantities in ‘meta.json’ plus any loaded from the auxiliary files.
- auxfile_typesdict
Only returned as a separate value when separate_auxfiletypes=True. A dict describing how members of loaded_qtys that weren’t loaded directly from the root document were serialized.
- pygsti.io.mongodb.write_obj_to_mongodb_auxtree(obj, mongodb, collection_name, doc_id, auxfile_types_member, omit_attributes=(), include_attributes=None, additional_meta=None, session=None, overwrite_existing=False)
Write the attributes of an object to a MongoDB database, potentially as multiple documents.
Parameters
- objobject
The object that is to be written.
- mongodbpymongo.database.Database
The MongoDB instance to write data to.
- collection_namestr
the MongoDB collection within mongodb to write to.
- doc_idbson.objectid.ObjectId
The identifier, of the root document to store in the database. If None a new id will be created.
- auxfile_types_memberstr, optional
The attribute of obj that is used to describe how other members should be serialized into separate “auxiliary” documents. Unless you know what you’re doing, leave this as the default.
- omit_attributeslist or tuple
List of (string-valued) names of attributes to omit when serializing this object. Usually you should just leave this empty.
- include_attributeslist or tuple or None
A list of (string-valued) names of attributs to specifically include when serializing this object. If None, then all attributes are included except those specifically omitted via omit_attributes. If include_attributes is not None then omit_attributes is ignored.
- additional_metadict, optional
A dictionary of additional meta-data to be included in the main document (but that isn’t an attribute of obj).
- sessionpymongo.client_session.ClientSession, optional
MongoDB session object to use when interacting with the MongoDB database. This can be used to implement transactions among other things.
- overwrite_existingbool, optional
Whether existing documents should be overwritten. The default of False causes a ValueError to be raised if a document with the given doc_id already exists. Setting this to True mimics the behaviour of a typical filesystem, where writing to a path can be done regardless of whether it already exists.
Returns
- bson.objectid.ObjectId
The identifer of the root document that was written.
- pygsti.io.mongodb.add_obj_auxtree_write_ops_and_update_doc(obj, doc, write_ops, mongodb, collection_name, auxfile_types_member, omit_attributes=(), include_attributes=None, additional_meta=None, overwrite_existing=False)
Similar to write_obj_to_mongodb_auxtree, but just collect write operations and update a main-doc dictionary.
This function effectively performs all the heavy-lifting to write an object to a MongoDB database without actually executing any write operations. Instead, a dictionary representing the main document (which we typically assume will be written later) is updated and additional write operations (for auxiliary documents) are added to a
WriteOpsByCollectionobject. This function is intended for use within aMongoSerializable-derived object’s _add_auxiliary_write_ops_and_update_doc method.Parameters
- objobject
The object that is to be written.
- docdict
The root-document data, which is updated as needed and is expected to be initialized at least with an _id key-value pair.
- write_opsWriteOpsByCollection
An object that keeps track of pymongo write operations on a per-collection basis. This object accumulates write operations to be performed at some point in the future.
- mongodbpymongo.database.Database
The MongoDB instance that is planned to be written to. Used to test for existing records and not to write to, as writing is assumed to be done later, potentially as a bulk write operaiton.
- collection_namestr
the MongoDB collection within mongodb that is planned to write to.
- auxfile_types_memberstr, optional
The attribute of obj that is used to describe how other members should be serialized into separate “auxiliary” documents. Unless you know what you’re doing, leave this as the default.
- omit_attributeslist or tuple
List of (string-valued) names of attributes to omit when serializing this object. Usually you should just leave this empty.
- include_attributeslist or tuple or None
A list of (string-valued) names of attributs to specifically include when serializing this object. If None, then all attributes are included except those specifically omitted via omit_attributes. If include_attributes is not None then omit_attributes is ignored.
- additional_metadict, optional
A dictionary of additional meta-data to be included in the main document (but that isn’t an attribute of obj).
- overwrite_existingbool, optional
Whether existing documents should be overwritten. The default of False causes a ValueError to be raised if a document with the given doc_id already exists. Setting this to True mimics the behaviour of a typical filesystem, where writing to a path can be done regardless of whether it already exists.
Returns
- bson.objectid.ObjectId
The identifer of the root document that was written.
- pygsti.io.mongodb.write_auxtree_to_mongodb(mongodb, collection_name, doc_id, valuedict, auxfile_types=None, init_meta=None, session=None, overwrite_existing=False)
Write a dictionary of quantities to a MongoDB database, potentially as multiple documents.
Write the dictionary by placing everything in valuedict into a root document except for special key/value pairs (“special” usually because they lend themselves to an non-JSON format or they can be particularly large and may exceed MongoDB’s maximum document size) which are placed in “auxiliary” documents formatted according to auxfile_types (which itself is saved in the root document).
Parameters
- mongodbpymongo.database.Database
The MongoDB instance to write data to.
- collection_namestr
the MongoDB collection within mongodb to write to.
- doc_idbson.objectid.ObjectId
The identifier, of the root document to store in the database. If None a new id will be created.
- valuedictdict
The dictionary of values to serialize to disk.
- auxfile_typesdict, optional
A dictionary whose keys are a subset of the keys of valuedict, and whose values are known “aux-file” types. auxfile_types[key] says that valuedict[key] should be serialized into a separate document with the given format rather than be included directly in the root document. If None, this dictionary is assumed to be valuedict[‘auxfile_types’].
- init_metadict, optional
A dictionary of “initial” meta-data to be included in the root document (but that isn’t in valuedict). For example, the class name of an object is often stored as in the “type” field of meta.json when the_model objects .__dict__ is used as valuedict.
- sessionpymongo.client_session.ClientSession, optional
MongoDB session object to use when interacting with the MongoDB database. This can be used to implement transactions among other things.
- overwrite_existingbool, optional
Whether existing documents should be overwritten. The default of False causes a ValueError to be raised if a document with the given doc_id already exists. Setting this to True mimics the behaviour of a typical filesystem, where writing to a path can be done regardless of whether it already exists.
Returns
- bson.objectid.ObjectId
The identifer of the root document that was written.
- pygsti.io.mongodb.add_auxtree_write_ops_and_update_doc(doc, write_ops, mongodb, collection_name, valuedict, auxfile_types=None, init_meta=None, overwrite_existing=False)
Similar to write_auxtree_to_mongodb, but just collect write operations and update a main-doc dictionary.
This function effectively performs all the heavy-lifting to write a dictionary to multiple documents within a MongoDB database without actually executing any write operations. Instead, a dictionary representing the main document (which we typically assume will be written later) is updated and additional write operations (for auxiliary documents) are added to a
WriteOpsByCollectionobject. This function is intended for use within aMongoSerializable-derived object’s _add_auxiliary_write_ops_and_update_doc method.Parameters
- docdict
The root-document data, which is updated as needed and is expected to be initialized at least with an _id key-value pair.
- write_opsWriteOpsByCollection
An object that keeps track of pymongo write operations on a per-collection basis. This object accumulates write operations to be performed at some point in the future.
- mongodbpymongo.database.Database
The MongoDB instance that is planned to be written to. Used to test for existing records and not to write to, as writing is assumed to be done later, potentially as a bulk write operaiton.
- collection_namestr
the MongoDB collection within mongodb that is planned to write to.
- valuedictdict
The dictionary of values to serialize to disk.
- auxfile_typesdict, optional
A dictionary whose keys are a subset of the keys of valuedict, and whose values are known “aux-file” types. auxfile_types[key] says that valuedict[key] should be serialized into a separate document with the given format rather than be included directly in the root document. If None, this dictionary is assumed to be valuedict[‘auxfile_types’].
- init_metadict, optional
A dictionary of “initial” meta-data to be included in the root document (but that isn’t in valuedict). For example, the class name of an object is often stored as in the “type” field of meta.json when the_model objects .__dict__ is used as valuedict.
- overwrite_existingbool, optional
Whether existing documents should be overwritten. The default of False causes a ValueError to be raised if a document with the given doc_id already exists. Setting this to True mimics the behaviour of a typical filesystem, where writing to a path can be done regardless of whether it already exists.
Returns
- bson.objectid.ObjectId
The identifer of the root document that was written.
- pygsti.io.mongodb.remove_auxtree_from_mongodb(mongodb, collection_name, doc_id, auxfile_types_member='auxfile_types', session=None, recursive=None)
Remove some or all of the MongoDB documents written by write_auxtree_to_mongodb
Removes a root document and possibly auxiliary documents.
Parameters
- mongodbpymongo.database.Database
The MongoDB instance to remove documents from.
- collection_namestr
the MongoDB collection within mongodb to remove document from.
- doc_idbson.objectid.ObjectId
The identifier of the root document stored in the database.
- auxfile_types_memberstr, optional
The key of the stored document used to describe how other members are serialized into separate “auxiliary” documents. Unless you know what you’re doing, leave this as the default.
- sessionpymongo.client_session.ClientSession, optional
MongoDB session object to use when interacting with the MongoDB database. This can be used to implement transactions among other things.
- recursiveRecursiveRemovalSpecification, optional
An object that filters the type of documents that are removed. Used when working with inter-related experiment designs, data, and results objects to only remove the types of documents you know aren’t being shared with other documents.
Returns
- pymongo.results.DeleteResult
The result of deleting (or attempting to delete) the root record
- pygsti.io.mongodb.read_dict_from_mongodb(mongodb, collection_name, identifying_metadata)
Read a dictionary serialized via
write_dict_to_mongodb()into a dictionary.The elements of the constructed dictionary are stored as a separate documents in a the specified MongoDB collection.
Parameters
- mongodbpymongo.database.Database
The MongoDB instance to read data from.
- collection_namestr
the MongoDB collection within mongodb to read from.
- identifying_metadatadict
JSON-able metadata that identifies the dictionary being retrieved.
Returns
dict
- pygsti.io.mongodb.write_dict_to_mongodb(d, mongodb, collection_name, identifying_metadata, overwrite_existing=False, session=None)
Write each element of d as a separate document in a MongoDB collection
A document corresponding to each (key, value) pair of d is created that contains: 1. the metadata identifying the collection (identifying_metadata) 2. the pair’s key, stored under the key “key” 3. the pair’s value, stored under the key “value”
If the element is json-able, it’s value is written as a JSON-like dictionary. If not, pickle is used to serialize the element and store it in a bson.binary.Binary object within the database.
Parameters
- ddict
the dictionary of elements to serialize.
- mongodbpymongo.database.Database
The MongoDB instance to write data to.
- collection_namestr
the MongoDB collection within mongodb to write to.
- identifying_metadatadict
JSON-able metadata that identifies the dictionary being serialized. This metadata should be saved for later retrieving the elements of d from mongodb_collection.
- overwrite_existingbool, optional
Whether existing documents should be overwritten. The default of False causes a ValueError to be raised if a document with the given doc_id already exists.
- sessionpymongo.client_session.ClientSession, optional
MongoDB session object to use when interacting with the MongoDB database. This can be used to implement transactions among other things.
Returns
None
- pygsti.io.mongodb.add_dict_to_mongodb_write_ops(d, write_ops, mongodb, collection_name, identifying_metadata, overwrite_existing)
Similar to write_dict_to_mongodb, but just collect write operations and update a main-doc dictionary.
Parameters
- ddict
the dictionary of elements to serialize.
- write_opsWriteOpsByCollection
An object that keeps track of pymongo write operations on a per-collection basis. This object accumulates write operations to be performed at some point in the future.
- mongodbpymongo.database.Database
The MongoDB instance that is planned to be written to. Used to test for existing records and not to write to, as writing is assumed to be done later, potentially as a bulk write operaiton.
- collection_namestr
the MongoDB collection within mongodb that is planned to write to.
- identifying_metadatadict
JSON-able metadata that identifies the dictionary being serialized. This metadata should be saved for later retrieving the elements of d from mongodb_collection.
- overwrite_existingbool, optional
Whether existing documents should be overwritten. The default of False causes a ValueError to be raised if a document with the given doc_id already exists.
Returns
None
- pygsti.io.mongodb.remove_dict_from_mongodb(mongodb, collection_name, identifying_metadata, session=None)
Remove elements of (separate documents) of a dictionary stored in a MongoDB collection
Parameters
- mongodbpymongo.database.Database
The MongoDB instance to remove data from.
- collection_namestr
the MongoDB collection within mongodb to remove documents from.
- identifying_metadatadict
JSON-able metadata that identifies the dictionary being serialized.
- sessionpymongo.client_session.ClientSession, optional
MongoDB session object to use when interacting with the MongoDB database. This can be used to implement transactions among other things.
Returns
None
- pygsti.io.mongodb.create_mongodb_indices_for_pygsti_collections(mongodb)
Create, if not existing already, indices useful for speeding up pyGSTi MongoDB operations.
Indices are created as necessary within pygsti_* collections. While not necessary for database operations, these indices may dramatically speed up the reading and writing of pygsti objects to/from a Mongo database. You only need to call this once per database, typically when the database is first setup.
Parameters
- mongodbpymongo.database.Database
The MongoDB instance to create indices in.
Returns
None