resistics_readers.spam.spam module¶

resistics_readers.spam.spam.update_xtr_data(xtr_data: Dict[str, Any], section: str, key: str, val: str) → Dict[str, Any][source]¶

Decide how to deal with an XTR file entry

Parameters

xtr_data (Dict[str, Any]) – The dictionary with the existing XTR data
section (str) – The section which has the key, value pair
key (str) – The key
val (str) – the value

Returns

The updated XTR dictionary

Return type

Dict[str, Any]

resistics_readers.spam.spam.read_xtr(xtr_path: pathlib.Path) → Dict[str, Any][source]¶

Function to read an XTR file.

XTR files are similar to INI files but with duplicate entries making them more annoying to read.

Parameters: xtr_path (Path) – Path to the XTR file
Returns: Data from the XTR file
Return type: Dict[str, Any]

pydantic model resistics_readers.spam.spam.TimeMetadataXTR[source]¶

Bases: resistics_readers.multifile.TimeMetadataSingle

This is an extension of TimeMetadataSingle for a single XTR file

Show JSON schema

{
   "title": "TimeMetadataXTR",
   "description": "This is an extension of TimeMetadataSingle for a single XTR file",
   "type": "object",
   "properties": {
      "file_info": {
         "$ref": "#/definitions/ResisticsFile"
      },
      "fs": {
         "title": "Fs",
         "type": "number"
      },
      "chans": {
         "title": "Chans",
         "type": "array",
         "items": {
            "type": "string"
         }
      },
      "n_chans": {
         "title": "N Chans",
         "type": "integer"
      },
      "n_samples": {
         "title": "N Samples",
         "type": "integer"
      },
      "first_time": {
         "title": "First Time",
         "pattern": "%Y-%m-%d %H:%M:%S.%f_%o_%q_%v",
         "examples": [
            "2021-01-01 00:00:00.000061_035156_250000_000000"
         ]
      },
      "last_time": {
         "title": "Last Time",
         "pattern": "%Y-%m-%d %H:%M:%S.%f_%o_%q_%v",
         "examples": [
            "2021-01-01 00:00:00.000061_035156_250000_000000"
         ]
      },
      "system": {
         "title": "System",
         "default": "",
         "type": "string"
      },
      "serial": {
         "title": "Serial",
         "default": "",
         "type": "string"
      },
      "wgs84_latitude": {
         "title": "Wgs84 Latitude",
         "default": -999.0,
         "type": "number"
      },
      "wgs84_longitude": {
         "title": "Wgs84 Longitude",
         "default": -999.0,
         "type": "number"
      },
      "easting": {
         "title": "Easting",
         "default": -999.0,
         "type": "number"
      },
      "northing": {
         "title": "Northing",
         "default": -999.0,
         "type": "number"
      },
      "elevation": {
         "title": "Elevation",
         "default": -999.0,
         "type": "number"
      },
      "chans_metadata": {
         "title": "Chans Metadata",
         "type": "object",
         "additionalProperties": {
            "$ref": "#/definitions/ChanMetadata"
         }
      },
      "history": {
         "title": "History",
         "default": {
            "records": []
         },
         "allOf": [
            {
               "$ref": "#/definitions/History"
            }
         ]
      },
      "data_file": {
         "title": "Data File",
         "type": "string"
      },
      "xtr_file": {
         "title": "Xtr File",
         "type": "string"
      },
      "data_byte_start": {
         "title": "Data Byte Start",
         "type": "integer"
      },
      "rec_chans": {
         "title": "Rec Chans",
         "type": "integer"
      }
   },
   "required": [
      "fs",
      "chans",
      "n_samples",
      "first_time",
      "last_time",
      "chans_metadata",
      "data_file",
      "xtr_file",
      "data_byte_start",
      "rec_chans"
   ],
   "definitions": {
      "ResisticsFile": {
         "title": "ResisticsFile",
         "description": "Required information for writing out a resistics file",
         "type": "object",
         "properties": {
            "created_on_local": {
               "title": "Created On Local",
               "type": "string",
               "format": "date-time"
            },
            "created_on_utc": {
               "title": "Created On Utc",
               "type": "string",
               "format": "date-time"
            },
            "version": {
               "title": "Version",
               "type": "string"
            }
         }
      },
      "ChanMetadata": {
         "title": "ChanMetadata",
         "description": "Channel metadata",
         "type": "object",
         "properties": {
            "name": {
               "title": "Name",
               "type": "string"
            },
            "data_files": {
               "title": "Data Files",
               "type": "array",
               "items": {
                  "type": "string"
               }
            },
            "chan_type": {
               "title": "Chan Type",
               "type": "string"
            },
            "chan_source": {
               "title": "Chan Source",
               "type": "string"
            },
            "sensor": {
               "title": "Sensor",
               "default": "",
               "type": "string"
            },
            "serial": {
               "title": "Serial",
               "default": "",
               "type": "string"
            },
            "gain1": {
               "title": "Gain1",
               "default": 1,
               "type": "number"
            },
            "gain2": {
               "title": "Gain2",
               "default": 1,
               "type": "number"
            },
            "scaling": {
               "title": "Scaling",
               "default": 1,
               "type": "number"
            },
            "chopper": {
               "title": "Chopper",
               "default": false,
               "type": "boolean"
            },
            "dipole_dist": {
               "title": "Dipole Dist",
               "default": 1,
               "type": "number"
            },
            "sensor_calibration_file": {
               "title": "Sensor Calibration File",
               "default": "",
               "type": "string"
            },
            "instrument_calibration_file": {
               "title": "Instrument Calibration File",
               "default": "",
               "type": "string"
            }
         },
         "required": [
            "name"
         ]
      },
      "Record": {
         "title": "Record",
         "description": "Class to hold a record\n\nA record holds information about a process that was run. It is intended to\ntrack processes applied to data, allowing a process history to be saved\nalong with any datasets.\n\nExamples\n--------\nA simple example of creating a process record\n\n>>> from resistics.common import Record\n>>> messages = [\"message 1\", \"message 2\"]\n>>> record = Record(\n...     creator={\"name\": \"example\", \"parameter1\": 15},\n...     messages=messages,\n...     record_type=\"example\"\n... )\n>>> record.summary()\n{\n    'time_local': '...',\n    'time_utc': '...',\n    'creator': {'name': 'example', 'parameter1': 15},\n    'messages': ['message 1', 'message 2'],\n    'record_type': 'example'\n}",
         "type": "object",
         "properties": {
            "time_local": {
               "title": "Time Local",
               "type": "string",
               "format": "date-time"
            },
            "time_utc": {
               "title": "Time Utc",
               "type": "string",
               "format": "date-time"
            },
            "creator": {
               "title": "Creator",
               "type": "object"
            },
            "messages": {
               "title": "Messages",
               "type": "array",
               "items": {
                  "type": "string"
               }
            },
            "record_type": {
               "title": "Record Type",
               "type": "string"
            }
         },
         "required": [
            "creator",
            "messages",
            "record_type"
         ]
      },
      "History": {
         "title": "History",
         "description": "Class for storing processing history\n\nParameters\n----------\nrecords : List[Record], optional\n    List of records, by default []\n\nExamples\n--------\n>>> from resistics.testing import record_example1, record_example2\n>>> from resistics.common import History\n>>> record1 = record_example1()\n>>> record2 = record_example2()\n>>> history = History(records=[record1, record2])\n>>> history.summary()\n{\n    'records': [\n        {\n            'time_local': '...',\n            'time_utc': '...',\n            'creator': {\n                'name': 'example1',\n                'a': 5,\n                'b': -7.0\n            },\n            'messages': ['Message 1', 'Message 2'],\n            'record_type': 'process'\n        },\n        {\n            'time_local': '...',\n            'time_utc': '...',\n            'creator': {\n                'name': 'example2',\n                'a': 'parzen',\n                'b': -21\n            },\n            'messages': ['Message 5', 'Message 6'],\n            'record_type': 'process'\n        }\n    ]\n}",
         "type": "object",
         "properties": {
            "records": {
               "title": "Records",
               "default": [],
               "type": "array",
               "items": {
                  "$ref": "#/definitions/Record"
               }
            }
         }
      }
   }
}

field xtr_file: str = PydanticUndefined¶: The XTR metadata file

field data_byte_start: int = PydanticUndefined¶: The byte offset from beginning of the file to the start of the data

field rec_chans: int = PydanticUndefined¶: The recorded channels

pydantic model resistics_readers.spam.spam.TimeReaderRAW[source]¶

Bases: resistics.time.TimeReader

Parent reader for reading from .RAW SPAM files. Associated metadata can come in XTR or XTRX (an XML style) format. Methods for reading from specific metadata file formats should be added in child classes inheriting from this one.

The raw data for SPAM is sensor Voltage in single precision float. However, if there are multiple data files for a single continuous dataset, each one may have a different gain. Therefore, a scaling has to be calculated for each data file and channel. Applying these scalings will convert all channels to mV.

More information about scalings can be found in readers for the various metadata types, where the scalars are calculated. This class simply implements the reading of data and not the calculation of the scalars.

Show JSON schema

{
   "title": "TimeReaderRAW",
   "description": "Parent reader for reading from .RAW SPAM files. Associated metadata can come\nin XTR or XTRX (an XML style) format. Methods for reading from specific\nmetadata file formats should be added in child classes inheriting from this\none.\n\nThe raw data for SPAM is sensor Voltage in single precision float. However,\nif there are multiple data files for a single continuous dataset, each one\nmay have a different gain. Therefore, a scaling has to be calculated for\neach data file and channel. Applying these scalings will convert all\nchannels to mV.\n\nMore information about scalings can be found in readers for the various\nmetadata types, where the scalars are calculated. This class simply\nimplements the reading of data and not the calculation of the scalars.",
   "type": "object",
   "properties": {
      "name": {
         "title": "Name",
         "type": "string"
      },
      "apply_scalings": {
         "title": "Apply Scalings",
         "default": true,
         "type": "boolean"
      },
      "extension": {
         "title": "Extension",
         "default": ".RAW",
         "type": "string"
      }
   }
}

field extension: Optional[str] = '.RAW'¶

read_data(dir_path: pathlib.Path, metadata: resistics.time.TimeMetadata, read_from: int, read_to: int) → resistics.time.TimeData[source]¶

Get data from data file, returned in mV

Calling this applies scalings calculated when the metadata are read. When a recording consists of multiple data files, each channel of each data file might have a different scaling, therefore, gain removals and other RAW file unique scalings need to be applied before the data is stitched together.

This method returns the data in mV for all channels.

Parameters

dir_path (path) – The directory path to read from
metadata (TimeMetadata) – Time series data metadata
read_from (int) – Sample to read data from
read_to (int) – Sample to read data to

Returns

Time data object

Return type

TimeData

scale_data(time_data: resistics.time.TimeData) → resistics.time.TimeData[source]¶

Scale data to physically meaningful units

Resistics uses field units, meaning physical samples will return the following:

Electrical channels in mV/km
Magnetic channels in mV or nT depending on the sensor
To convert magnetic in mV to nT, calibration is required

Notes

Conversion to mV (gain removal) is performed on read as each RAW file in a dataset can have a separate scalar. Because gain is removed when reading the data and all channel data is in mV, the only calculation that has to be done is to divide by the dipole lengths (east-west spacing and north-south spacing).

Parameters: time_data (TimeData) – TimeData read in from files
Returns: Scaled TimeData
Return type: TimeData

field apply_scalings: bool = True¶

pydantic model resistics_readers.spam.spam.TimeReaderXTR[source]¶

Bases: resistics_readers.spam.spam.TimeReaderRAW

Data reader for SPAM RAW data with XTR metadata

Show JSON schema

{
   "title": "TimeReaderXTR",
   "description": "Data reader for SPAM RAW data with XTR metadata",
   "type": "object",
   "properties": {
      "name": {
         "title": "Name",
         "type": "string"
      },
      "apply_scalings": {
         "title": "Apply Scalings",
         "default": true,
         "type": "boolean"
      },
      "extension": {
         "title": "Extension",
         "default": ".RAW",
         "type": "string"
      }
   }
}

read_metadata(dir_path: pathlib.Path) → resistics_readers.multifile.TimeMetadataMerge [source]¶

Read and merge XTR metadata files

For SPAM data, there may be more than one XTR metadata file as data can be split up into smaller files as it is recorded and each data file will have an associated XTR metadata file. In this case, the following steps are taken:

Read all XTR files
Generate a table with metadata about each data file (times, scalings)
Merge metadata for all the data files

Parameters

dir_path (Path) – Directory path with SPAM data

Returns

Merged TimeMetadataXTR

Return type

TimeMetadataMerge

Raises

TimeDataReadError – If not all data files exist
TimeDataReadError – If the extension of the data files is incorrect

field apply_scalings: bool = True¶

field extension: Optional[str] = '.RAW'¶

field name: Optional[str] = PydanticUndefined¶

Validated by

validate_name