resistics_readers.lemi.b423 module¶

Module for Lemi B423 data Lemi B423 always records channels in the following order

Hx, Hy, Hz, Ex, Ey

The Lemi B423 binary files are constructed from a 1024 text header followed by repeating records with the following definitions (PPS is short for deviation from PPS and PLL is short for PLL accuracy):

SECOND_TIMESTAMP, SAMPLE_NUM [0-FS], HX, HY, HZ, EX, EY, PPS, PLL

These are interpreted to have byte types

L, H, l, l, l, l, l, h, h

pydantic model resistics_readers.lemi.b423.TimeMetadataB423[source]¶

Bases: resistics_readers.multifile.TimeMetadataSingle

This is an extension of TimeMetadataSingle for a single B423 file

Show JSON schema

{
   "title": "TimeMetadataB423",
   "description": "This is an extension of TimeMetadataSingle for a single B423 file",
   "type": "object",
   "properties": {
      "file_info": {
         "$ref": "#/definitions/ResisticsFile"
      },
      "fs": {
         "title": "Fs",
         "type": "number"
      },
      "chans": {
         "title": "Chans",
         "type": "array",
         "items": {
            "type": "string"
         }
      },
      "n_chans": {
         "title": "N Chans",
         "type": "integer"
      },
      "n_samples": {
         "title": "N Samples",
         "type": "integer"
      },
      "first_time": {
         "title": "First Time",
         "pattern": "%Y-%m-%d %H:%M:%S.%f_%o_%q_%v",
         "examples": [
            "2021-01-01 00:00:00.000061_035156_250000_000000"
         ]
      },
      "last_time": {
         "title": "Last Time",
         "pattern": "%Y-%m-%d %H:%M:%S.%f_%o_%q_%v",
         "examples": [
            "2021-01-01 00:00:00.000061_035156_250000_000000"
         ]
      },
      "system": {
         "title": "System",
         "default": "",
         "type": "string"
      },
      "serial": {
         "title": "Serial",
         "default": "",
         "type": "string"
      },
      "wgs84_latitude": {
         "title": "Wgs84 Latitude",
         "default": -999.0,
         "type": "number"
      },
      "wgs84_longitude": {
         "title": "Wgs84 Longitude",
         "default": -999.0,
         "type": "number"
      },
      "easting": {
         "title": "Easting",
         "default": -999.0,
         "type": "number"
      },
      "northing": {
         "title": "Northing",
         "default": -999.0,
         "type": "number"
      },
      "elevation": {
         "title": "Elevation",
         "default": -999.0,
         "type": "number"
      },
      "chans_metadata": {
         "title": "Chans Metadata",
         "type": "object",
         "additionalProperties": {
            "$ref": "#/definitions/ChanMetadata"
         }
      },
      "history": {
         "title": "History",
         "default": {
            "records": []
         },
         "allOf": [
            {
               "$ref": "#/definitions/History"
            }
         ]
      },
      "data_file": {
         "title": "Data File",
         "type": "string"
      },
      "data_byte_start": {
         "title": "Data Byte Start",
         "type": "integer"
      },
      "scalings": {
         "title": "Scalings",
         "type": "object",
         "additionalProperties": {
            "type": "number"
         }
      }
   },
   "required": [
      "fs",
      "chans",
      "n_samples",
      "first_time",
      "last_time",
      "chans_metadata",
      "data_file",
      "data_byte_start",
      "scalings"
   ],
   "definitions": {
      "ResisticsFile": {
         "title": "ResisticsFile",
         "description": "Required information for writing out a resistics file",
         "type": "object",
         "properties": {
            "created_on_local": {
               "title": "Created On Local",
               "type": "string",
               "format": "date-time"
            },
            "created_on_utc": {
               "title": "Created On Utc",
               "type": "string",
               "format": "date-time"
            },
            "version": {
               "title": "Version",
               "type": "string"
            }
         }
      },
      "ChanMetadata": {
         "title": "ChanMetadata",
         "description": "Channel metadata",
         "type": "object",
         "properties": {
            "name": {
               "title": "Name",
               "type": "string"
            },
            "data_files": {
               "title": "Data Files",
               "type": "array",
               "items": {
                  "type": "string"
               }
            },
            "chan_type": {
               "title": "Chan Type",
               "type": "string"
            },
            "chan_source": {
               "title": "Chan Source",
               "type": "string"
            },
            "sensor": {
               "title": "Sensor",
               "default": "",
               "type": "string"
            },
            "serial": {
               "title": "Serial",
               "default": "",
               "type": "string"
            },
            "gain1": {
               "title": "Gain1",
               "default": 1,
               "type": "number"
            },
            "gain2": {
               "title": "Gain2",
               "default": 1,
               "type": "number"
            },
            "scaling": {
               "title": "Scaling",
               "default": 1,
               "type": "number"
            },
            "chopper": {
               "title": "Chopper",
               "default": false,
               "type": "boolean"
            },
            "dipole_dist": {
               "title": "Dipole Dist",
               "default": 1,
               "type": "number"
            },
            "sensor_calibration_file": {
               "title": "Sensor Calibration File",
               "default": "",
               "type": "string"
            },
            "instrument_calibration_file": {
               "title": "Instrument Calibration File",
               "default": "",
               "type": "string"
            }
         },
         "required": [
            "name"
         ]
      },
      "Record": {
         "title": "Record",
         "description": "Class to hold a record\n\nA record holds information about a process that was run. It is intended to\ntrack processes applied to data, allowing a process history to be saved\nalong with any datasets.\n\nExamples\n--------\nA simple example of creating a process record\n\n>>> from resistics.common import Record\n>>> messages = [\"message 1\", \"message 2\"]\n>>> record = Record(\n...     creator={\"name\": \"example\", \"parameter1\": 15},\n...     messages=messages,\n...     record_type=\"example\"\n... )\n>>> record.summary()\n{\n    'time_local': '...',\n    'time_utc': '...',\n    'creator': {'name': 'example', 'parameter1': 15},\n    'messages': ['message 1', 'message 2'],\n    'record_type': 'example'\n}",
         "type": "object",
         "properties": {
            "time_local": {
               "title": "Time Local",
               "type": "string",
               "format": "date-time"
            },
            "time_utc": {
               "title": "Time Utc",
               "type": "string",
               "format": "date-time"
            },
            "creator": {
               "title": "Creator",
               "type": "object"
            },
            "messages": {
               "title": "Messages",
               "type": "array",
               "items": {
                  "type": "string"
               }
            },
            "record_type": {
               "title": "Record Type",
               "type": "string"
            }
         },
         "required": [
            "creator",
            "messages",
            "record_type"
         ]
      },
      "History": {
         "title": "History",
         "description": "Class for storing processing history\n\nParameters\n----------\nrecords : List[Record], optional\n    List of records, by default []\n\nExamples\n--------\n>>> from resistics.testing import record_example1, record_example2\n>>> from resistics.common import History\n>>> record1 = record_example1()\n>>> record2 = record_example2()\n>>> history = History(records=[record1, record2])\n>>> history.summary()\n{\n    'records': [\n        {\n            'time_local': '...',\n            'time_utc': '...',\n            'creator': {\n                'name': 'example1',\n                'a': 5,\n                'b': -7.0\n            },\n            'messages': ['Message 1', 'Message 2'],\n            'record_type': 'process'\n        },\n        {\n            'time_local': '...',\n            'time_utc': '...',\n            'creator': {\n                'name': 'example2',\n                'a': 'parzen',\n                'b': -21\n            },\n            'messages': ['Message 5', 'Message 6'],\n            'record_type': 'process'\n        }\n    ]\n}",
         "type": "object",
         "properties": {
            "records": {
               "title": "Records",
               "default": [],
               "type": "array",
               "items": {
                  "$ref": "#/definitions/Record"
               }
            }
         }
      }
   }
}

field data_byte_start: int = PydanticUndefined¶: The byte offset from the beginning of the file to the start of the data

field scalings: Dict[str, float] = PydanticUndefined¶: Scalings in the B423 ASCII header

resistics_readers.lemi.b423.make_subdir_B423_metadata(dir_path: pathlib.Path, fs: float, hx_serial: int = 0, hy_serial: int = 0, hz_serial: int = 0, h_gain: int = 1, dx: float = 1, dy: float = 1, folders: Optional[List[str]] = None) → None[source]¶

Construct B423 headers for sub directories of a folder

Parameters

dir_path (Path) – The path to the folder
fs (float) – The sampling frequency, Hz
hx_serial (str, optional) – The x direction magnetic serial, used for calibration
hy_serial (str, optional) – The y direction magnetic serial, used for calibration
hz_serial (str, optional) – The z direction magnetic serial, used for calibration
h_gain (int) – Any gain on the magnetic channels which will need to be removed
dx (float, optional) – Distance between x electrodes
dy (float, optional) – Distance between y electrodes
folders (List, optional) – An optional list of subfolders, by default None. If None, all the subfolders will be processed

resistics_readers.lemi.b423.make_B423_metadata(dir_path: pathlib.Path, fs: float, hx_serial: int = 0, hy_serial: int = 0, hz_serial: int = 0, h_gain: int = 1, dx: float = 1, dy: float = 1) → None[source]¶

Read a single B423 measurement directory, construct and write out metadata

Parameters

dir_path (Path) – The path to the measurement
fs (float) – The sampling frequency, Hz
hx_serial (str, optional) – The x direction magnetic serial, used for calibration
hy_serial (str, optional) – The y direction magnetic serial, used for calibration
hz_serial (str, optional) – The z direction magnetic serial, used for calibration
h_gain (int) – Any gain on the magnetic channels which will need to be removed
dx (float, optional) – Distance between x electrodes
dy (float, optional) – Distance between y electrodes

pydantic model resistics_readers.lemi.b423.TimeReaderB423[source]¶

Bases: resistics.time.TimeReader

Data reader for Lemi B423 data

There is no separate metadata file for Lemi B423 data detailing the sampling frequency, the number of samples, the sensors etc.. Unfortunately, such a metadata file is a pre-requisite for resistics. There are helper methods to make one.

In situations where a Lemi B423 dataset is recorded in multiple files, it is required that the recording is continuous.

Other important notes about Lemi B423 files

1024 bytes of ASCII metadata in the data file with scaling information
Lemi B423 raw measurement data is signed long integer format

Important points about scalings

Raw data is integer counts for electric and magnetic channels
Scalings in B423 files convert electric channels to uV (microvolt)
Scalings in B423 files convert magnetic channels to millivolts
Scaling for the magnetic channels in B423 files leaves internal gain on
Internal gain should be specified when creating metadata

If apply_scaling is False, data will be returned in:

microvolts for the electric channels
millivolts for the magnetic with the gain applied

Which is equivalent to applying the scalings in the B423 headers

With apply_scaling True, the following additional scaling will be applied:

Electric channels converted to mV
Dipole length corrections are applied to electric channels
Magnetic channel gains are removed

Note

For more information about Lemi B423 format, please see: http://lemisensors.com/?p=485

Show JSON schema

{
   "title": "TimeReaderB423",
   "description": "Data reader for Lemi B423 data\n\nThere is no separate metadata file for Lemi B423 data detailing the sampling\nfrequency, the number of samples, the sensors etc.. Unfortunately, such a\nmetadata file is a pre-requisite for resistics. There are helper methods to\nmake one.\n\nIn situations where a Lemi B423 dataset is recorded in multiple files, it is\nrequired that the recording is continuous.\n\nOther important notes about Lemi B423 files\n\n- 1024 bytes of ASCII metadata in the data file with scaling information\n- Lemi B423 raw measurement data is signed long integer format\n\nImportant points about scalings\n\n- Raw data is integer counts for electric and magnetic channels\n- Scalings in B423 files convert electric channels to uV (microvolt)\n- Scalings in B423 files convert magnetic channels to millivolts\n- Scaling for the magnetic channels in B423 files leaves internal gain on\n- Internal gain should be specified when creating metadata\n\nIf apply_scaling is False, data will be returned in:\n\n- microvolts for the electric channels\n- millivolts for the magnetic with the gain applied\n\nWhich is equivalent to applying the scalings in the B423 headers\n\nWith apply_scaling True, the following additional scaling will be applied:\n\n- Electric channels converted to mV\n- Dipole length corrections are applied to electric channels\n- Magnetic channel gains are removed\n\n.. note::\n\n    For more information about Lemi B423 format, please see:\n    http://lemisensors.com/?p=485",
   "type": "object",
   "properties": {
      "name": {
         "title": "Name",
         "type": "string"
      },
      "apply_scalings": {
         "title": "Apply Scalings",
         "default": true,
         "type": "boolean"
      },
      "extension": {
         "title": "Extension",
         "default": ".B423",
         "type": "string"
      },
      "record_bytes": {
         "title": "Record Bytes",
         "default": 30,
         "type": "integer"
      }
   }
}

field record_bytes: int = 30¶

read_metadata(dir_path: pathlib.Path) → resistics_readers.multifile.TimeMetadataMerge [source]¶

Read metadata

Parameters

dir_path (Path) – The data directory

Returns

TimeMetadata with a data table

Return type

TimeMetadataMerge

Raises

MetadataReadError – If the channels are not correct for B423
TimeDataReadError – If not all data files exist
TimeDataReadError – If extensions do not match

read_data(dir_path: pathlib.Path, metadata: resistics.time.TimeMetadata, read_from: int, read_to: int) → resistics.time.TimeData[source]¶

Get data from data files

Lemi B423 data always has five channels, in order Hx, Hy, Hz, Ex, Ey. The raw data is integer counts. However, additional scalings from the B423 files are applied to give:

microvolts for the electric channels
millivolts for the magnetic with the gain applied

The scalings are as follows:

Hx = (Hx * Kmx) + Ax
Hx = (Hy * Kmy) + Ay
Hx = (Hz * Kmz) + Az
Ex = (Ex * Ke1) + Ae1
Ey = (Ey * Ke2) + Ae2

Parameters

dir_path (path) – The directory path to read from
metadata (TimeMetadata) – Time series data metadata
read_from (int) – Sample to read data from
read_to (int) – Sample to read data to

Returns

Time data object

Return type

TimeData

scale_data(time_data: resistics.time.TimeData) → resistics.time.TimeData[source]¶

Get data scaled to physical values

resistics uses field units, meaning physical samples will return the following:

Electrical channels in mV/km
Magnetic channels in mV
To get magnetic fields in nT, calibration needs to be performed

Notes

When Lemi data is read in, scaling in the headers is applied. Therefore, the magnetic channels is in mV with gain applied and the electric channels are in uV (microvolts). To complete the scaling to field units, the below additional corrections need to be applied.

Electric channels need to divided by 1000 along with dipole length division in km (east-west spacing and north-south spacing) to return mV/km.

Magnetic channels need to be divided by the internal gain value which should be set in the metadata

Parameters: time_data (TimeData) – Input time data
Returns: Time data in field units
Return type: TimeData