resistics_readers.phoenix.mtu5 module¶
Data readers for Phoenix time series data
Phoenix data has multiple concurrent sampling frequencies per measurement. This does not fit the resistics model of data very well.
There is usually one continuous recording sampling frequency and two other discontinuous recording sampling frequencies. Sampling frequencies are determined by the TS file and are usually as below (though they could change).
.TS2: 24000 Hz
.TS3: 2400 Hz
.TS4: 150 Hz
.TS5: 15 Hz
It is usally the highest .TS file that is the continuous recording.
Note
The scaling has not been thoroughly tested with multiple different Phoenix instruments. Where in doubt, it is suggested to check the values from this package versus those from Phoenix software. As more people use the package, confidence will grow and bug fixes will take place as required.
- pydantic model resistics_readers.phoenix.mtu5.TimeMetadataTS[source]¶
Bases:
resistics.time.TimeMetadataTimeMetadata for a single TS file
Show JSON schema
{ "title": "TimeMetadataTS", "description": "TimeMetadata for a single TS file", "type": "object", "properties": { "file_info": { "$ref": "#/definitions/ResisticsFile" }, "fs": { "title": "Fs", "type": "number" }, "chans": { "title": "Chans", "type": "array", "items": { "type": "string" } }, "n_chans": { "title": "N Chans", "type": "integer" }, "n_samples": { "title": "N Samples", "type": "integer" }, "first_time": { "title": "First Time", "pattern": "%Y-%m-%d %H:%M:%S.%f_%o_%q_%v", "examples": [ "2021-01-01 00:00:00.000061_035156_250000_000000" ] }, "last_time": { "title": "Last Time", "pattern": "%Y-%m-%d %H:%M:%S.%f_%o_%q_%v", "examples": [ "2021-01-01 00:00:00.000061_035156_250000_000000" ] }, "system": { "title": "System", "default": "", "type": "string" }, "serial": { "title": "Serial", "default": "", "type": "string" }, "wgs84_latitude": { "title": "Wgs84 Latitude", "default": -999.0, "type": "number" }, "wgs84_longitude": { "title": "Wgs84 Longitude", "default": -999.0, "type": "number" }, "easting": { "title": "Easting", "default": -999.0, "type": "number" }, "northing": { "title": "Northing", "default": -999.0, "type": "number" }, "elevation": { "title": "Elevation", "default": -999.0, "type": "number" }, "chans_metadata": { "title": "Chans Metadata", "type": "object", "additionalProperties": { "$ref": "#/definitions/ChanMetadata" } }, "history": { "title": "History", "default": { "records": [] }, "allOf": [ { "$ref": "#/definitions/History" } ] }, "data_table": { "title": "Data Table" } }, "required": [ "fs", "chans", "n_samples", "first_time", "last_time", "chans_metadata" ], "definitions": { "ResisticsFile": { "title": "ResisticsFile", "description": "Required information for writing out a resistics file", "type": "object", "properties": { "created_on_local": { "title": "Created On Local", "type": "string", "format": "date-time" }, "created_on_utc": { "title": "Created On Utc", "type": "string", "format": "date-time" }, "version": { "title": "Version", "type": "string" } } }, "ChanMetadata": { "title": "ChanMetadata", "description": "Channel metadata", "type": "object", "properties": { "name": { "title": "Name", "type": "string" }, "data_files": { "title": "Data Files", "type": "array", "items": { "type": "string" } }, "chan_type": { "title": "Chan Type", "type": "string" }, "chan_source": { "title": "Chan Source", "type": "string" }, "sensor": { "title": "Sensor", "default": "", "type": "string" }, "serial": { "title": "Serial", "default": "", "type": "string" }, "gain1": { "title": "Gain1", "default": 1, "type": "number" }, "gain2": { "title": "Gain2", "default": 1, "type": "number" }, "scaling": { "title": "Scaling", "default": 1, "type": "number" }, "chopper": { "title": "Chopper", "default": false, "type": "boolean" }, "dipole_dist": { "title": "Dipole Dist", "default": 1, "type": "number" }, "sensor_calibration_file": { "title": "Sensor Calibration File", "default": "", "type": "string" }, "instrument_calibration_file": { "title": "Instrument Calibration File", "default": "", "type": "string" } }, "required": [ "name" ] }, "Record": { "title": "Record", "description": "Class to hold a record\n\nA record holds information about a process that was run. It is intended to\ntrack processes applied to data, allowing a process history to be saved\nalong with any datasets.\n\nExamples\n--------\nA simple example of creating a process record\n\n>>> from resistics.common import Record\n>>> messages = [\"message 1\", \"message 2\"]\n>>> record = Record(\n... creator={\"name\": \"example\", \"parameter1\": 15},\n... messages=messages,\n... record_type=\"example\"\n... )\n>>> record.summary()\n{\n 'time_local': '...',\n 'time_utc': '...',\n 'creator': {'name': 'example', 'parameter1': 15},\n 'messages': ['message 1', 'message 2'],\n 'record_type': 'example'\n}", "type": "object", "properties": { "time_local": { "title": "Time Local", "type": "string", "format": "date-time" }, "time_utc": { "title": "Time Utc", "type": "string", "format": "date-time" }, "creator": { "title": "Creator", "type": "object" }, "messages": { "title": "Messages", "type": "array", "items": { "type": "string" } }, "record_type": { "title": "Record Type", "type": "string" } }, "required": [ "creator", "messages", "record_type" ] }, "History": { "title": "History", "description": "Class for storing processing history\n\nParameters\n----------\nrecords : List[Record], optional\n List of records, by default []\n\nExamples\n--------\n>>> from resistics.testing import record_example1, record_example2\n>>> from resistics.common import History\n>>> record1 = record_example1()\n>>> record2 = record_example2()\n>>> history = History(records=[record1, record2])\n>>> history.summary()\n{\n 'records': [\n {\n 'time_local': '...',\n 'time_utc': '...',\n 'creator': {\n 'name': 'example1',\n 'a': 5,\n 'b': -7.0\n },\n 'messages': ['Message 1', 'Message 2'],\n 'record_type': 'process'\n },\n {\n 'time_local': '...',\n 'time_utc': '...',\n 'creator': {\n 'name': 'example2',\n 'a': 'parzen',\n 'b': -21\n },\n 'messages': ['Message 5', 'Message 6'],\n 'record_type': 'process'\n }\n ]\n}", "type": "object", "properties": { "records": { "title": "Records", "default": [], "type": "array", "items": { "$ref": "#/definitions/Record" } } } } } }
- field data_table: Any = PydanticUndefined¶
The record sample ranges
- pydantic model resistics_readers.phoenix.mtu5.TimeMetadataPhoenix[source]¶
Bases:
resistics.common.MetadataTimeMetadata for the Phoenix data
Show JSON schema
{ "title": "TimeMetadataPhoenix", "description": "TimeMetadata for the Phoenix data", "type": "object", "properties": { "ts_nums": { "title": "Ts Nums", "type": "array", "items": { "type": "integer" } }, "ts_files": { "title": "Ts Files", "type": "object", "additionalProperties": { "type": "string" } }, "ts_continuous": { "title": "Ts Continuous", "type": "integer" }, "ts_metadata": { "title": "Ts Metadata", "type": "object", "additionalProperties": { "$ref": "#/definitions/TimeMetadataTS" } } }, "required": [ "ts_nums", "ts_files", "ts_continuous", "ts_metadata" ], "definitions": { "ResisticsFile": { "title": "ResisticsFile", "description": "Required information for writing out a resistics file", "type": "object", "properties": { "created_on_local": { "title": "Created On Local", "type": "string", "format": "date-time" }, "created_on_utc": { "title": "Created On Utc", "type": "string", "format": "date-time" }, "version": { "title": "Version", "type": "string" } } }, "ChanMetadata": { "title": "ChanMetadata", "description": "Channel metadata", "type": "object", "properties": { "name": { "title": "Name", "type": "string" }, "data_files": { "title": "Data Files", "type": "array", "items": { "type": "string" } }, "chan_type": { "title": "Chan Type", "type": "string" }, "chan_source": { "title": "Chan Source", "type": "string" }, "sensor": { "title": "Sensor", "default": "", "type": "string" }, "serial": { "title": "Serial", "default": "", "type": "string" }, "gain1": { "title": "Gain1", "default": 1, "type": "number" }, "gain2": { "title": "Gain2", "default": 1, "type": "number" }, "scaling": { "title": "Scaling", "default": 1, "type": "number" }, "chopper": { "title": "Chopper", "default": false, "type": "boolean" }, "dipole_dist": { "title": "Dipole Dist", "default": 1, "type": "number" }, "sensor_calibration_file": { "title": "Sensor Calibration File", "default": "", "type": "string" }, "instrument_calibration_file": { "title": "Instrument Calibration File", "default": "", "type": "string" } }, "required": [ "name" ] }, "Record": { "title": "Record", "description": "Class to hold a record\n\nA record holds information about a process that was run. It is intended to\ntrack processes applied to data, allowing a process history to be saved\nalong with any datasets.\n\nExamples\n--------\nA simple example of creating a process record\n\n>>> from resistics.common import Record\n>>> messages = [\"message 1\", \"message 2\"]\n>>> record = Record(\n... creator={\"name\": \"example\", \"parameter1\": 15},\n... messages=messages,\n... record_type=\"example\"\n... )\n>>> record.summary()\n{\n 'time_local': '...',\n 'time_utc': '...',\n 'creator': {'name': 'example', 'parameter1': 15},\n 'messages': ['message 1', 'message 2'],\n 'record_type': 'example'\n}", "type": "object", "properties": { "time_local": { "title": "Time Local", "type": "string", "format": "date-time" }, "time_utc": { "title": "Time Utc", "type": "string", "format": "date-time" }, "creator": { "title": "Creator", "type": "object" }, "messages": { "title": "Messages", "type": "array", "items": { "type": "string" } }, "record_type": { "title": "Record Type", "type": "string" } }, "required": [ "creator", "messages", "record_type" ] }, "History": { "title": "History", "description": "Class for storing processing history\n\nParameters\n----------\nrecords : List[Record], optional\n List of records, by default []\n\nExamples\n--------\n>>> from resistics.testing import record_example1, record_example2\n>>> from resistics.common import History\n>>> record1 = record_example1()\n>>> record2 = record_example2()\n>>> history = History(records=[record1, record2])\n>>> history.summary()\n{\n 'records': [\n {\n 'time_local': '...',\n 'time_utc': '...',\n 'creator': {\n 'name': 'example1',\n 'a': 5,\n 'b': -7.0\n },\n 'messages': ['Message 1', 'Message 2'],\n 'record_type': 'process'\n },\n {\n 'time_local': '...',\n 'time_utc': '...',\n 'creator': {\n 'name': 'example2',\n 'a': 'parzen',\n 'b': -21\n },\n 'messages': ['Message 5', 'Message 6'],\n 'record_type': 'process'\n }\n ]\n}", "type": "object", "properties": { "records": { "title": "Records", "default": [], "type": "array", "items": { "$ref": "#/definitions/Record" } } } }, "TimeMetadataTS": { "title": "TimeMetadataTS", "description": "TimeMetadata for a single TS file", "type": "object", "properties": { "file_info": { "$ref": "#/definitions/ResisticsFile" }, "fs": { "title": "Fs", "type": "number" }, "chans": { "title": "Chans", "type": "array", "items": { "type": "string" } }, "n_chans": { "title": "N Chans", "type": "integer" }, "n_samples": { "title": "N Samples", "type": "integer" }, "first_time": { "title": "First Time", "pattern": "%Y-%m-%d %H:%M:%S.%f_%o_%q_%v", "examples": [ "2021-01-01 00:00:00.000061_035156_250000_000000" ] }, "last_time": { "title": "Last Time", "pattern": "%Y-%m-%d %H:%M:%S.%f_%o_%q_%v", "examples": [ "2021-01-01 00:00:00.000061_035156_250000_000000" ] }, "system": { "title": "System", "default": "", "type": "string" }, "serial": { "title": "Serial", "default": "", "type": "string" }, "wgs84_latitude": { "title": "Wgs84 Latitude", "default": -999.0, "type": "number" }, "wgs84_longitude": { "title": "Wgs84 Longitude", "default": -999.0, "type": "number" }, "easting": { "title": "Easting", "default": -999.0, "type": "number" }, "northing": { "title": "Northing", "default": -999.0, "type": "number" }, "elevation": { "title": "Elevation", "default": -999.0, "type": "number" }, "chans_metadata": { "title": "Chans Metadata", "type": "object", "additionalProperties": { "$ref": "#/definitions/ChanMetadata" } }, "history": { "title": "History", "default": { "records": [] }, "allOf": [ { "$ref": "#/definitions/History" } ] }, "data_table": { "title": "Data Table" } }, "required": [ "fs", "chans", "n_samples", "first_time", "last_time", "chans_metadata" ] } } }
- field ts_nums: List[int] = PydanticUndefined¶
The TS file numbers
- field ts_files: Dict[int, str] = PydanticUndefined¶
The name of the TS files
- field ts_continuous: int = PydanticUndefined¶
The continuous TS
- field ts_metadata: Dict[int, resistics_readers.phoenix.mtu5.TimeMetadataTS] = PydanticUndefined¶
Metadata for a single TS
- resistics_readers.phoenix.mtu5.strip_control(in_bytes: bytes) → str[source]¶
Strip control characters from byte string
- resistics_readers.phoenix.mtu5.read_table_entry(entry_bytes: bytes) → Tuple[str, Any][source]¶
Read a single table entry, this should return an entry name and value
- Parameters
entry_bytes (bytes) – The entry bytes
- Returns
The name and value
- Return type
Tuple[str, Any]
- Raises
ValueError – If unable to read entry name
KeyError – If entry name is unknown
TypeError – If unable to read entry value
- resistics_readers.phoenix.mtu5.read_table_file(table_path: pathlib.Path) → OrderedDict[str, Any][source]¶
Read a TBL file
- Parameters
table_path (Path) – Path to the table file
- Returns
The table data in an Ordered dictionary
- Return type
Dict[str, Any]
- resistics_readers.phoenix.mtu5.get_date(value: bytes) → attotime.objects.attodatetime.attodatetime[source]¶
Convert bytes to a resistics DateTime
- Parameters
value (bytes) – The bytes with the datetime information
- Returns
The datetime
- Return type
RSDateTime
- resistics_readers.phoenix.mtu5.read_tag(f: BinaryIO) → Dict[str, Any][source]¶
Read the tag from a .TS data file
Tags are used to separate records in a data file. Each tag is 32 bytes long and contains data about the next data record
Some notes about particular tag entries
units of sample rate: 0 = Hz, 1 = minute, 2 = hour, 3 = day
bit-wise saturation flags
clock error in micro seconds
- Parameters
f (BinaryIO) – Binary file type object
- Returns
The tag data
- Return type
Dict[str, Any]
- resistics_readers.phoenix.mtu5.get_records(dir_path: pathlib.Path, ts_file: str) → pandas.core.frame.DataFrame[source]¶
Get details for all the records
Phoenix MTU5C data files have multiple records separated by tags. Each record will have a number of scans. A single scan is all the channel data for one timestamp. The number of scans in a record is equal to the number of samples in the record.
Usually, a record will be a second long, so the number of scans in the record is determined by the sampling frequency.
When reading data, it commonly needs to be read from multiple scans, therefore this table helps find which records need to be read to get the data to cover a particular time range.
Note that the time given in the tag is the start time of the next record.
- Parameters
dir_path (Path) – The path with the data file
ts_file (str) – The name of the data file
- Returns
A DataFrame with details about each record
- Return type
pd.DataFrame
- resistics_readers.phoenix.mtu5.get_time_dict(ts: int, table_data: OrderedDict[str, Any], record_df: pandas.core.frame.DataFrame) → Dict[str, Any][source]¶
Get the time dictionary for a single TS that will be used to initialise a TimeMetadata instance.
Start and end times are provided in the metadata. However, it is only for the continuous sampling frequency that the data is continuous between these dates. For the other frequencies, whilst these may be the timestamps of the first and last sample, it is not necessary that all the timestamps in between are present.
- Parameters
ts (int) – The TS number
table_data (OrderedDict[str, Any]) – The data read in from the .TBL file
record_df (pd.DataFrame) – Information about the records. This will be primarily be used to get the total number of samples in the recording and the last time.
- Returns
A dictionary of data about the recording
- Return type
Dict[str, Any]
- resistics_readers.phoenix.mtu5.get_chans_metadata(table_data: OrderedDict[str, Any], ts_file: str) → Dict[str, Any][source]¶
Get ChanMetadata for each channel
- Parameters
table_data (OrderedDict[str, Any]) – The table data
ts_file (str) – The TS file name
- Returns
Channel metadata for each channel
- Return type
Dict[str, Any]
- resistics_readers.phoenix.mtu5.get_ts_metadata(dir_path: pathlib.Path, ts_file: str, ts: int, table_data: Dict[str, Any]) → resistics.time.TimeMetadata[source]¶
Get TimeMetadata for a single .TS file
- Parameters
dir_path (Path) – The directory path with the data file
ts_file (str) – The name of the data file
ts (int) – The TS number
table_data (Dict[str, Any]) – The table data
- Returns
TimeMetadata
- Return type
TimeMetadata
- resistics_readers.phoenix.mtu5.read_metadata(dir_path: pathlib.Path) → resistics_readers.phoenix.mtu5.TimeMetadataPhoenix[source]¶
Read metadata for Phoenix data
For phoenix data, the metadata is in the table file and it is binary formatted.
- Parameters
dir_path (Path) – The directory path with the data
- Returns
TimeMetadataPhoenix which has a dictionary of TimeMetadata for each .TS file
- Return type
TimeMetadataPhoenix
- Raises
MetadataReadError – If the number of .TBL files in the directory != 1
- resistics_readers.phoenix.mtu5.read_record(data_bytes: bytes, n_chans: int, from_sample: int, to_sample: int) → numpy.ndarray[source]¶
Read a record
- Parameters
data_bytes (bytes) – The bytes in the record
n_chans (int) – The number of channels
from_sample (int) – The first sample to read in the record
to_sample (int) – The last sample to read in the record
- Returns
Data with shape n_chans x n_samples
- Return type
np.ndarray
- resistics_readers.phoenix.mtu5.read_records(data_path: pathlib.Path, metadata: resistics_readers.phoenix.mtu5.TimeMetadataTS, df_to_read: pandas.core.frame.DataFrame) → numpy.ndarray[source]¶
Read data records from a TS file
- Parameters
data_path (Path) – The path to the TS file
metadata (TimeMetadataTS) – The metadata for the TS file
df_to_read (pd.DataFrame) – The DataFrame with the records to read
- Returns
The read data
- Return type
np.ndarray
- pydantic model resistics_readers.phoenix.mtu5.TimeReaderTS[source]¶
Bases:
resistics.time.TimeReaderPhoenix time data reader only for the continuous time series data
There is no data reader for the other TS files, these should be reformatted.
Show JSON schema
{ "title": "TimeReaderTS", "description": "Phoenix time data reader only for the continuous time series data\n\nThere is no data reader for the other TS files, these should be reformatted.", "type": "object", "properties": { "name": { "title": "Name", "type": "string" }, "apply_scalings": { "title": "Apply Scalings", "default": true, "type": "boolean" }, "extension": { "title": "Extension", "default": ".TS", "type": "string" } } }
- field extension: Optional[str] = '.TS'¶
- read_metadata(dir_path: pathlib.Path) → resistics_readers.phoenix.mtu5.TimeMetadataTS[source]¶
Read the metadata for the continuous data
- Parameters
dir_path (Path) – The directory path to the data
- Returns
Metadata for the continuous TS file
- Return type
TimeMetadataTS
- Raises
TimeDataReadError – If the data files do not exist
- field apply_scalings: bool = True¶
- read_data(dir_path: pathlib.Path, metadata: resistics.time.TimeMetadata, read_from: int, read_to: int) → resistics.time.TimeData[source]¶
Read data from the continuous time series data
In a TS file, each sample is recorded as a scan (all channels recorded at the same time). To get the number of bytes to read, multiply number of samples by number of channels by the number of bytes for a single sample
- Parameters
dir_path (Path) – The directory path
metadata (TimeMetadata) – The phoenix data metadata
read_from (int) – Sample to read from
read_to (int) – Sample to read to
- Returns
The read in time data
- Return type
TimeData
- scale_data(time_data: resistics.time.TimeData) → resistics.time.TimeData[source]¶
Get data scaled to physical values
This information comes from:
Instrument and Sensor Calibration: Concepts and Utility Programs
And applies to:
V5 System 2000, System2000.net, and MTU-net (MTU, MTU-A, V8, RXU, MTU-net; MTC and AMTC coils and AL-100 loop).
The important factors for scaling the data and defined in the TBL file
FSCV: the full scale value in volts [V8, RXU, MTU-A, 2.45V] [MTU, 6.40V]; full scale is 2^23 or 8,388,608 du
ExLN: the length of the N–S dipole (Ex) in metres
EyLN: the length of the E–W dipole (Ey) in metres
EGN: the gain used for the E channels [MTU-A x1 x4 x16] [MTU x10 x40 x160]
HGN: the gain used for the H channels [MTU-A x1 x4 x16] [MTU x3 x12 x48]
HNUM: the scale factor for coil sensors in (mV/nT) [AMTC-30 100] [MTC-50 1000]
HATT: interconect board factor [MTU, MTU-A, MTU-net 0.233] [V8, RXU 1]
du refers to the digital unit or the value out of the machine
Note that HNUM is only applicable to AMTC-30 and MTC-50. For other systems, HNUM does not appear in the table file.
These are read in and kept in the metadata
FSCV / 2^23 is in the scaling key for each channel
ExLn and EyLn are in their appropriate channel in metres
EGN is in gain1 for electric channels
HGN * HATT is in gain1 for magnetic channels
If HNUM is in the table file, gain2 is set to (1000/HNUM) for magnetic channels. If HNUM is not present, gain2 is set to 1 for magnetic channels.
To scale the electric channels, apply the following:
E-Channel (mV/km) = du * (FSCV/2^23) * (1/EGN) * (1/E_LN) * (1000*1000)
Units derivation: (mV/km) = integer * (V/integer) * real * (1/m) * (mV/V * m/km)
Given the metadata, this becomes
\[ \begin{align}\begin{aligned}Ex = Ex * scaling * (1/gain1) * (1/dx) * (1000*1000)\\Ey = Ey * scaling * (1/gain1) * (1/dy) * (1000*1000)\end{aligned}\end{align} \]For the magnetic channels:
H-Channel (nT) = du * (FSCV/2^23) * (1/HGN) * (1/HATT) * (1000/HNUM)
Units derivation: (nT) = integer * (V/integer) * real* real * (mV/V / mV/nT)
With the metadata, this is:
\[ \begin{align}\begin{aligned}Hx = Hx * scaling * (1/gain1) * gain2\\Hy = Hz * scaling * (1/gain1) * gain2\\Hy = Hz * scaling * (1/gain1) * gain2\end{aligned}\end{align} \]- Parameters
time_data (TimeData) – Input time data
- Returns
Time data in field units
- Return type
TimeData
- resistics_readers.phoenix.mtu5.read_discontinuous_data(dir_path: pathlib.Path, ts_num: int, metadata: resistics_readers.phoenix.mtu5.TimeMetadataTS) → resistics.time.TimeData[source]¶
Read data from a discontinuous TS file
Note that all the gaps are lost and the data is essentially considered single continuous data beginning at the first record.
- Parameters
dir_path (Path) – The directory path to read from
ts_num (int) – The TS file to read
metadata (TimeMetadataTS) – The TimeMetadataTS for the TS file with details about all records
- Returns
The read time data
- Return type
TimeData
- Raises
TimeDataReadError – If an incorrect number of files are found for the TS number
- resistics_readers.phoenix.mtu5.reformat(dir_path: pathlib.Path, metadata: resistics_readers.phoenix.mtu5.TimeMetadataPhoenix, ts_num: int, write_path: pathlib.Path, processors: Optional[List[resistics.time.TimeProcess]] = None) → None[source]¶
Reformat a discountinuous TS file