adcpy.EPICstuff package¶
Submodules¶
adcpy.EPICstuff.ADCPcdf2ncEPIC module¶
This code takes a raw netcdf file containing data from any 4 beam Janus acoustic doppler profiler, with or without a center beam, and transforms the data into Earth coordinates. Data are output to netCDF using controlled vocabulary for the variable names, following the EPIC convention wherever possible.
ADCPcdf2ncEPIC.doEPIC_ADCPfile(cdfFile, ncFile, attFile, settings)
cdfFile = path to a USGS raw netCDF ADCP data file
ncFile = a netcdf file structured according to PMEL EPIC conventions
attFile = a file containing global attributes (metadata) for the data. See below
settings = a dictionary of preferences for the processing:
'good_ensembles': [0, np.inf] # starting and ending indices of the input file. For all data use [0,np.inf]
'orientation': 'UP' # uplooking ADCP, for downlooking, use DOWN
'transducer_offset_from_bottom': 1.0 # a float in meters
'transformation': 'EARTH' # | BEAM | INST
'adjust_to_UTC': 5 # for EST to UTC, if no adjustment, set to 0 or omit
Depth dependent attributes are compute from the mean Pressure found in the raw data file. So it is best to have the time series trimmed to the in water time or to provide the good ensemble indices for in water time
Note that file names and paths may not include spaces
Example contents of a Global Attribute file:
SciPi; J.Q. Scientist
PROJECT; USGS Coastal Marine Geology Program
EXPERIMENT; MVCO 2015 Stress Comparison
DESCRIPTION; Quadpod 13.9m
DATA_SUBTYPE; MOORED
COORD_SYSTEM; GEOGRAPHIC + SAMPLE
Conventions; PMEL/EPIC
MOORING; 1057
WATER_DEPTH; 13.9
WATER_DEPTH_NOTE; (meters), nominal
WATER_DEPTH_source; ship fathometer
latitude; 41.3336633
longitude; -70.565877
magnetic_variation; -14.7
Deployment_date; 17-Nov-2015
Recovery_date; 14-Dec-2015
DATA_CMNT;
platform_type; USGS aluminum T14 quadpod
DRIFTER; 0
POS_CONST; 0
DEPTH_CONST; 0
Conventions; PMEL/EPIC
institution; United States Geological Survey, Woods Hole Coastal and Marine Science Center
institution_url; http://woodshole.er.usgs.gov
Created on Tue May 16 13:33:31 2017
@author: mmartini
-
EPICstuff.ADCPcdf2ncEPIC.
add_VAR_DESC
(cdf)¶ add the VAR_DESC global attribute constructed from variable names found in the file
Parameters: cdf (object) – netCDF file object
-
EPICstuff.ADCPcdf2ncEPIC.
beam2inst
(adcpo, reverse=False, force=False)¶ Rotate velocities from beam to instrument coordinates.
Parameters: - adcpo (dict) – containing the beam velocity data.
- reverse (bool) – If True, this function performs the inverse rotation (inst->beam).
- force (bool) – When true do not check which coordinate system the data is in prior to performing this rotation.
-
EPICstuff.ADCPcdf2ncEPIC.
cal_earth_rotmatrix
(heading=0, pitch=0, roll=0, declination=0)¶ this transformation matrix is from the R.D. Instruments Coordinate Transformation booklet. It presumes the beams are in the same position as RDI Workhorse ADCP beams, where, when looking down on the transducers:
Beam 3 is in the direction of the compass' zero reference Beam 1 is to the right Beam 2 is to the left Beam 4 is opposite beam 3 Pitch is about the beam 2-1 axis and is positive when beam 3 is raised Roll is about the beam 3-4 axis and is positive when beam 2 is raised Heading increases when beam 3 is rotated towards beam 1
Nortek Signature differs in these ways:
TRDI beam 3 = Nortek beam 1 TRDI beam 1 = Nortek beam 2 TRDI beam 4 = Nortek beam 3 TRDI beam 2 = Nortek beam 4 Heading, pitch and roll behave the same as TRDI
Parameters: - heading (float) – ADCP heading in degrees
- pitch (float) – ADCP pitch in degrees
- roll (float) – ADCP roll in degrees
- declination (float) – heading offset from true, Westerly is negative
Returns:
-
EPICstuff.ADCPcdf2ncEPIC.
calc_beam_rotmatrix
(theta=20, convex=True, degrees=True)¶ Calculate the rotation matrix from beam coordinates to instrument head coordinates. per dolfyn rotate.py code here: https://github.com/lkilcher/dolfyn
Parameters: - theta (float) – is the angle of the heads (usually 20 or 30 degrees)
- convex (int) – is a flag for convex or concave head configuration.
- degrees (bool) – is a flag which specifies whether theta is in degrees or radians (default: degrees=True)
-
EPICstuff.ADCPcdf2ncEPIC.
doEPIC_ADCPfile
(cdfFile, ncFile, attFile, settings)¶ Convert a raw netcdf file containing data from any 4 beam Janus acoustic doppler profiler, with or without a center beam, and transforms the data into Earth coordinates. Data are output to netCDF using controlled vocabulary for the variable names, following the EPIC convention wherever possible.
Parameters: - cdfFile (str) – raw netCDF input data file name
- ncFile (str) – output file name
- attFile (str) – text file containing metadata
- settings (dict) –
a dict of settings as follows:
'good_ensembles': [] # starting and ending indices of the input file. For all data use [0,np.inf] 'orientation': 'UP' # uplooking ADCP, for downlooking, use DOWN 'transducer_offset_from_bottom': 2.02 # in meters 'transformation': 'EARTH' # | BEAM | INST 'adjust_to_UTC': 5 # for EST to UTC, if no adjustment, set to 0 or omit
-
EPICstuff.ADCPcdf2ncEPIC.
floor
(dec)¶ convenience function to round down provided to avoid loading the math package and because np.floor was causing unexpected behavior w.r.t ints
Parameters: dec (float) – Returns: rounded number
-
EPICstuff.ADCPcdf2ncEPIC.
inst2earth
(adcpo, reverse=False, fixed_orientation=False, force=False)¶ Rotate velocities from the instrument to earth coordinates.
Parameters: - adcpo (dict) – containing the data in instrument coordinates
- reverse (bool) – If True, this function performs the inverse rotation (earth->inst).
- fixed_orientation (bool) – When true, take the average orientation and apply it over the whole record.
- force (bool) – When true do not check which coordinate system the data is in prior to performing this rotation.
Notes
The rotation matrix is taken from the Teledyne RDI ADCP Coordinate Transformation manual January 2008
When performing the forward rotation, this function sets the ‘inst2earth:fixed’ flag to the value of fixed_orientation. When performing the reverse rotation, that value is ‘popped’ from the props dict and the input value to this function`fixed_orientation has no effect. If ‘inst2earth:fixed’ is not in the props dict then the input value is used.
-
EPICstuff.ADCPcdf2ncEPIC.
read_globalatts
(fname)¶ read_globalatts: read in file of metadata for a tripod or mooring
reads global attributes for an experiment from a text file (fname) called by all data processing programs to get uniform metadata input one argument is required- the name of the file to read- it should have this form:
SciPi; J.Q. Scientist PROJECT; USGS Coastal Marine Geology Program EXPERIMENT; MVCO 2015 Stress Comparison DESCRIPTION; Quadpod 13.9m DATA_SUBTYPE; MOORED COORD_SYSTEM; GEOGRAPHIC + SAMPLE Conventions; PMEL/EPIC MOORING; 1057 WATER_DEPTH; 13.9 WATER_DEPTH_NOTE; (meters), nominal WATER_DEPTH_source; ship fathometer latitude; 41.3336633 longitude; -70.565877 magnetic_variation; -14.7 Deployment_date; 17-Nov-2015 Recovery_date; 14-Dec-2015 DATA_CMNT; platform_type; USGS aluminum T14 quadpod DRIFTER; 0 POS_CONST; 0 DEPTH_CONST; 0 Conventions; PMEL/EPIC institution; United States Geological Survey, Woods Hole Coastal and Marine Science Center institution_url; http://woodshole.er.usgs.gov
Parameters: fname (str) – input file name Returns: dict of metadata
-
EPICstuff.ADCPcdf2ncEPIC.
setupEPICnc
(fname, rawcdf, attfile, settings)¶ Construct an empty netCDF output file to EPIC conventions
Parameters: - fname (str) – output netCDF file name
- rawcdf (Dataset) – input netCDF raw data file object
- attfile (str) – metadata text file
- settings (dict) –
settings as follows:
'good_ensembles': [] # starting and ending indices of the input file. For all data use [0,np.inf] 'orientation': 'UP' # uplooking ADCP, for downlooking, use DOWN 'transducer_offset_from_bottom': 2.02 # in meters 'transformation': 'EARTH' # | BEAM | INST 'adjust_to_UTC': 5 # for EST to UTC, if no adjustment, set to 0 or omit
Returns: netCDF file object
-
EPICstuff.ADCPcdf2ncEPIC.
writeDict2atts
(cdfobj, d, tag)¶ write a dictionary to netCDF attributes
Parameters: - cdfobj (object) – netcdf file object
- d (dict) – metadata
- tag (str) – tag to add before each atrribute name
Returns: dict of metadata as written to file
adcpy.EPICstuff.EPICmisc module¶
Helper functions, mostly EPIC specific
-
EPICstuff.EPICmisc.
EPICtime2datetime
(time, time2)¶ convert EPIC time and time2 to python datetime object
Parameters: - array time (numpy) –
- array time2 (numpy) –
Returns: gregorian time as a list of int, datetime object
-
EPICstuff.EPICmisc.
ajd
(dto)¶ Given datetime object returns Astronomical Julian Day. Day is from midnight 00:00:00+00:00 with day fractional value added.
Parameters: dto (object) – datetime Returns: int Astronomical Julian Day
-
EPICstuff.EPICmisc.
apply_timezone
(cf_units)¶ - In xarray, the presence of time zone information in the units was causing decode_cf to ignore the hour,
- minute and second information. This function applys the time zone information and removes it from the units
Parameters: cf_units (str) – Returns: str
-
EPICstuff.EPICmisc.
catEPIC
(datafiles, outfile)¶
-
EPICstuff.EPICmisc.
cftime2EPICtime
(timecount, timeunits)¶
-
EPICstuff.EPICmisc.
check_fill_value_encoding
(ds)¶ - restore encoding to what it needs to be for EPIC and CF compliance
- variables’ encoding will be examined for the correct _FillValue
Parameters: ds – xarray Dataset Returns: xarray Dataset with corrected encoding, dict with encoding that can be used with xarray.to_netcdf
-
EPICstuff.EPICmisc.
fix_missing_time
(ds, delta_t)¶ fix missing time values change any NaT values in ‘time’ to a time value based on the last known good time, iterating to cover larger gaps by constructing time as we go along. xarray.DataArray.dropna is one way to do this, automated and convenient, and will leave an uneven time series, so if you don’t mind time gaps, that is a better tool.
Parameters: - ds – xarray Dataset, time units are in seconds
- deltat – inter-burst time, sec, for the experiment’s sampling scheme
Returns:
-
EPICstuff.EPICmisc.
jdn
(dto)¶ convert datetime object to Julian Day Number
Parameters: dto (object) – datetime Returns: int Julian Day Number
-
EPICstuff.EPICmisc.
make_encoding_dict
(ds)¶ prepare encoding dictionary for writing a netCDF file later using xarray.to_netcdf
Parameters: ds – xarray Dataset Returns: dict with encoding prepared for xarray.to_netcdf to EPIC/CF conventions
-
EPICstuff.EPICmisc.
resample_cleanup
(datafiles)¶
-
EPICstuff.EPICmisc.
s2hms
(secs)¶ convert seconds to hours, minutes and seconds
Parameters: secs (int) – Returns: hours, minutes and seconds
adcpy.EPICstuff.repopulateEPIC module¶
repopulateEPIC¶
Distribute burst data output from reshapeEPIC along the sample dimension
a burst shaped file output from reshapeEPIC will have two issues that need to be addressed before the data can be used with xarray:
-- reshape time to be one dimension
-- make sure the samples within each burst are index according to their
time stamps. Within burst time stamps will not be preserved
- Usage:
python repopulateEPIC.py shaped_file new_file sample_rate [start=’left’] [drop = None]
param str shaped_file: output from reshapeEPIC, the expected shape of the data is one of:: [time, sample] [time, sample, depth] [time, sample, depth, lat, lon] param str new_file: a new file with the adjusted time, this file, if it exists, will be overwritten param int sample_rate: the sample rate the instrument was intended to use during each burst, in seconds param str start: what the time stamp should be for each burst:: left = beginning of the burst based on the first sample time center = middle of the burst based on first sample and last sample times right = end of the burst based on the last sample time param list drop: variable names to omit from the output file
Created on Wed Oct 3 15:21:53 2018 @author: mmartini
-
EPICstuff.repopulateEPIC.
repopulateEPIC
(*args, **kwargs)¶
adcpy.EPICstuff.reshapeEPIC module¶
reshapeEPIC¶
apportion a continuous time series file into bursts (e.g. reshape)
Notes
- the expected dimensions are [time, depth, lat, lon] for continuous data in EPIC
- we are reshaping to [time, sample, depth, lat, lon]
- for ADCP files, beams are expressed as variables vel1, vel2, … veln
- if there is a shape problem then the data file might not be properly understood.
It might be that this code won’t work, this problem will become evident if an error is produced and operation returns to the keyboard in debug mode. If this happens, check the shapes of the variables.
WARNING: time may not be monotonically increasing within bursts (e.g. along the sample dimension) this means that if the number of samples per burst is inconsistent, or if there are gaps in time, the end of a burst may be fill_value, including time values
Marinna Martini for the USGS in Woods Hole, 9/20/2018 originally coded for MATLAB as reshapeEPIC https://cmgsoft.repositoryhosting.com/trac/cmgsoft_m-cmg/browser/trunk/MMstuff/reshapeEPIC.m
Created on Thu Sep 20 14:52:42 2018
@author: mmartini https://github.com/mmartini-usgs
-
EPICstuff.reshapeEPIC.
find_boundaries
(data, edges)¶ using a list of start and end timestamps (edges) that delineate the beginning times and ending times of burts of measurements, find the indices into the data that correspond to these edges. The time base may be irregular, it does not matter.
Parameters: - data (list) – time stamps from the data
- edges (list[tuple]) – start and end times
Returns: list of indices
-
EPICstuff.reshapeEPIC.
find_first_masked_value
(x)¶ helper function to find the first occurrence of a masked value in a numpy masked array returns None if no masked values are found :param numpy array x: :return: index
-
EPICstuff.reshapeEPIC.
generate_expected_start_times
(cdffile, dim, burst_start_offset, burst_interval, burst_length, sample_rate)¶ generate a regular and recurring set of start and end timestamps that delineate the beginning times and ending times of burts of measurements
Parameters: - cdffile (str) – name of a continuous time series data file
- dim (str) – the unlimited or time dimension which we will find the indices to reshape
- burst_start_offset (int) – when to start to make bursts in the continuous data, seconds
- burst_interval (int) – time between start of bursts, seconds
- burst_length (int) – number of samples in a burst
- sample_rate (int) – Hertz
Returns: list of tuples of start and end times for each burst
-
EPICstuff.reshapeEPIC.
reshapeEPIC
(cont_file, burst_file, burst_length, dim='time', edges=None, drop=None, variable_attributes_to_omit=None, verbose=False)¶ apportion a continuous time series file into bursts (e.g. reshape)
Usage: issue_flags = reshapeEPIC(cont_file, burst_file, burst_length, dim=None, edges=None, drop=None)
Parameters: - cont_file (str) – name of netCDF file with continuous data
- burst_file (str) – name of file to store the reshaped data, attributes will be copied
- burst_length (int) – maximum number of samples in each burst
- dim (str) – name of dimension along which we will split the data, usually ‘time’ or ‘Rec’
- edges (list[tuple]) – [(start0, end0), (start1, end1), …] of edges defining the edges of each burst
- drop (str) – set of variable names to omit from the output file
- variable_attributes_to_omit (str) – variable attributes to omit from output file
- verbose (bool) – get lots of feedback to STDOUT
Returns: dictionary of problem types and status
-
EPICstuff.reshapeEPIC.
save_indexes_to_file
(cdffile, edge_tuples, index_file=None)¶ write indexes to a file with the time stamps for QA/QC
Parameters: - cdffile (str) – the continuous time series netCDF file being operated upon
- edge_tuples (list[tuple]) – the bursts to output
- index_file (str) – a file to output a string listing of time stamps