Live File Processor Agent Example

1.Add the following settings to the ST_CONFIG table (CONFIG_SECTION = 'Live.FileProcessor', CONFIG_KEY = 'Agent').

OBJECT_TYPE	OBJECT_VALUE	STRING_VALUE	Description
YSI-6	dateTimeColumnHeaderRegex	Date\|Date Time	DT_LOGGER_DATUM.DATUM_UTC_DT date; column header text contains Date, or Date Time.
YSI-6	dateTimeFormats	M/d/y\|d-M-y H:m	Note: This is not a realistic example. The dates will probably be either month-first or day-first, not both. Also, this may be problematic if the date and time can be interpreted as both d/M/y and M/d/y.
YSI-6	folderWatcherPath	C:\EarthSoft\Live\YSI-6
YSI-6	headerLinesRegex	Date.Time\n.	The header line contains Date + anything (.*) + Time.
YSI-6	loggerCodeFileNameRegex Groups	^(?'a'[^_.]*)	DT_LOGGER.LOGGER_CODE = file name text before first underscore (_) or dot (.).
YSI-6	seriesNameColumnHeaderRegex Groups	^(?'a'.*)\n	DT_LOGGER_SERIES.SERIES_NAME
YSI-6	seriesUnitColumnHeaderRegex Groups	\n(?'a'.*)$	DT_LOGGER_SERIES.SERIES_UNIT
YSI-6	timeColumnHeaderRegex	Time	DT_LOGGER_DATUM.DATUM_UTC_DT time; column header text contains Time.
YSI-6	utcOffsetDefaultValue	-5	DT_LOGGER_DATUM.DATUM_UTC_DT = file dates and times + 5 hours (Eastern Standard Time; UTC – 5 hours; not adjusted for DST).

Example File 1

2.Start the agent and copy the following file, named SpringField_A_2019-02-01.csv, to folder C:\EarthSoft\Live\YSI-6.

"Date","Time","Temp","SpCond","Depth","pH","pHmV","Turbidity+","ODO%","ODO Conc"

"M/D/Y","hh:mm:ss","C","mS/cm","m","","mV","NTU","%","mg/L"

"10/11/11","12:21:06",24.32,0.399,0.362,7.59,-55.2,0.0,92.5,7.73

"10/11/11","12:21:43",24.36,0.400,1.112,7.59,-55.1,-0.1,92.3,7.71

"10/11/11","12:22:14",24.37,0.400,2.028,7.59,-54.8,0.0,92.1,7.70

3.The file's name contains the DT_LOGGER.LOGGER_CODE field. (The location/logger code is not in a YSI 6-series file's header or data lines; it can only be obtained from the file's name, or a default value.) Find/add the logger row. For example:

LOGGER_CODE	LIVE_DATA_SOURCE	UTC_OFFSET_HRS
SpringField	COMPUTER_NAME/C:/EarthSoft/Live/YSI-6	-5 (= utcOffsetDefaultValue)

4.The file's Date,Time line, and the line that follows it, are the header lines; they contain the DT_LOGGER_DATUM.DATUM_UTC_DT columns Date and Time. The other columns are the series; they contain DT_LOGGER_SERIES.SERIES_NAMEs and SERIES_UNITs.

5.For each subsequent line:

a.For each series (each column, excluding Date and Time columns):

i.Find/add the series row for the SERIES_NAME.

ii.Add the datum row (e.g., for the first three series in the first data line):

SERIES_NAME	DATUM_UTC_DT	DATUM_VALUE	SERIES_UNIT
Temp	2011-10-11 17:21:06 (= 2011-10-11 12:21:06 - utcOffsetDefaultValue)	24.32	C
SpCond	2011-10-11 17:21:06	0.399	mS/cm
Depth	2011-10-11 17:21:06	0.362	m

Example File 2

The following file has one date and time column, instead of two.

Date Time,Temp,SpCond,DOsat,DO,Depth,pH,pH,Baro,Battery,TDS

D-M-Y HH:MM:SS,C,uS,%,mg/L,meters,,mV,mmHg,volts,g/L

26-07-13 14:18,19.36,492,108.6,9.99,0.082,7.71,-89.6,768.7,11.9,0.32

26-07-13 14:18,19.36,492,109.6,10.08,0.093,7.71,-89.5,768.6,11.9,0.32

26-07-13 14:18,19.37,492,110.5,10.16,0.145,7.71,-89.6,768.7,11.9,0.32

Note: The date and time formats in the file's header, line:

"M/D/Y","hh:mm:ss","C","mS/cm","m","","mV","NTU","%","mg/L"

are not used/supported. Instead, the dateTimeFormats are used.

Also, in Example File 2, the file's header includes seconds, but the data lines do not have seconds.

Live File Processor Agent Example – YSI-6

Live File Processor Agent Example – YSI-6

Example File 1

Example File 2