Live File Processor Agent Example

1.Add the following settings to the ST_CONFIG table (CONFIG_SECTION = 'Live.FileProcessor', CONFIG_KEY = 'Agent').

OBJECT_TYPE	OBJECT_VALUE	STRING_VALUE	Description
E-BAM	folderWatcherPath	C:\EarthSoft\Live\E-BAM
E-BAM	folderWatcherFilterRegex	^(.[^\s].)_ ([^_][^\s_][^_])_ ([^_][^\s_][^_])_ ([^_]*)$	Ignores files whose names do not match this pattern, e.g., VNTR_AM_MV-1_PM10_06162019.csv •DT_LOGGER.FACILITY_ID = DT_FACILITY.FACILITY_ID where FACILITY_CODE = 'VNTR_AM' (facility codes may contain an '_'). •DT_LOGGER.SYS_LOC_CODE = 'MV-1'. •DT_LOGGER_SERIES.LOCATION_PARAM_CODE = 'PM10'.
E-BAM	facilityCodeFileNameRegexGroups	^(?'a'.)_(.)_(.)_(.)$	DT_LOGGER.FACILITY_ID = DT_FACILITY.FACILITY_ID where FACILITY_CODE = 'VNTR_AM'
E-BAM	sysLocCodeFileNameRegexGroups	^(.)_(?'a'.)_(.)_(.)$	DT_LOGGER.SYS_LOC_CODE
E-BAM	loggerCodeFileNameRegexGroups	^(.)_(?'a'.)_(.)_(.)$	DT_LOGGER.LOGGER_CODE
E-BAM	locationParamCodeFileNameRegex Groups	^(.)_(.)_(?'a'.)_(.)$	DT_LOGGER_SERIES. LOCATION_PARAM_CODE
E-BAM	dateTimeColumnHeaderRegex	^Time$	DT_LOGGER_DATUM.DATUM_UTC_DT; column header text is Time.
E-BAM	seriesNameColumnHeaderRegex Groups	((?'a'.)\(\|(?'b'.))	DT_LOGGER_SERIES.SERIES_NAME
E-BAM	seriesUnitColumnHeaderRegex Groups	$(?'a'[^)])$[^(]$	DT_LOGGER_SERIES.SERIES_UNIT

2.Start the agent and copy the following file, named VNTR_AM_MV-1_PM10_06162019.csv, to folder C:\EarthSoft\Live\E-BAM.

Time,ConcRT (mg/m3),ConcHr (mg/m3),Flow (l/m),WS (m/s),WD (Deg),AT (C),RHx (%),RHi (%),BV (V),FT (C),Alarm,Type

10/23/2018 8:00,0.006,0.009,16.7,1.3,71,26.5,0,49,14,34.2,0,1

10/23/2018 8:30,0.006,0.009,16.5,2.1,79,29.2,0,50,14.1,34.6,0,1

10/23/2018 9:00,0.024,0.018,16.7,2.8,93,30.3,0,50,14,34.9,0,1

3.The file's name contains the DT_LOGGER.FACILITY_ID, SYS_LOC_CODE/LOGGER_CODE (and DT_LOGGER_SERIES.LOCATION_PARAM_CODE) fields. Find/add the logger row, for example:

LOGGER_CODE	LIVE_DATA_SOURCE	FACILITY_ID	SYS_LOC_CODE
MV-1	COMPUTER_NAME/ C:/EarthSoft/Live/E-BAM	DT_FACILITY.FACILITY_ID, where FACILITY_CODE = 'VNTR_AM'	MV-1

4.The file's first non-empty line is the header line (because headerLinesRegex is not defined); it contains the DT_LOGGER_DATUM.DATUM_UTC_DT column Time. The other columns are the series; they contain DT_LOGGER_SERIES.SERIES_NAMEs and SERIES_UNITs.

5.For each subsequent line:

a.For each series (each column, excluding Time column):

i.Find/add the series row for the SERIES_NAME.

ii.Add the datum row (e.g., for the first three series in the first data line):

SERIES_NAME	DATUM_UTC_DT	DATUM_VALUE	SERIES_UNIT
ConcRT	2018-10-23 08:00:00	0.006	mg/m3
ConcHr	2018-10-23 08:00:00	0.009	mg/m3
Flow	2018-10-23 08:00:00	16.7	l/m

The DT_LOGGER_SERIES.LOCATION_PARAM_CODE = 'PM10', for each of these series.

Live File Processor Agent Example – E-BAM

Live File Processor Agent Example – E-BAM