Live File Processor Agent Example – RST-2

<< Click to Display Table of Contents >>

Navigation:  Live > Agents > File Agents > File Processor > Examples >

Live File Processor Agent Example – RST-2

1.Add the following settings to the ST_CONFIG table (CONFIG_SECTION = 'Live.FileProcessor', CONFIG_KEY = 'Agent').

 

OBJECT_TYPE

OBJECT_VALUE

STRING_VALUE

Description

RST-2

dateTimeColumnHeaderRegex

TIMESTAMP

DT_LOGGER_DATUM.DATUM_UTC_DT; column header text contains TIMESTAMP.

RST-2

folderWatcherPath

C:\EarthSoft\Live\RST-2

RST-2

headerLinesRegex

TIMESTAMP\nUNITS

The header has 2 lines that look like this:

1.Contains TIMESTAMP

2.Contains UNITS

 

Note: Lines starting with ,,, are ignored, so the TIMESTAMP and UNITS lines are 'read' one after the other.

RST-2

headerLinesRegexReplacements

(?'a'[^,]+),,,,,(?'b'[^,]+) => ${a},,,,${b}

Replace ,,,,, with ,,,, in the file's header.

 

Note: The example file has an extra comma in the TIMESTAMP line, which is not present in the UNITS and data lines.

RST-2

ignoreColumnHeadersRegex

RECORD

Do not add series or datum for columns whose header text contains RECORD.

RST-2

ignoreLinesRegex

^,,,

Ignore lines starting with ,,,.

RST-2

logDuplicateDatumRowErrors

false

See Example File 2, below.

RST-2

loggerCodeHeaderLinesRegex
Groups

^Name,(?'a'[^,]*)

DT_LOGGER.LOGGER_CODE

RST-2

seriesNameColumnHeaderRegex
Groups

^(?'a'[^\n]*)

DT_LOGGER_SERIES.SERIES_NAME

RST-2

seriesUnitColumnHeaderRegex
Groups

\n(?'a'.*)$

DT_LOGGER_SERIES.SERIES_UNIT

RST-2

utcOffsetDefaultValue

2

DT_LOGGER_DATUM.DATUM_UTC_DT = file dates and times – 2 hours (Solwezi, Zambia time; UTC + 2 hours; not adjusted for DST).

 

 

Example File 1

 

2.Start the agent and copy the following file to folder C:\EarthSoft\Live\RST-2.

 

 

Model,DT2055 VW Multi Channel Logger

Name,CGT086

Serial Number,02593

Firmware Version,v1.20,Software Version,v1.31.0

Sampling Status,LOGGING,APPEND

Sampling Rate,FIXED

Current Interval,12 Hours  0 Minutes  0 Seconds

Start Time,Mon  Oct  22  18  12:00:00  PM

Current Time,Tue  Nov  27  18  10:12:42  AM

Number Of Sensors,8

Number Of Records,72

Sensor Upload Num / Error Code,7 / 80000

 

"TIMESTAMP","RECORD","BATTERY","VW 1","Therm 1","VW 2","Therm 2","VW 3","Therm 3","VW 4","Therm 4",,,,,"VW 1","Therm 1","VW 2","Therm 2","VW 3","Therm 3","VW 4","Therm 4","LOGGER TEMP","RESET"

,,,,,,,,,,,,,Unit Type,"Pressure","Disabled","Pressure","Disabled","Pressure","Disabled","Pressure","Disabled"

,,,,,,,,,,,,,Input Units,"kPa","Disabled","kPa","Disabled","kPa","Disabled","kPa","Disabled"

,,,,,,,,,,,,,Output Units,"m H2O","Disabled","m H2O","Disabled","m H2O","Disabled","m H2O","Disabled"

,,,,,,,,,,,,,Offset,"8.84600e+001","Disabled","5.69600e+001","Disabled","3.94600e+001","Disabled","2.24600e+001","Disabled"

,,,,,,,,,,,,,Cal A =,"5.64600e-004","Disabled","2.73700e-004","Disabled","2.89500e-004","Disabled","2.98200e-004","Disabled"

,,,,,,,,,,,,,Zero B =,"7.28600e+003","Disabled","7.01000e+003","Disabled","7.70800e+003","Disabled","8.34800e+003","Disabled"

,,,,,,,,,,,,,Cal C =," ","Disabled"," ","Disabled"," ","Disabled"," ","Disabled"

,,,,,,,,,,,,,Temp CF,"4.35600e-001","Disabled","9.30800e-002","Disabled","7.44500e-002","Disabled","1.09500e-001","Disabled"

,,,,,,,,,,,,,Temp Initial,"2.50000e+001","Disabled","2.30000e+001","Disabled","2.50000e+001","Disabled","2.50000e+001","Disabled"

,,,,,,,,,,,,,Linked Channel,"1","Disabled","3","Disabled","5","Disabled","7","Disabled"

"UNITS",,"Volts","m H2O","deg C","m H2O","deg C","m H2O","deg C","m H2O","deg C",,,,"B Units","Ohms","B Units","Ohms","B Units","Ohms","B Units","Ohms"

2018/10/22 12:00:00,1,2.9,88.32,22.86,56.92,22.39,39.42,22.14,22.41,22.11,,,,8110.05,3297.91,8102.74,3368.05,8493.80,3406.23,8994.10,3410.74,29.50,0

2018/10/23 00:00:00,2,2.9,88.32,22.86,56.92,22.39,39.42,22.14,22.41,22.11,,,,8109.56,3297.91,8101.93,3368.05,8493.53,3406.23,8993.73,3410.74,16.60,0

2018/10/23 12:00:00,3,2.9,88.32,22.86,56.92,22.39,39.42,22.14,22.41,22.11,,,,8109.31,3297.91,8102.43,3368.05,8493.80,3406.23,8993.81,3410.74,32.30,0

 

 

3.The file's Name line contains the DT_LOGGER.LOGGER_CODE column. Find/add the logger row. For example:

 

LOGGER_CODE

LIVE_DATA_SOURCE

UTC_OFFSET_HRS

CGT086

COMPUTER_NAME/C:/EarthSoft/Live/RST-2

2

 

4.The line that contains TIMESTAMP, followed by the line that contains UNITS (lines starting with ,,, are ignored), are the header lines; they contain the DT_LOGGER_DATUM.DATUM_UTC_DT column. The RECORD column is ignored. The other columns are the series; they contain DT_LOGGER_SERIES.SERIES_NAMEs and DT_LOGGER_SERIES.SERIES_UNITs.

 

5.For each subsequent line:

a.For each series (each column, excluding TIMESTAMP and RECORD columns):

i.Find/add the series row for the SERIES_NAME and SERIES_UNIT.

ii.Add the datum row (e.g., for the first three series in the first data line):

 

SERIES_NAME

DATUM_UTC_DT

DATUM_VALUE

SERIES_UNIT

BATTERY

2018-10-22 10:00:00 (= 2018-10-22 12:00:00 - utcOffsetDefaultValue)

2.9

Volts

VW 1

2018-10-22 10:00:00

88.32

m H2O

Therm 1

2018-10-22 10:00:00

22.86

deg C

 

 

Note: The offsets, between the TIMESTAMP and UNITS, lines:
 
,,,,,,,,,,,,,Offset,"8.84600e+001","Disabled","5.69600e+001","Disabled","3.94600e+001","Disabled","2.24600e+001","Disabled"
 
are not added to the datum values.
 
The Data Computation Agent or Data Deriver Agent may be used to add the offsets.

 

 

Example File 2

 

The following file contains two VW 1, 2, and 3 columns, with the same units (B Units), and values (and dates and times). RST-2.logDuplicateDatumRowErrors = false, does not log errors for these.

 

 

Model,DT2055 VW Multi Channel Logger

Name,CPD-002

Serial Number,02574

Firmware Version,1.20,Software Version,1.15.4

Sampling Status,LOGGING

Sampling Rate,FIXED

Current Interval,12:0:0

Start Time,2018/09/24,08:50:33

Current Time,2018/11/27,11:43:40,Logger Time,2018/11/27,11:41:55

Number Of Sensors,6

Number Of Records,129,USB

Sensor Upload Num / Status Code,7,80000,FFFFFFFF,1048704,Bat Type Not set,,WD 4,PO 0,EXT 0,BO 0,SW 255,PDI 255

 

"TIMESTAMP","RECORD","BATTERY","VW 1","Therm 1","VW 2","Therm 2","VW 3","Therm 3",,,,"VW 1","Therm 1","VW 2","Therm 2","VW 3","Therm 3","LOGGER TEMP","RESET","USB","RSSI","STATUS 0x"

,,,,,,,,,,,Unit Type,"Disabled"," ","Disabled"," ","Disabled"," ",,,,,"[code][P#]"

,,,,,,,,,,,Input Units,"Disabled"," ","Disabled"," ","Disabled"," "

,,,,,,,,,,,Output Units,"Disabled"," ","Disabled"," ","Disabled"," "

,,,,,,,,,,,Offset,"Disabled"," ","Disabled"," ","Disabled"," "

,,,,,,,,,,,Cal A =,"Disabled"," ","Disabled"," ","Disabled"," "

,,,,,,,,,,,Zero B =,"Disabled"," ","Disabled"," ","Disabled"," "

,,,,,,,,,,,Cal C =,"Disabled"," ","Disabled"," ","Disabled"," "

,,,,,,,,,,,Temp CF,"Disabled"," ","Disabled"," ","Disabled"," "

,,,,,,,,,,,Temp Initial,"Disabled"," ","Disabled"," ","Disabled"," "

,,,,,,,,,,,Linked Channel,"Disabled"," ","Disabled"," ","Disabled"," "

,,,,,,,,,,,Sensor,"U User","3K NTC","U User","3K NTC","U User","3K NTC"

"UNITS->",,"Volts","B Units","deg C","B Units","deg C","B Units","deg C",,,,"B Units","Ohms","B Units","Ohms","B Units","Ohms","deg C"

2018/09/24 08:50:41,1,3.65,8420.08,23.28,8668.99,22.71,9063.68,22.70,,,,8420.080078,3236.779053,8668.990234,3320.150879,9063.679688,3321.922363,23.30,1,0,0,000

2018/09/24 20:50:33,2,3.00,8421.59,23.28,8669.47,22.70,9064.20,22.69,,,,8421.589844,3237.213867,8669.469727,3321.922363,9064.200195,3323.694336,23.00,0,0,0,000

2018/09/25 08:50:33,3,3.00,8422.64,23.27,8669.40,22.69,9064.05,22.68,,,,8422.639648,3238.083740,8669.400391,3323.251221,9064.049805,3325.023926,23.40,0,0,0,000

 

 

 



Copyright © 2023 EarthSoft, Inc • Modified: 20 Dec 2023