Live File Processor Agent Example – AQS RD Transaction

<< Click to Display Table of Contents >>

Navigation:  Live > Agents > File Agents > File Processor > Examples >

Live File Processor Agent Example – AQS RD Transaction

The Air Quality System (AQS) is an information management system for ambient outdoor airborne pollutants and related meteorological data that is administered by the United States Environmental Protection Agency. The Live File Processor Agent can be used to load AQS Raw Data (RD) Transaction Type data into EQuIS. See the EQuIS AQS Module documentation for more information about AQS data.

 

Add the following settings to the ST_CONFIG table (CONFIG_SECTION = 'Live.FileProcessor', CONFIG_KEY = 'Agent').

 

OBJECT_TYPE

OBJECT_VALUE

STRING_VALUE

Description

Default

dateTimeFormats

en-AU

Note, this is a default setting (for all processors, unless overridden by a specific processor).

AQS-RD Transaction

dateTimeColumnHeaderRegex

(?i)Date

DT_LOGGER_DATUM.DATUM_UTC_DT; column header text starts with Date Time.

AQS-RD Transaction

dateTimeFormats

yyyyMMdd

The format used to process DateTimes.

AQS-RD Transaction

datumValueColumnHeaderRegex

=> NULL || NaN => NULL ||NULL => NULL

Replace these case-insensitive string datum values (from the file) with these float or NULL values.

AQS-RD Transaction

delimiters

|

The fields are separated by tabs and/or commas.

AQS-RD Transaction

folderWatcherPath

C:\EarthSoft\Live\AQS-RD-Transaction

The Folder that will be watched for new data.

AQS-RD Transaction

headerLinesDefaultValue

RD|Action Code |State Code|County Code|Site ID|Parameter|POC|Sample Duration|Unit|Method|Date|Start Time|Sample Value|Null Data Code|Sampling Frequency|Monitor Protocol (MP) ID|Qualifier - 1|Qualifier - 2|Qualifier - 3|Qualifier - 4|Qualifier - 5|Qualifier - 6|Qualifier - 7|Qualifier - 8|Qualifier - 9|Qualifier - 10|Alternate Method Detectable Limit|Uncertainty

Used if headerLinesRegex is not defined or is empty, or a file does not start with headerLinesRegex.

 

AQS-RD Transaction

ignoreLinesRegex

^#

Line that starts with the given regex will be ignored.

AQS-RD Transaction

loggerCodeColumnHeaderRegex

(?i)State Code || (?i)County Code || (?i)Site ID || (?i)RD || (?i)Transaction Type

LOGGER_CODE from file column value(s).

AQS-RD Transaction

remarkQualifierColumnHeaderRegex

(?i)Qualifier

The column header regex to use for the remark qualifier.

AQS-RD Transaction

remarkQualifierType

AQS Qualifier

The type of qualifier.

 

Qualifiers are saved in the  DT_LOGGER_REMARK table.

AQS-RD Transaction

remarkQualifierJoin

|

The delimiter to be used when multiple qualifiers are present.

AQS-RD Transaction

remarkNullDataCodeColumnHeaderRegex

(?i)Null Data Code

The column to be used as the Null Data Code.

AQS-RD Transaction

remarkNullDataCodeType

Null Data Code

The type of Null Data Code.

AQS-RD Transaction

remarkNullDataCodeJoin

|

The delimiter to be used when multiple Null Data Codes are present.

AQS-RD Transaction

seriesNameColumnHeaderRegex

(?i)Parameter || (?i)POC ||
(?i)Method$ || (?i)Unit ||
(?i)Assessment Number ||
(?i)Sample Duration

SERIES_NAME from file column value(s).

AQS-RD Transaction

seriesNameColumnHeaderRegexGroups

(?'a'.*)

SERIES_NAME from file column header.

AQS-RD Transaction

sysLocCodeColumnHeaderRegex

(?i)State Code || (?i)County Code || (?i)Site ID

The column to be used as the Null Data Code.

AQS-RD Transaction

timeColumnHeaderRegex

(?i)Time

Time (only) column header text pattern.

 

Example File

1.Start the agent, and copy the following files to folder C:\EarthSoft\Live\AQS-RD-Transaction.

# RD|Action Code|State Code|County Code|Site ID|Parameter|POC|Sample Duration|Unit|Method|Date|Start Time|Sample Value|Null Data Code|Sampling Frequency|Monitor Protocol (MP) ID|Qualifier - 1|Qualifier - 2|Qualifier - 3|Qualifier - 4|Qualifier - 5|Qualifier - 6|Qualifier - 7|Qualifier - 8|Qualifier - 9|Qualifier - 10|Alternate Method Detectable Limit|Uncertainty

RD|I|27|003|1002|11101|1|7|001|091|20120403|0:00||AN|1|||||||||||||

RD|I|27|003|1002|11101|1|7|001|091|20120403|1:00||AN|1||DI|DN|EH|MD|MX|NS|QP|SS|SX|TB||

2.If headerLinesRegex is not defined, then the default column order defined in headerLinesDefaultValue is used.