The Process Data Manager (PDM) provides the capability to import parameter values from real time data collection systems to your Essential database. The data can be used in emission calculations and for regulatory reporting to meet environmental and GHG data quality requirements.
Key features include:
Time-series data mapping to parameters
Rule-based validation
Review/Approval work flow
Notifications/Alerts
Automatic and manual import modes
Error and audit logging
Logical grouping of parameters for efficient management
The PDM can provide a connection for a wide range of process historians, process control systems, time series data managers, etc. Examples include OSIsoft PI, AspenTech IP21, LIMS, and Honeywell PHD. The PDM embeds the OSIsoft PI system (referred to as E-PI). Microsoft Excel and other systems that follow OPC standards can also be used to interface with the PDM.
The diagram below provides a high-level illustration of how your source data flows through the PDM and populates parameter results in your Essential database. Additional details about the data flow follow the diagram.
|
|
The PDM E-PI process validates the imported raw data, performs aggregation, and maps tags to parameters according to the established configuration. |
|
|
Valid data is exported to the PDM Pre-staging table at a frequency that is based on a user-defined automatic schedule. An option is also available to manually import the E-PI data into Pre-staging. |
|
|
The Frequency Expansion process uses Pre-staging data to identify missing values. Pre-staging is updated with the missing points when found. |
|
|
The Rule Engine applies user-defined operations and rules to Pre-staging data. Based on rule definitions, the engine may query the Essential database for supporting data. |
|
|
Deviation percentages are calculated and all values are added to the PDM Staging table. When approval is not required, the Essential database is automatically updated with the final parameter results. Any records that failed the automatic update process are flagged in the Staging table and available for manual review and adjustment. |
|
|
The PDM Staging table stores final parameter results that have been imported into your Essential database and pending parameter results. A user interface is available to review pending values, override invalid results, revalidate incomplete results, approve valid results, and export the final parameter results to the Essential database. |
|
|
A separate validation process is available for parameter results that were entered into Essential using methods other than the PDM, e.g., the data entry results forms or Data Importer tool. |
For additional information about the PDM, click the following links or use the scroll bar to scan the page.
Establish Parameter Groups and assign parameters, user authorization, and alert/notification contacts to each group.
Custom validation and substitution rules can be defined and are applied by the Rule Engine when specific conditions outlined by the rule are met. Rules can be configured to define how missing values are processed, flag values that are BDL, and populate the Comments field for Essential data entry results upon a successful import. Once a rule is defined and compiled, typically by your system administration group, it is registered using the Rule Library form. Registered rules can be assigned to the parameter-to-tag mapping relationship. System administrators should refer to the Process Data Manager Rule Development Guidelines document for additional information.
After determining which tags and properties, e.g., temperature, pressure, flow, leak percent, etc., are to be added for each piece of monitoring equipment, parameter-to-tag mapping relationships need to be established on the Parameter Tag Mapping form. Specify the source tag name and the corresponding parameter, i.e., the parameter name, and when applicable, the material qualifier. A unit of measure for the parameter result is required. Parameter results are flagged for review and approval when the change between the previous parameter result and the current result is above or below 10% (default). The Deviation Percentage +/- field is available to enter a different percentage. Flagged parameter results are stored in the PDM Staging table and are available for manual processing. One or more validation rules can be assigned to the mapping relationship, as well as the order in which the rules will be applied.
Tag values are imported in batches based on parameter groups and start/end dates. On the Parameter Group form, establish groups of parameters that have been mapped to tags whose monitoring results are collected at the same time. For each parameter group, monitoring parameters from any parameter level (entity, process unit, production unit, or emission unit) can be assigned.
Parameter Group Import Approvers are assigned by entity and are authorized to approve pending parameter results in the PDM Staging table and initiate the import of approved results into your Essential database. Approvers are also permitted to override parameter results prior to approval.
In order to identify missing values for a parameter, a frequency with a defined Recurrence Pattern must be assigned to the parameter. A recurrence pattern defines the interval between expected parameter monitoring data imports and can be based on intervals of minutes, hours, days, weeks, months, or years. For weekly, monthly, or yearly recurrence patterns, a Frequency Pattern Effective Date is required to calculate the date range of each interval in the recurrence series. If no frequency is assigned to a parameter, or if a frequency with no recurrence pattern is assigned, the Frequency Expansion process will not be applied for the parameter. The existing Pre-staging data will be used. Define recurrence patterns on the Frequencies form and assign frequencies to parameters on the Parameter Definition form.
The following logic determines the unit of measure assigned to missing value records generated by the Frequency Expansion process:
Use the UOM for the parameter-to-tag mapping record.
Use the default result UOM for the parameter definition record if no mapping relationship exists.
Use the base UOM for the parameter validation record if no mapping relationship exists or no default result UOM has been defined.
Each record in the PDM is identified by its import timestamp (date/time). Date ranges are part of import batch definitions and parameter charting criteria. When a date range is used in the PDM, the start date is inclusive and the end date is exclusive. Exclusive end dates are used to ensure records with timestamps between 11:59:59 p.m. and 12:00:00 a.m. are captured. For example, to import process data for the month of November, 2009, the appropriate start date = 11/1/2009 12:00:00 a.m. and end date = 12/1/2009 12:00:00 a.m. Records with timestamps beginning on 11/1/2009, at 12:00:00 a.m. through just before 12/1/2009 at 12:00:00 a.m. are imported.
When tag values are received by the PDM E-PI, the data is processed as follows:
Aggregation, (minimum, maximum, average, sum total) is performed.
Tag values are mapped to parameters.
Parameter group configurations are applied.
Valid data is exported to the PDM Pre-staging table at a frequency that is based on an automatic schedule defined on the Parameter Group Import Scheduler form. It is recommended that a daily import schedule be defined. An option is also available to manually import E-PI data into Pre-staging when necessary.
For manual imports, the parameter results are placed in the PDM Staging table for approval; user intervention is required to export the data to your Essential database. The manual method allows authorized approvers to review and override results before the data populates the Essential database. For automatic imports, the parameter results are added directly to your Essential database and stored in the Staging table when the import process is complete—unless a parameter group has been designated as requiring approval. In such cases, the parameter results for the group are added to the Staging table only and must be manually approved and imported into your Essential database. With both methods, any data that fails the import process is flagged and stored in the Staging table for manual processing.
Process data resides in the PDM Pre-staging table. The source of the process data is tracked for Origin field population. Existing data can be updated by the results of the Frequency Expansion evaluation, by the Staging table revalidation process, by parameter results validation, or by the manual import of E-PI data. The logic for updating existing Pre-staging values is described below.
If the new value is a parameter result already entered in the Essential database, the existing value will be overwritten.
If the new value is from a manual import, the existing value is overwritten only if its Origin is E-PI or Frequency Expansion.
If the new value is due to Staging table revalidation, the existing value is overwritten only if its Origin is E-PI, Frequency Expansion, or Review (previous Staging table revalidation).
Origin updates can be viewed on the Parameter Group Log and Parameter Log reports.
When a frequency with a recurrence pattern has been assigned to a parameter, the Frequency Expansion process evaluates Pre-staging data to identify missing points. Recurrence patterns define the interval between expected parameter monitoring data imports via the PDM and can be based on intervals of minutes, hours, days, weeks, months, or years.
For Minute-, Hourly-, and Daily-based recurrence patterns, the Frequency Expansion process checks for data during the entire day beginning at 12:00 a.m. In addition, the entire hour is checked for an Hourly-based recurrence pattern. For Weekly, Monthly, or Yearly recurrence patterns, the process checks for data during the entire 24 hours of the day specified.
The Frequency Pattern Effective Date required for Weekly, Monthly, or Yearly recurrence patterns is used to calculate the date range of each interval in the recurrence series.
The following logic determines the UOM assigned to a missing value record:
Use the UOM for the parameter-to-tag mapping record.
Use the default result UOM for the parameter definition record if no mapping relationship exists.
Use the base UOM for the parameter validation record if no mapping relationship exists or no default result UOM has been defined.
If no frequency is assigned to a parameter, or if a frequency with no recurrence pattern is assigned, missing data cannot be identified. In such cases, existing Pre-staging values skip the Frequency Expansion process and are evaluated by the Rule Engine.
Custom validation and substitution rules can be defined for data flowing from the PDM to your Essential database. Rules are applied to each value within a batch by the Rule Engine. If rules depend on reference parameters, any rules for those reference parameters will run first. For this reason, it is highly recommended that users include the reference parameters in the same parameter group so that any new data comes into the Rule Engine in the same batch. If reference parameters are not included in the batch, or do not contain any new data, their values will be retrieved from the following Essential database tables:
Parameter results
Staging
Prestaging
The property value will be set to the value from the first table in this list that contains a value for the reference parameter at the timestamp nearest to the current timestamp. If no value exists, the property will not be set.
After rules have been applied, the current value is compared to the previous parameter result and the deviation percentage is calculated. All parameter results are added to the PDM Staging table. When parameter results are part of an automatic import job and user-approval is not required, the results also automatically populate your Essential database. Otherwise, the parameter results must be approved and manually imported via the Parameter Group Import Review form.
System administrators should refer to the Process Data Manager Rule Development Guidelines document for additional information.
The Parameter Group Import Review form is the interface to the data in the PDM Staging table. It is a comprehensive view of parameter results data that includes:
Parameter Result Import Status, e.g., Pending Approval, Failed, Imported, etc. For records marked as Failed, the failure reason is provided.
Parameter identification details that include parameter type, unit name, parameter name, material qualifier, tag name, and material name
Import date and time
Parameter result information, such as the imported result, the prior result, and the deviation percentage between these two results. Indicators show whether the result has been validated and/or substituted based on rule application, has been identified as a missing point, or is below the detection limit. Override values and comments are also available for review.
Data can be viewed by parameter group; several filtering options are available to display a specific subset of the data. When applicable, the Parameter Group Imports Pending Approval Review and Parameter Group Imports Failed Validation Review browse views also provide access to the Parameter Group Import Review form with a subset of data based on the import status. Review, override, discard, re-validate, and approve pending parameter results on the form, as well as initiate the import of approved results to your Essential database. Just the reviewers authorized as an import approver for the parameter group can import results. Reviewers without approval authorization can edit data, but must submit the data for approval.
The Parameter Chart analysis tool is available to help with parameter data quality assurance before import in to your Essential database. View relationships between different parameter values stored in the PDM Staging table. For example, a chart can be used to review historical parameters for trending as well as projecting futures values. Plot Time Series data for one or more parameters or plot Parameter vs. Parameter data. Line, column, area, bubble, and point (scatter) chart formats are available. Data used for charting can also be exported to Excel. The Parameter Chart tool can be accessed from the Parameter Import Review form and the Parameter Group Import Review form for the selected parameter.
Note:
A line chart of the values for a selected parameter in an import
batch can be viewed on the Parameter
Group Import Review form. Both staging values (pending import)
and parameter results (imported to Essential) are plotted.
After PDM PI system processing is complete, parameter results are added to the PDM Staging table and for automatic imports when no approval is required, your Essential database is updated. If approval is required, an authorized user must approve the pending parameter results and manually initiate the import of data to your Essential database.
Note:
When a parameter result to be imported is not different than the
parameter's previous result, the record in the Essential database is not
updated.
A separate validation process is available for parameter results that were entered into your Essential database using methods other than the PDM, such as the data entry forms or the Data Importer tool. On the Validate Parameter Results form, parameter results from your Essential database can be directly imported into the Pre-staging table where the Frequency Expansion process is initiated, rules are applied, and the Staging table is populated with the results. An option to preview the records before initiating the validation process is available.
At key points in PDM processing, automatic email notifications can be sent to one or more Essential contacts. The available notification types are identified below. For each parameter group, specify the contacts who are to receive automatic email alerts, as well as the type(s) of notifications, on the Parameter Group form. The contact's email address provided in the contact and personnel record populates the recipient field and the email notification-related configuration file setting identifies the email sender.
Success - a notification message is sent when a manual or automatic batch import occurs.
Failed - a notification message is sent when a manual or automatic batch import fails.
Rule Notifications - if a rule has been configured to generate an event-driven type of notification, a message is sent when an operation associated with a parameter-to-tag mapping relationship is executed during the PDM PI process.
Pending Approval - a notification message is sent when there are parameter results requiring review and approval.
Submitted for Approval - a notification message is sent when parameter results have been reviewed and submitted for approval.
The Parameter Group Log Report (filtered to parameter groups) and Parameter Log Report (filtered to individual parameters) are available to help track and manage PDM activities. Both log reports allow users to track parameter result data, including all changes, from the Pre-staging table through Essential database population. Several date and transaction type filters are available to help quickly locate a specific set of data for review. Log data can be exported to Excel.
Data import interruptions can be reviewed on the Parameter Group Import History Gap browse view. The date and time of the previous import and the next import are available to help determine if import data is missing. The parameter group link provides access to the Parameter Group Import Review form.
The Purge Pre-staging/Staging form is available to delete records no longer needed from the PDM Pre-staging and/or Staging tables. Enter a date range to purge records with timestamps that fall within that range. Or, enter an end date and time only to purge all records with timestamps prior to that date/time. Optionally, use one or more of the following filters with the date range or end date to further define the specific records to be purged:
Enterprise Entity
Parameter Group
Process Unit
Parameter
Import Status
Purged records are completely deleted from the table(s) specified.
