Parameter values are imported from E-PI to your Essential database in batches based on parameter groups and start/end dates. On the Process Data Import Scheduler form, set up import schedules to automatically import the data. It is recommended that a daily import schedule be defined, although other options, such as hourly and weekly, are available. Data can also be imported manually at any time.
A separate validation process is available for parameter results that were entered into your Essential database using methods other than the PDM, e.g., via the data entry forms or the Data Importer tool. Use the Validate Parameter Results form to import parameter results from your Essential database into the PDM Pre-staging table. The Frequency Expansion process is automatically initiated, rules are applied, and the PDM Staging table is populated with the results.
Parameter results that require approval before being imported into your Essential database reside in the PDM Staging table where they can be marked for approval and manually exported to Essential. Refer to Parameter Import Results Review and Approval for additional information about this import process.
For instruction on importing data for the PDM, click the following links or use the scroll bar to scan the page.
Use the Process Data Import Scheduler form to set up process data import jobs that run automatically according to the schedule configuration. It is recommended that a daily import schedule be defined, although other options, such as hourly and weekly, are available. The form can also be used to run an existing schedule on-demand or to schedule a one-time job to import data manually when necessary. The following information is required to import data:
Schedule Description - an identifying name for the schedule
Import Period - relative time period for recurring automatic schedules, absolute time period for a manual import
Parameter Groups - one or more parameter groups must be associated with the schedule, otherwise no data will be imported.
Job Schedule - several options, such as hourly, monthly, yearly, are available for recurring automatic schedules, as well as one-time options for manual imports. The job schedule also includes the option to flag the imported data for manual review.
The Frequency Expansion process is initiated when an import job runs. If no frequency is assigned to a parameter, or if a frequency with no recurrence pattern is assigned, missing values cannot be identified. Existing Pre-staging values skip the Frequency Expansion process and are evaluated by the Rule Engine. For manual review imports, the parameter results are placed in the PDM Staging table for approval; user intervention is required to transfer the data to your Essential database. The manual review method allows authorized approvers to review and override results before the data populates the Essential database. For automatic imports, the parameter results are added directly to your Essential database and stored in the Staging table when the import process is complete—unless a parameter group has been designated as requiring approval. In such cases, the parameter results for the group are added to the Staging table only and must be manually approved and imported into your Essential database. With both methods, any data that fails the import process is flagged and stored in the Staging table for manual processing.
The Parameter Group Import Review form is the interface to the data in the PDM Staging table. Review, change, and approve pending parameter results on the form, as well as import approved results to your Essential database. The Parameter Group Imports Pending Approval Review and Parameter Group Imports Failed Validation Review browse views are available to expedite the review and approval processes.
Note:
When a parameter result to be imported is not different than the
parameter's previous result, the record in the Essential database is not
updated.
View job and import statuses in the Job Log and Scheduled Imports sections on the Process Data Import Scheduler form.
Specify a relative time period whose start and end date definition is based on the schedule configuration. For example, you may want to import data for the previous month on the 10th day of the current month to ensure all data has been secured, validated, and entered into your collection system. In this case, enter the number 1 and select the Month time period for the Import Values for previous fields and configure the schedule to run on the 10th of every month. On February 10th, the job would run and import data for the month of January; on March 10th, the job would run and import data for the month of February; and so on. All relative time periods begin at the top of the hour, e.g., 12:00, 1:00, 2:00, etc.
Import data with timestamps after the end of the last import up to the beginning of the relative time period specified. For example, you may want to import data on the 16th of each month that includes data from the 16th of the previous month through the 15th of the current month. Select the Month time period from the Import until beginning of field configure the schedule to run on the 16th of every month. On February 16th, the job would run and import date from January 16th through February 15th; on March 16th, the job would run and import data from February 16th through March 15th, and so on.
An Initial start date and time are required for the first time the import job is run; all records with a timestamp beginning on this date/time through the beginning time period selected are imported. Subsequent runs pull data beginning with the last run date/time which is displayed in parentheses next to the Since end date of last import option.
Use the Fixed dates option to manually import data. The Start Date and End Date of the date/time range that includes the timestamp of the records to be imported are required. The dates can be entered by typing directly into each field, selecting a date from the Calendar, or using one of the following options:
Click one of the relative time period filter options. For example, click Last Week and the previous week's Sunday date plus 12:00:00 AM (inclusive) are entered as the start date/time. The end date is the following Sunday date and 12:00:00 AM (exclusive).
When data has been imported, the Last Import Date fields are automatically populated. Click the Since Last Import option; the Start Date fields are completed with the last import date and time. The End Date fields contain the current date and time.
Tip: A future end date can be entered; however, the end date
will be automatically updated with the date/time the job is executed on
the server where the job runs.
To run the manual import, configure the job schedule to run once at a specific date/time or to run once at the next available time. If an email address has been entered on the contact and personnel record of the user who created the schedule, an email message will be sent to the address with a link to the Parameter Group Import Review form after the import process is complete. If an email address does not exist, a message will not be generated. This confirmation process cannot be edited and is separate from the types configured on the Parameter Group form. However, all other associated email alerts (e.g., success, fail, etc.) will be generated as configured. Refer to Establishing Data Import Parameter Groups for additional information.
For non-manual imports, an Emissions Calculator job and/or an ARS job can be scheduled to automatically run for the process data imported once the import is complete. Options are available on the Process Data Import Scheduler form if your Essential implementation includes the Air module and/or the ARS tool. When both post-import jobs are scheduled the calculator job runs first, followed by the ARS job.
Emissions Calculator - the job calculates emission models/scenarios in production schedules for each imported parameter group's entity that are within the date range of the import job. Calculator jobs are run with the Continuous Calculation method. A calculator job will not be triggered when the import job is not successful, e.g., a fatal error has occurred. Invalid imported values will not stop the calculator job from running.
ARS - the job can be run 1) with the imported data's date range and entity or 2) with the parameter of a specified ARS job schedule. When the job is run with the imported data's date range and entity, the job is not listed in the ARS Supervisor Console and no standard ARS email messages that provide integration process run results are generated. If the job is run based on an ARS job schedule, the job is listed in the ARS Supervisor Console and standard ARS email messages are sent to the owner of the scheduled job.
The following notifications will be included in the PDM parameter group email alerts, as applicable:
If the Emissions Calculator was scheduled to automatically run, please review the following for any errors: Calculation Log. (A link to the Calculation Log is provided.)
If ARS was scheduled to automatically run, please review the ARS tables for new data.
Note: Post-import jobs are not triggered for the following:
manual PDM imports
imported Parameter Results requiring manual review
Parameter Groups with Require Import Approver checked
To establish process data import schedules
Tip: At least one parameter group and a job schedule (with
a schedule type that is not No Schedule)
must be specified to import data.
Click Tools > Import Scheduler > Process Data Import Scheduler or Process Data Import Scheduler by Group in the Navigation Tree to open the Import Scheduler list.
Click the New button and enter a name for the schedule in the Schedule Description field.
Select an Import Period and enter the data required for the period selected. Refer to Import Period Descriptions and Examples for additional information when necessary.
Specify post-import Emissions Calculator jobs and/or ARS jobs when applicable. Refer to Post-Import Job Schedules for additional information.
Click Save.
The Parameter Groups, Job Schedule, Job
Log, and Scheduled Imports
sections are displayed.
Click Add Parameter Groups in the Parameter Groups section.
Select one or more groups to be
included in the schedule and click Add
Parameter Groups.
Each group selected is displayed as a line item in the Parameter
Groups section. To remove a group from the schedule, click
the Delete button adjacent
to the line item.
Expand the Job
Schedule section and select a Schedule
Type from the list. Based on the schedule type selected, any
required scheduling options are automatically displayed.
Tip: Be sure the appropriate
parameter groups have been selected before
selecting the Once, next available
time Schedule Type
option. Otherwise, the import job could run without a parameter group
association—in which case, no data would be imported since the job
is scheduled to run just one time.
Use the guidelines below to complete the run schedule.
Select the Manual Review option when parameter results are to be placed in the PDM Staging table for approval before import to your Essential database.
Enter a Start Date and time.
Click the No End Date check box or enter an End Date and time.
For date fields, the Calendar button is available to select a date.
Select all applicable Days of the week for daily-related schedule types.
Select a Schedule Time from the list for each applicable field.
Enter a numerical value that represents the interval between runs in the Run Every field.
A schedule example for the Hourly type might be on Monday, Tuesday, Wednesday, Thursday, and Friday (Days), at 15 minutes after the hour (Schedule Time), every 8 hours (Run Every), beginning November 1, 2009 at 8:00 a.m. (Start Date). The import job would run at 8:15 a.m., 4:15 p.m., and 12:15 a.m. Monday through Friday starting November 1.
Click the Save button.
Repeat steps 2 through 10 to set up another schedule.
To run import schedules on-demand
Click Tools > Import Scheduler > Process Data Import Scheduler or Process Data Import Scheduler by Group in the Navigation Tree to open the Import Scheduler list.
Click the Schedule Description link to open the Process Data Import Scheduler form for the applicable schedule.
Expand the Job
Schedule section and click the Run
On Demand? check box to select it.
The Run on Demand option is
not available for the No Schedule
and one-time only Schedule Type
options.
Click Save.
If the Manual Review option
is not selected, a message is displayed warning that the job may take
a long time to process and there is no way to cancel the job while
in progress. Click OK to continue.
Track job and import statuses in the Job Log and Scheduled Imports sections.
The Process Data Import Scheduler configuration triggers import jobs. A separate job is created for each one-time instance and each recurrence. An entry is added to the APE Job Log for each job. The start and finish date/time, the run status, the results, and machine name are provided for each log entry. Jobs are processed according to a pre-configured priority in a managed, queue-based environment to minimize system contention. Use the APE Job Log to determine your job's position among all the jobs scheduled to run and to determine the current/final run status. When a job is running, the Results field can also be used to track the progress of the import; the total number of records and the number of those records that have been currently processed are provided.
The Application Log can be used to determine if application-related errors were encountered for import jobs. The timestamp, severity, priority, and critical/error/warning message details are displayed. This log may not be available to all users; element-level security applies. Refer to Setting Field-Level (Element) Security for additional information.
The AutoRefresh option allows you to switch between a working mode and the Job/Application Log refresh mode. In the refresh mode, the logs are automatically updated every 10 seconds. When the Job/Application Log is refreshed and there are new entries, the list of log entries is reset to display the new ones at the top of the list. By switching to the working mode, you can continue viewing the list without interruption. Select the AutoRefresh check box to activate the refresh mode and clear the check box to switch to a working mode. The Refresh link can be used at any time to update the logs. For notification purposes, a Loading... message appears when the logs are being refreshed.
Entries can be deleted from the APE Job Log when no longer needed on the Purge APE Log form.
You can cancel import jobs run on-demand that are marked for Manual Review and are in a Running or Postponed status. Any records that were processed to the PDM Pre-staging or PDM Staging tables before the cancellation was executed are purged. A Parameter Group Log report will not be generated for a cancelled job.
Click the Cancel Import Job button in the Job Log section on the Process Data Import Scheduler form. The APE Job Log will reflect the job has been cancelled when the process is complete. If no job is eligible for cancellation, the Cancel Import Job button is not be displayed.
The Scheduled Imports section provides a list of all active imports for each parameter group. The batch start and ends dates are provided along with the import date. Ascending/descending column sorting is also available; click a column heading to sort by the values in the column. The AutoRefresh option allows you to switch between a working mode and the scheduled imports refresh mode. In the refresh mode, the list is automatically updated every 10 seconds. It is possible to use the sorting options in the refresh mode; however, when the list is refreshed and there are new entries, all sorted results could be lost and the list will be reset to display new entries at the top of the list. By switching to the working mode, you can continue working without interruption. Select the AutoRefresh check box to activate the refresh mode and clear the check box to switch to a working mode. The Refresh link can be used at any time to update the list. For notification purposes, a Loading... message appears when the list is being refreshed. Click the Review link to view batch records on the Parameter Group Import Review form or click the Log Report link to access the Parameter Group Log report.
A separate validation process is available for parameter results that were entered into your Essential database using methods other than the PDM, e.g., via the data entry forms or the Data Importer tool. Parameter results from your Essential database are directly imported into the PDM Pre-staging table where the Frequency Expansion process is automatically initiated, rules are applied, and the PDM Staging table is populated with the results. If a process data value already exists in the Pre-staging table, the parameter result overwrites the existing value.
On the Validate Parameter Results form, specify the entity and parameter group with a start and end date/time that includes the monitoring date and time of the records to be imported. The date/time of the last import is provided for reference purposes. If the date range for the entity/parameter group includes data previously populated by the PDM, that data will also be imported for validation. An option to preview the records before initiating the validation process is available.
A message is displayed when the import process begins. If an email address has been entered on the logged-in user's contact and personnel record, an email message will be sent to the address with a link to the Parameter Group Import Review form after the import process is complete. If there is no email address, a message will not be generated. Track the progress of the import job in the Job Status section.
To validate parameter results
Click Tools > Advanced Tools > Validate Parameter Results in the Navigation Tree.
Select an Entity and Parameter Group from the list for each field.
Use the following guidelines to specify the date and time of the tag values to be imported:
Enter the Start Date and End Date of the date range that includes the timestamp of the records to be imported. The Calendar button is available to select a date.
Select a period of time by clicking one of the available options. The Start Date and End Date fields will be automatically completed based on the selection. For example, click Last Week and the previous week's Sunday date plus 12:00:00 AM (inclusive) are entered as the start date/time. The end date is the following Sunday date and 12:00:00 AM (exclusive).
When data has been imported for the entity and parameter group selected, the Last Import Date fields are automatically populated for reference purposes.
View the records before importing by clicking the Preview link in the Preview section.
Click Submit
for Validation.
A message is displayed when the import process begins. If an email
address has been entered on the logged-in user's contact and personnel
record, an email message will be sent to the address with a link to
the Parameter
Group Import Review form after the import process is complete.
If there is no email address, a message will not be generated.
When data is added to your Essential database via a scheduled import job, just the final result is logged in the Change Log. Since there is no opportunity to enter a Reason for Change for the imported data, a reason that describes the import origin is automatically added with the data as follows:
When imported via a PDM import job, the Reason for Change is "Scheduled PDM import job from PI Tag xxxx", where xxxx is the tag identifier
When imported using the PDM Validate Parameter Results process, the Reason for Change is "Scheduled PDM import job"
