Batch Load Data

The Data Importer tool provides batch load capabilities for importing template data. Upload multiple Excel template files or multiple delimited text files based on a user-defined schedule. When a batch load job runs, Data Importer uploads data files from the specified location at the scheduled time and initiates the validate and import function. Refer to Validate and Import Data for additional information about the process.

Use the Data Importer Schedule form to set up batch load jobs. Specify the location of the data and define the run schedule. Email messages that provide batch run results are sent to the specified owner of the scheduled job. Enter the owner's email address or select the owner from the list of contacts. The email address on the contact and personnel record will be used when a selection is made. A group email address can also be specified. Enter an email address for the group or select the group from the list of contact and personnel teams. The email address entered on the Contact and Personnel Team form for the group will be used when a selection is made. Once a schedule exists, the job it represents can be run on demand at any time.

The Data Importer Supervisor Console is available to monitor the status and outcome of batch load job runs.

For instruction on the Data Importer batch load process, click the following links or use the scroll bar to scan the page.

Establishing Batch Folders for Excel Template Files

To load multiple Excel template files, create a folder in a location that can be accessed by the Data Importer tool. Add one or more template files populated with the data to be uploaded. Multiple files for the same Template Type can be included in the folder, as well as files with different Template Types. Specify the folder name and its location when setting up your batch load schedule.

Configuring Files and Batch Folders for Delimited Text Data

Data from delimited text files can be imported using the Data Importer batch load process. Text file requirements, batch folder requirements, and configurable settings are described below.

Delimited Text File Requirements

• Semicolon- and comma-separated formats are supported. A semicolon is the default and can be changed to a comma in the configuration file.

• The delimiter must be included at the end of the file to signify the end of the data and the last item to be imported.

• The text file's field format (including display sequence) must match the column format in the Data Importer template the text file is associated with.

Batch Folder Requirements

A separate batch load folder must be created for each Template Type, e.g., Entity Information, Permits, Contacts and Personnel, etc. One Excel template file for the Template Type and one or more delimited text files that contain data matching the Template Type's data must be included in the batch folder. The Excel template file is renamed to Sample.xls (default) and contains no data. It is used to provide the template format for the text files to Data Importer. The default Excel template file name can be changed in the configuration file.

To establish batch folders to load delimited text files

1. Create a folder in a location that can be accessed by the Data Importer tool.

2. Create a template for a Template Type that corresponds with the data in your delimited text files. Refer to Create Data Templates for additional information.

3. Change the Excel template file name to Sample.xls (default) and copy it to the folder created in step 1.

4. Copy the delimited text files that contain data for the template type to the folder created in step 1.

5. Repeat steps 1 through 4 to create a folder for another Template Type.

6. Set up your batch load schedule

Configurable Settings

The default batch load-related settings listed below can be changed in the configuration file. Refer to Configurable Settings for additional information.

• FileDelimiter = ;

• BatchConfigFileName = Sample.xls

Scheduling Batch Load Runs

Note:  To set up a schedule, a batch folder must exist and the name and location must be specified on the Data Importer Schedule form. Refer to Establishing Batch Folders for Excel Template Files and Configuring Files and Batch Folders for Delimited Text Data for additional information.

Use the Data Importer Schedule form to establish a schedule that specifies when the batch load process runs. The particular hour and minute can be defined as well as the start and stop dates for the schedule. Once a schedule has been established, the job it represents can be run on-demand at any time. See To run a scheduled job on demand for instructions.

To schedule batch load runs

Tip:  There is just one Data Importer component, DIBatchLoadBO, so the Component Name field is automatically populated when the schedule type is selected or the schedule is saved.

1. Click Tools > Data Importer > Batch Load Scheduling in the Navigation Tree.
   
   The Data Importer Supervisor Console browse view is displayed.

2. Click the New button to open the Schedule form.

3. Enter the folder name, including the full path, in the Please Enter Server Folder Location field.

4. Enter the applicable email address in the Owner Email field and/or Group Email field, or click the Select From List check box and select a contact/team.
   
   The Component Status field is displayed below the email-related fields.

5. Enter a Schedule Description.
   
   The schedule description is a link to the Data Importer Schedule form on the Data Importer Supervisor Console.

6. Select a Schedule Type from the list.
   
   Based on the schedule type selected, any required scheduling options are automatically displayed.

7. Use the following guidelines to complete the run schedule:

1. • Enter a Start Date and time. The Calendar button is available to select the date.

   • Click the No End Date check box or enter an End Date and time. The Calendar button is available to select the date.

   • Select all applicable Days of the week for daily-related schedule types.

   • Select a Schedule Time from the list.

   • Enter a numerical value that represents the interval between runs in the Run Every field.

A schedule example for the Hourly type might be on Monday, Tuesday, Wednesday, Thursday, and Friday (Days), at 15 minutes after the hour (Schedule Time), every 8 hours (Run Every), beginning November 1, 2014 at 8:00 a.m (Start Date). The component would run at 8:15 a.m., 4:15 p.m., and 12:15 a.m. Monday through Friday starting November 1.

8. Click the Save button.

9. Monitor job progress and results using the Data Importer Supervisor Console.

To run a scheduled job on demand

1. Click Tools > Data Importer > Batch Load Scheduling in the Navigation Tree.
   
   The Data Importer Supervisor Console browse view is displayed.

2. Locate the schedule and click the Schedule Description link to open the Data Importer Schedule form.

3. Review the Component Status field. If the status is Error, click the Reset button; an on-demand job will not run for a schedule with an Error status.

4. Click the  Run on Demand? check box and click Save.

5. Monitor job progress and results using the Data Importer Supervisor Console.

Monitoring Batch Load Jobs

View each job's run status, when the last run took place, and the date of the next scheduled run, along with schedule details, on the Data Importer Supervisor Console. Field descriptions are provided below.

The Data Importer Supervisor Console should be reviewed on a regular basis to validate your data loading operations. Use the information provided to promptly resolve all error conditions. A Reset button is displayed below the Component Status field on the Data Importer Schedule form when an Error status applies to a scheduled job run. After error resolution, click this Reset button to change the status to Idle. The next scheduled job, as well as any on-demand jobs, will not run when the status is Error. You can run a scheduled job on demand to validate that all errors have been cleared.

Click Tools > Data Importer > Batch Load Scheduling in the Navigation Tree to access the Data Importer Supervisor Console.

Related topic