ESP Task and Job Scheduler

Introduction

This is part of the Heirloom Computing Elastic Batch Platform forum on paas.heirloomcomputing.com.

The Job Entry Subsystem operates on a cloud instance (possibly controlled through the Heirloom Dashboard at paas.heirloomcomputing.com), Windows, Linux or UNIX server within a datacenter or a Windows or Linux desktop computer.  The EBP system is controlled directly through its user interface, configuration parameters and System Operator Console.  However, one or more EBP systems can also be controlled through the Heirloom Computing Cloud-based Task and Job Scheduler.  Whereas EBP is responsible for controlling job entry, job execution and output dataset management the Elastic Scheduling Platform (ESP) duties involve scheduling tasks to be be run (which include JCL and other jobs handled by EBP) and collects the output reports in a central repository.  It also maintains a clear history of when jobs, tasks and their output were run and the results of those runs.  ESP is cloud-based, being an extension of the Heirloom Portal.  Job results are available even if the EBP system(s) used to run the jobs are idle or powered off.  If cloud resources need to be available in order for a job to run within EBP the scheduler will see to it instance(s) are powered on before jobs are shipped to them.  As a so-called batch window closes the scheduler will see to it that unneeded resources are powered off.  As such ESP is an integral part of IT's cloud solution.

Note:  The cloud scheduler is available to schedule EBP on Heirloom Windows and Linux instances and EBP is included in the subscription to these products.  You can also download EBP here to run on-premise, other cloud-based application servers or in an Heirloom or Elastic COBOL development instances.  A separate subscription is required for EBP in these environments although ESP is available to any registered Heirloom user.

Overview Video

 

 

If video does not display in your browser, right-click, download and play in VLC, Windows Media Player or other video player or click on https://www.youtube.com/watch?v=mANO2BULOu4 to bring this up in a separate window.

Portal

 

The Heirlooom Dashboard controls ESP  There are chiefly 4 main components that control the scheduler:

1. Application or Application Data Volumes that represent instances of Job Entry Subsystems on which to run jobs
2. Job Classes used to define resources and parallel processing that EBP can manage
3. Jobs that consist of JCL or other specifications of what to do
4. Tasks that indicate when jobs are run, consolidate the history of past runs, and their output reports if any

The screen capture of the Heirloom Dashboard shows these 4 components:

 

To display the ESP portlet in the Heirloom Dashboard bring up the primary menu by clicking on the header.  Select Show Data Volumes to display the list of volumes associated with your subscription or, if you're a group administrator for your organization, all of the volumes.  Here a Data Volume corresponds to an Application the scheduler will manage.  The other three portlets are available as sub-menu items to the primary options menu's Scheduler item.  Select Scheduler > EBP Classes, Scheduler > Jobs or Scheduler > Tasks to show the windows.  History for both Jobs and Tasks are available from the Jobs and Tasks portlet windows.  Select the row corresponding to a job in the Jobs Portlet and right-click to Show Job History command -- the Job History calendar and the Job History portlet will be shown.  Likewise, select a task in the Tasks Portlet and right click to Show Task History.

 

Data Volumes Portlet

 

Other uses of the Data Volumes Portlet is described in Data Volume Portlet section of the Heirloom Dashboard.  Here we're simply referring to the portlet view showing the Name column that describes the volume and application name.  Since COBOL and other applications are deployed to a particular running cloud instance designated by the application name, the same name will be used when defining tasks.  Tasks run on specific applications designated by their name but also refer to the JCL job that can run on many application instances.  Although data volumes are designated for all types of Heirloom Computing product subscriptions, the Heirloom (Windows) for Amazon EC2 and Heirloom (Linux) for Amazon EC2 are the ones that support the Elastic Batch Platform by default.  Therefore, you can launch or schedule the launching of these types of application instances through the ESP Scheduler.

ESP Classes Portlet

 

 

This portlet describe job execution classes that will operate on a EBP instance.  Define the job classes to run across multiple applications a single time in this portlet.  Click the Add button to define a new job execution class or select one and click Remove to undefine it.   Double-click on a line to edit an existing job class. Job class definitions include the same resource definitions (job type, maximum CPU time, etc). ESP will use the EBP commands DEFINE and UNDEFINE on the EBP instances when called upon as part of a task.

You can duplicate class definitions by selecting and right-clicking on a EBP class in the window and selecting Create Copy.  You must edit (double-click) the EBP class to assign a new class ID.

When clicking Add you will be presented with this job execution class dialog box:

 

Define the job execution class with the following information:

• Class Name -- a simple letter A through Z or number 0 through 9.  EBP will only use the first letter if more are entered.
• Class Type - a job class can run jobs of only one type, select from the list the possible job class types:
• JCL - the execution class runs JCL type jobs
• COBOL - the execution class will run "command line" COBOL applications
• PYTHON - run Python interpreter on scripts
• JAVA - the class type consists only of Java applications
• Execution Time - The maximum amount of CPU time (in CPU seconds) jobs of this class are expected to take.  Recall that EBP will abort jobs that exceed this or other resource requirements.  Most COBOL applications and JCL decks utilize a single CPU core but if using the Elastic COBOL mutlithreading capabilities this is the total CPU utilization used by all cores of the job.  A maximum must be specified (0 is not allowed for this resource).
• Wall Time - The maximum amount of "wall clock" or "elapsed" time (in seconds) jobs of this class should run in.  Use 0 for no restriction.
• File Size - The maximum amount of file size (in K bytes). 0 for no restriction.
• Network Size - The maximum amount of data (in K bytes) sent to and from the instance, if it is an network application.

Jobs Portlet

 

 

Define a new Job by clicking on the Add button or select one from the list and click Removeto delete it.   Double-click on a line to edit an existing job.  

Select a job and right-click the mouse to Show Job History for the job in a separate portlet window.

When defining a new job the following dialog will appear:

Enter the following information to define a job:

• Job Name - the simple name of the job you are defining.  This may be the same name as the JCL job name or it may refer to the application and job type
• Script Type -  choose between one of the possible job types.  EBP can handle jobs of the following types:
• JCL - the job is a JCL card deck referring to programs and built-in utilities and PROCs that are resident on the application data volume
• COBOL - a simple COBOL application (jar or class files) that has been deployed to the application data volume.  Use this selection to run simple "command lines" referencing your COBOL applications.
• PYTHON - Python is a powerful multiple platform (Windows, Linux, UNIX) scripting language that can carry out utility and other maintenance operations outside the scope of JCL batch jobs on your instance
• JAVA - like COBOL, JAVA will invoke a Java application (jar or class file) that has been deployed to your application data volume.
• Enabled - by default all new jobs are enabled by leaving this box checked.  You can quickly keep any jobs of this class from executing on any EBP system, even if they are scheduled to run at a specific time, by unchecking this box.
• Use Script On Persistent Drive - Check this box if you won't be uploading any JCL decks, applications or scripts but will reference those that have already been deployed to your instance.
• Script - Click the Browse... button and select a local file representing the JCL deck (.jcl suffix) or application (.class, .jar suffix) or script (.py) that you wish to save with the job definition.  It will be made available to instances of the Job Entry Subsystem when needed. 

Tasks Portlet

 

Whereas the Job Portlet defines what is to be done, the Task Portlet schedules when they or other operations should be performed.  Define tasks that reference jobs and job classes or simply define other attributes such as launching new cloud-based instances or starting initiators for a class.  Use the Add button to create new tasks or select and existing item and click the Remove button to delete it.  Double-click on a line to edit an existing task.

Select a task and right-click the mouse to Show Task History for the task in a separate portlet window.

When adding a new task a dialog box appears with two tabs that define the task attributes:

 

Specify either an exact time the job should run (once) by specifying a specific date or a schedule of regular iterations of when the job should run. Fill out the following fields on the Schedule tab:

• Run On A Specific Date -- click the calendar button to select a date the task should invoke the job.
• Run Every Week On The Following Days -- for daily or weekly jobs, click this radio button and then select the days of the week the job should run.
• Run Every Month On -- choose this radio button selection and then enter a specific day of the month this job should run.  Jobs will not run on days that don't exist in that month, for example September 31st.
• Run Every Year On -- choose this radio selection and enter the day month and day for this yearly job
• Run At The Following Time -- choose a specific time of the day the task should run the job.

Note:  All fields are relative to the timezone set for the device on which the browser is running.  Selecting a time of 2am would mean 2am PST (6pm the previous evening UTC time) if you're setting up the task information in Pacific Standard Time.

The Task Info tab relates to the predefined job or operation that should be performed for this task.  Fill in the following information:

• Task Name - a name for your task that will appear in the Task Portlet.
• Task Type - the type of task that should be performed selected from the list of:
• Run Job - run the specified job on the application's Job Entry Subsystem at the scheduled time.  This is the most common setting.
• Initiator - change the number of initiators of a specific job class on EBP at this time.  Initiators define parallellism of jobs in a class.  Specifying 2 class B jobs indicates that EBP will start (or shutdown) the number of inititors to reach this number at this time.
• Terminate - terminate the cloud-based instance or quiesce an on-premise EBP system at the scheduled time.
• Enabled - by default all new tasks are enabled by leaving this box checked.  You can quickly prevent a task from running (without having to change the schedule info) by unchecking this box.

Job History Portlet

 

The Job History Portlet appears when you Show Job History from the Jobs Portlet.  The table shows the history in any one month.  Advance or backtrack through the history by clicking on the Prior or Next arrows in the header.  The lines show a combined history of all runs of the job.  You may see Start, Stop, Terminate or other Information lines posted about the execution.  The Stop history lines will include the final job termination condition code for the job.

Task History Portlet

 

 

There are two stages to the Task History Portlet.  The first stage is brought about by selecting Show Task History from the Tasks Portlet and includes a calendar showing (in bold) when the task was run.  Tasks can run multiple times during the day or be restarted automatically by EBP upon failure based on JCL operation.  Clicking on a date within the calendar will bring up a drop-down list of tasks run on that day.  It shows the time and the EBP Job ID number assigned to the job when it was submitted.  Select one to bring up the second stage of the Task History Portlet.

 

Similar to the Job History, the Task History list will show the activity for the task while it executed.  Typically a Start and a Stop message will appear in the list and the Activity will show the status recorded by EBP when the job was submitted for execution and when it completed.  The Output column for a job contains a link that displays a list of all job output for the job.  In the case of JCL jobs this is any DD Name with a SYSOUT specification.  Select any "output dataset" in the list to show the report.  The dataset name includes the job number, the step number and the DD name.  Datasets ending in __msg.txt are for the JCL messages themselves, as shown below:

 

 

Summary

 

For more information about the Heirloom Dashboard see the collection of Heirloom Dashboard articles on this site.  For more information on the Job Entry Subsystem and Job Control Language that can run COBOL batch jobs see the other EBP and JCL articles in this same forum.