Introduction to EBP
The Heirloom Computing Elastic Batch Platform (EBP), or Job Entry Subsystem (JES), is a cloud-centric batch job execution and operating environment. EBP is a business process manager in the form of an IBM JES2 subsystem equivalent, complete with MVS Job Control Language (JCL) compatible interfaces, a RESTful Web Services and XML control interface. When coupled with the Heirloom Platform-as-a-Service (Heirloom PaaS) administration dashboard, a cloud based scheduler front-ends many cloud-resident EBP systems allowing jobs to be fanned out to a pool of cloud execution engines.
EBP can also operate on a private cloud, server farm, backoffice server or even a desktop environment. EBP operates under Java EE servers such as Oracle WebLogic, IBM WebSphere, Red Hat Wildfly (JBOSS) and Apache Tomcat on Window, UNIX or Linux based environments. It can also be started from the command line in "stand-alone mode" thanks to a built-in Eclipse Jetty servlet engine.
- Job Classes which define specific Heirloom PaaS resources that are available for batch programs
- Job Initiators which define a batch window and address concurrency issues
- Input Job Queue where submitted and scheduled batch jobs await execution
- Output Spool where status and held output datasets await retrieval and analysis following a job's execution
- Job Groups that define interrelationships among jobs that drive the scheduling of the job's execution
- Output Spoolers output classes and programs that can print, hold, or discard job output datasets
- Network Job Entry in which batch jobs can be submitted for execution via a RESTful Web services interface
- Remote Job Entry in which batch jobs can be submitted for execution via other jobs (internal reader)
- JCL Engine executing a batch job consisting of jobs, steps, data definition assignments, inline and cataloged procedures and control logic that execute Elastic COBOL or other executable programs
- JCS Engine executing a batch job consisting of IBM VSE style jobs, steps, assignment statements, extents (JCS mode JCL), with job class types and initiators for them
- JEC Job Scheduler supports the latest IBM Job Execution Control introduced in August, 2015 to manage parallel and dependent jobs
- Script Engine Batch jobs consisting of Shell scripts or Python scripts are accepted and managed by EBP in the same manner as JCL jobs
- Program Engine provides a mechanism for command line programs, such as COBOL or Java, can be managed by EBP so that installations that do not require the complexity of JCL can nonetheless run overnight background command line invoking Elastic COBOL
- IMS-DB supported in EBP and ETP
- Utility Programs such as flat-file, partitioned dataset and record-sequenced key dataset manipulations
- Cloud Scheduler allowing multiple EBP nodes (an "EBP-Plex") installations to be coordinated from a central point and work dispatched to servers on a timely basis using the Elastic Scheduling Platform (ESP)
Access the Elastic Batch Platform through the scheduling facilities of the Heirloom PaaS Dashboard at paas.heirloomcomputing.com.
IBM JCL Support
Central to the EBP system is the statement-oriented language used to control the execution of batch jobs and the standard dataset utilities that may be invoked (along with user programs) by jobs specifications conforming to IBM's Job Control Language, JCL. EBP also supports an IBM variant called JCS available on other systems as well as the definition of Job Groups through the use of Job Execution Control, JEC. The following documents define the syntax accepted by the subsystem:
- JCL Language Reference Manual
- Standard Utility Command Language
- Standard Utility Programmers Guide
- JCS Language Reference Manual
- JEC Language Reference Manual
Dataset Name Mapping
Datasets are central to JCL, they're used for program storage, cataloged procedures and data. In JCL datasets are referenced as 5-part names, each of 8 charcters each with an optional qualifier for partitioned dataset member or generation data group (GDG). Here are some examples of JCL datasets:
- datalib - location of data files referenced on DD cards within job steps
- systemlib - location of executable programs invoked through job step EXEC cards
- jcllib - location of cataloged procedures referenced in JCLLIB datasets names or INCLUDE statements, both system and user generated
- classlib - location of class libraries paths (.jar files or directories with .class files) that will be placed on the class path of invoked programs
- ebp.newdsndirectory - look to translate DSNs to directory trees first if yes and flat files if no (default no)
- ebp.newdsnlower - look at lower case directories and files first if yes (default no).
When JCL batch jobs invoke Job Steps they do so with cards of the form,
//STEPNAME EXEC PGM=MYPGM
EBP will first look for an executable jar MYPGM.JAR, mypgm.jar and its variants in the configured systemlib1..9 directories and the job's current working directory. Using the cataloged stored procedures HCECOBOL or HCEPLI will compile and leave jars in the current working directory if no other location is specified. If the executable jar is found EBP will execute the jar with the command "java -jar mypgm.jar." The executable jar is expected to have all classes (expanded JDBC drivers, class files) and a MANIFEST.MF that names the main-class of that jar. The java command cannot contain both the '-jar" and "-cp" flags, so no elements of the classpath1..9 configuration can be added to the java command line.
Unable to find a MYPGM.JAR, EBP next looks for a MYPGM.CLASS, mypgm.class, Mypgm.class files in the classpath1..9 configurations, the current working directory and the $CLASSPATH as seen by the servlet container executing EBP. It will also attempt to use the servlet container's classloader to load the program directly. If found it will execute the program with the command "java -cp classlib1:classlib2:...classlib9 mypgm". You can specify mixed case on the "PGM=MyPgm" parameter to make it easier for EBP to find a specific class.
Unable to find Java artifacts matching the "PGM=" parameter to the EXEC card, EBP will then look for executable programs such as C, C++ programs, operating system built-in commands and executable shell scripts in the systemlib1..9 configuration. If found, EBP will execute those directly without "-jar" or "-cp" operands.
The JCL batch job will ABEND if the program cannot be found using one of these methods. To see what command EBP uses to execute a job step set the JOB parameter MSGLEVEL=(4,4) and look in the MSGCLASS dataset after the job completes.
Tape Handling and Unit Specification
Tapes are regular files in EBP, but configuration settings may be used to allocate and manage them on potentially slower or cheaper media. Using VOL=SER=TP451,LABEL=(2,SL) would allocate a file named TP451.L0002 in one of the datalib directories. Using UNIT=TAPE (or DATACLAS=xxxx, MGMTCLAS=xxxx or STORCLAS=xxxx) on DISP=NEW datasets (or VOL=SER=serial numbered tapes) will look for datalib configurations that include that unit name or storage class. For example, UNIT=TAPE and datalib.9=/data/tape will restrict the dataset allocation to this directory. Using multiple datalibs, each referencing both storage class and unit names can segment allocation into certain areas. Consider the following datalibs used for allocating tape datasets:
would allow DD names to be allocated on the various classes
//DD1 DD DISP=NEW,DSN=HCI00.MY.DAT allocate /data/sysda/HCI00.MY.DAT
//DD7 DD DISP=NEW,UNIT=TAPE,VOL=SER=TPS431,LABEL=(1,SL) allocate /data/lwrclass/tape/3490/TPS431.L0001
//DD8 DD DISP=NEW,UNIT=3420,VOL=SER=TPS432,LABEL=(2,SL) allocate /data/lwrclass/tape/3420/TPS431.L0002
//DD9 DD DISP=NEW,STORCLAS=UPRCLASS allocate /data/uprclass/cart/3880/SYSEBP.TEMP2345.T1234567.DAT
Note: UNIT=SYSDA does not require that the datalib contain "sysda", but those that do are given preferential ordering ahead of others.
RETPD and EXPDT DD parameters and LABEL subparameters work as expected. Run the standard utility TMSCLEAN to scratch expired datasets and volumes. Using VOL=(PRIVATE,,,) or VOL=SCRATCH will allocate scratch serial numbered tapes. See Examples.
Unit specification also implies certain sizes that are calculated and checked during new dataset allocation. The underlying datalib device must have enough disk space to hold the given allocation using traditional IBM device type specifications even though the space is not allocated until the COBOL program requires it. If there is not enough space the datalib will be rejected and the next datalib will be examined for the required space. If no datalib contains enough free space then the job is aborted. SPACE required is calculated from the traditional UNIT sizes as shown in the following table,
* - calculated from block size specified in the dataset DCB
Examples of calculating SPACE and UNIT together,
//DD1 DD DISP=NEW,DSN=HCI00.MY1.DAT,UNIT=3350,SPACE=(TRK,(10,10)) allocate if 190KB available
//* on /data/nfsmount-3350-1 or -2
//DD2 DD DISP=NEW,DSN=HCI00.MY2.DAT,UNIT=SYSDA,SPACE=(CYL,(10,10)) allocate if 10MB available on any datalib
//DD3 DD DISP=NEW,DSN=HCI00.MY3.DAT,SPACE=(CYL,(100,7)) allocate if 100MB available on any datalib
//DD4 DD DISP=NEW,DCB=BLKSIZE=1000,SPACE=(BLK,(5,5)) allocate if 5000bytes available
JCL, Program and Scripts
EBP is capable of executing JCL jobs but also jobs consisting of Python or Shell script code and COBOL or Java programs directly, without the need for them to be contained in JCL. Specify the job type when defining the EBP job class. Jobs of all types can be submitted to EBP through the submit Web service or through the internal reader from other JCL jobs. Submit COBOL or Python jobs by submitting a Java archive (jar) file. Where possible, EBP will look for hints in a "job card" on the first comment line of a script for information regarding classification. For example,
# //PY JOB TIME=10,MSGLEVEL=(1,1),MSGCLASS=Y,CLASS=P
# Python script
Job Class Resource Limits
Job classes are defined to EBP through the define Web service. A number of resource limits may be specified on the class:
- Maximum CPU execution time
- Maximum elapsed (wall-clock) time
- Maximum memory (region) used for each step
- Total file sizes created during step execution
- Total network transmission allowed during the job
The TIME keyword on the JOB card will help EBP determine its class if not specified explicitly on the JOB card. Elapsed time restrictions are checked regardless, but maximum CPU and other resource limits require checking by a command outside the Java Virtual Machine environment. Set the ebpstartcommand configuration parameter to a command that can interpret these configuration settings and start an arbitrary command under resource control. Although not initially configured (i.e., jobs will not ABEND if their steps take too long), ebpstartcommand may be set to any of the available commands in the bin directory within the webapps folder containing EBP. For example, on Linux 64-bit systems configure ebpstartcommand=bin/ebpstart_linuxx86_64.
Checkpoint / Restart
EBP supports the notion of checkpoint / restart initiated through Web services. The checkpoint Web service will suspend a running job at its currently executing step and place it in a suspended state. Dataset locks are kept, the temporary spool (working directory of a job), its output spool (held output datasets) and the input spool (input job) are retained.
Suspended jobs have their intermediate MSG and ouptut spool datasets available for listing.
A subsequent restart Web service for a job ID will place a suspended job into the queued state where, when an initiator of that class becomes available, it will resume execution at the beginning of the step that was executing at the time of suspension. The restarted will rescan the JCL and reprocess any symbolic parameters that may be in effect at the time of resumption. The MSG dataset will reflect both the job execution before and after the restart.
On some systems (e.g., Linux) it is possible to not only restart a job at the beginning of the a step but restart the program that was executing at the point of its last checkpoint. Such programs restart in the middle of their processing. Support relies on the availability of an external checkpoint/restart facility, such as the Distributed Multi-Threaded CheckPointing (DMTCP). Such process checkpointing requires additional EBP configurations to take advantage of these features and these are done on a per-job-class basis. Those classes whose jobs are eligible for DMTCP checkpoint/restart have the checkpointcommand-classname and restartcommand-classname configurations set to the command sequence needed to carry out process-level checkpoint (which produces a ckpt file at regular intervals) and later restart. For example, if Job Class R is defined for restartable jobs, the two configurations might be:
checkpointcommand-R: rm ckpt_*;/usr/local/bin/dmtcp_launch --ckpt-open-files
For each step in class R jobs EBP erases all prior checkpoint files created by DMTCP (ckpt_*.dmtcp) and then the command to invoke the step's program (e.g., a standard utility or COBOL application) is pre-pended with the dmtcp_launch command. And if a job step is coming out of a suspended mode, the dmtcp_restart command will be executed instead of the restarted job step command and given the corresponding ckpt_*.dmtcp file.
A DMTCP coordinator must be started and listening for launch commands and a checkpoint interval must be in place to create checkpoint files every so often. Start the coordinator with a command such as:
/usr/local/bin/dmtcp_coordinator -i 20
There are other dmtcp_launch and dmtcp_coordinator options that control other checkpointing parameters. See DMTCP documentation for more info.
EBP supports output spooler classes named A through Z, 0 through 9 as well as DEFAULT or CONSOLE such that SYSOUT DD assignments and MSGCLASS designations can define the corresponding output datasets (hold) be held in the output spool for later retrieval, (lpr) designate automatic printing to defined printers, (log) redirect to Java log4j logging systems, (cat) sent to arbitrary spool programs, (none) or discarded entirely. Define the output classes through the EBP Configuration service with the following values:
|DEFAULT||hold||Hold output in output spool|
|CONSOLE||none||EBP console messages|
|L||log||Send to Java log4j logging system|
|P||lpr||Send to the CUPS printing system|
|Y||hold||Hold output in output spool|
|All others||hold||Same as DEFAULT|
Set additional parameters to lpr or define new spooling programs by setting its output class descriptors to a command name optional parameters, including redirection parameters. For example, the following:
output.P=lpr -P node22/printer44 -C EBP -T Job-Output
- output class P configured to send to a remote printer,
- output class S is configured to append all output to a file,
- output class 4 is configured to a Java Logger (e.g., log4j) named EBP-4log, and the
- output class CONSOLE configuration is such that EBP console messages (job start/stop, errors, warnings) are written to a files (changed daily with a time stamp) into the output spool directory
EBP integrates with the supporting J2EE Application Server or Servlet Container to provide three layers of security:
- Transport Level Security - ensuring data between EBP and the clients is secure
- Authentication - validating who can use EBP
- Authorization - indicating who can use what services against which objects
See Operating EBP in a Secure Environment for detailed information and examples for EBP, container and security subsystem configuration.
Once the EBP and container has been set up to provide the three layers of security you add users to the container's security environment and add them to roles. EBP classifies operations in terms of roles, users and administrators. By default these role names are EBP-user and EBP-admin but they can be changed to group names you may have defined in your mainframe security system, should you wish to continue to use these. Users submit jobs, administrators change configuration. Note that administrators don't ordinarily have the rights for user operations -- the ability to submit jobs. But, you can define users that take on multiple roles in most server environments. This allows you to define the same groupings that are common in mainframe resource access control environments such as RACF, TopSecret or ACF2. In many environments you can configure EBP to use the same RACF system for authentication and authorization using its lightweight directory access protocol (LDAP) facility. Or, you can extract users and groups from the mainframe system and import them into an interchange format (LDIF) to be imported into local directory systems.
When configured appropriately the EBP Web services (submit, configure, purge, etc.) are protected by the security system and jobs will run under authenticated users. Generally, the EBP index.html home page and the sysoper.html System Operators console and their components can be allowed to be accessed by all users. But, your installation may set up other logon pages that set the security contexts before any part of EBP becomes visible.
When security is turned on EBP will enforce a cascading set of ownership of jobs. The authenticated user (in the EBP-user role) will become the owner of jobs submitted by the submit Web service. Listing the status of jobs will include that user name in the output, visible on the System Operators Console. When JCL jobs are run by EBP the &SYSUID symbolic parameter is set to the upper-case representation of the user (&sysuid is the lower case representation). Only that same user or one with both an EBP-user and EBP-admin role may cancel, purge, checkpoint or restart that job. Further, any jobs submitted to the internal reader either in JCL or through the COBOL API will take on the same user as that which submitted the original job.
The users authenticated by the EBP container will override the USER keyword that may be present on the JOB card. the USER and PASSWORD job card keywords are not used to authenticate against, for example, OpenLDAP. However, the USER and PASSWORD entry on the job card will still be used to authenticate access to resources such as with database connections performed by DSNTIAD and IKJEFT01. Note that EBP cannot learn LDAP passwords in order to pass them to database connections. However, JNDI naming and JDBC datasources can be established to allow the container to validate database connections used by job steps.
IBM IMS DB Support
IMS-DB is supported in EBP with utilities such as DBDGEN and PCBGEN as well as the ability to run IMS DB Batch applications with the DFSRRC00 utility.
IBM JCS Support
The IBM VSE Job Control Subsystem and its flavor of JCL called JCS is fully supported. Define a JCS job class and start initiators. Then submit JCS decks with the SUBMIT web service.
The following defines the syntax accepted by the subsystem:
IBM JEC Support
IBM introduced Job Execution Control with z/OS JES2 Version 2, Release 2 (August, 2015). JEC allows the definition of interrelationships among jobs. JEC is fully supported within EBP. Define a JEC job class and start initiators in order to SUBMIT JEC definitions of JOBGROUPs.
Also introduced is the new SCHEDULE JCL statement. Jobs can be given instructions on when to start (SCHEDULE HOLDUNTL parameter), indicate that they must begin by a certain time and date (SCHEDULE STARTBY parameter), that they should be initiated at the same time as another job (SCHEDULE WITH parameter) and conform to the more advanced scheduling criteria defined in a JEC JOBGROUP (SCHEDULE JOBGROUP parameter).
The following defines the syntax accepted by the subsystem:
EBP Web Services
The dashboard controls EBP through the Web services that define an HTTP Web URL with one or more parameters for each service and returning an XML document describing what was done. Report or held output datasets from jobs in the output queue are also retrieved through Web services. See EBP Web Services for a complete list or one of the specific services listed below.
- define - new job class
- start - job initiators of a specific class
- submit - jobs to the input queue to be executed by initiators
- list - list job classes, job initiators, input jobs, job outputs, output datasets
- cancel - a running or queued job
- purge - output of a running or finished job
- quiesce - suspend job initiation, complete running jobs
- stop - running job initiators, halting job execution
- undefine - existing job classes, stopping all initiators of that class
- checkpoint - a job by saving the current state (at the most recent completed job step)
- restart - a previously checkpointed (after last successful job step) or held job (at start)
- config - the Elastic Batch Platform
A typical sequence of Web service invocation to define and start jobs is shown below.
- Define A, JCL
- Start A
- Submit JCL
- List jobid
- List outputid
- List outputid, , MSG
- List outputid, step, ddname
You can purge old output of completed jobs, cancel running or queued jobs and adjust configuration with other services.
EBP Error Messages
Error messages generated by the EBP subsystem fall into a number of categories:
- Errors during startup.shutdown
- Errors issueing Web service requests
- Errors scanning input job datasets
- Errors executing JCL
- Errors occuring during execution of standard utilities
- Errors as a result of executing scripts, programs or procedures referenced by input jobs
The list of all error messages may be found here:
See the following document for a list of sample JCL and script job inputs.
EBP System Operators Console
See the following document for a list of sample JCL and script job inputs.
EBP is distributed as a Java-based Web Application Archive (war). The Announcements forum contains the latest information on new features, functions and fixes as well as the download location. EBP is distributed as a zip file ebp-vv.vv.vv.zip containing two files:
- ebp##vv.vv.vv.war -- the Web Application Archive used with Java Application Servers and Java Servlet Containers
- ebp-standalone-vv.vv.vv.jar -- the Java Application Archive used as a stand-alone version of EBP without the need of other software
The Elastic Batch Platform operates under a Java EE server or the simpler servlet management system such as Apache Tomcat. Install Tomcat on Windows, UNIX, Linux (Heirloom PaaS managed clouds have EBP pre-installed and configured), then deploy the EBP system (ebp##vv.vv.vv.war) by using the server's administration interface or copying the deployment file into its webapps directory. See the Announcements forum for the latest EBP news, downloads and updated installation instructions. See FAQs Wildfly or Weblogic for special installation instructions for those systems.
The EBP subsystem is a licensed program product of Heirloom Computing, Inc. Ensure that the registration file elasticcobol.properties or licensing.properties for your account is installed in either (1) the same webapps/ebp folder where EBP has been deployed, or (2) the home directory of the user under which the application server operates (e.g. /home/tomcat8 or /var/lib/tomcat8 on Linux Tomcat systems operating under that user id). Alternatively, set the environment variable ebppropdir to the directory containing the license prior to starting up EBP.
Start your application server or servlet management server prescribed by the manufacturers documentation. If your EBP is operational it will begin accepting RESTful Web services from any client. Although not the primary interface to EBP, the default index page is acceptable for testing basic operations. If your server is listening on port 8080 you can access the EBP index page with:
Jobs, classes and started iniatiators persist across restart of the application server. If the app server stops while a job is running the job will be terminated and requeued to begin again after the server restarts (depending on your job restart parameters). A redeploy of EBP into the app server will reset the state as will the quiesce Web service call with the resetindication. All jobs will be flushed from the queue.
The EBP default configuration is usually fine for running the subsystem but you can change that either by modifying the ebp.properties file deployed with the subsystem under the server's webapps/ebp folder or by issuing the config Web service. Settings persist through application server restarts unless a redeploy overwrites them.
See the following document for EBP configuration settings:
See your Java Application Server (Apache Geronimo, Oracle Weblogic, IBM Websphere, Red Hat JBOSS) documentation for installation and configuration of WAR files. Most have a Web interface with a "Deploy" operation or simply allow you to drop war files into a "deploy" or "webapps" folder. EBP begins operating immediately upon start-up of the container. Depending on the configuration, you should be able to access the RESTful Web Services control point of EBP from a browser with the Web address:
Alternatively, you can run EBP without the need of a container. Start EBP in stand-alone mode with the command:
java -jar ebp-standalone-vv.vv.vv.jar
Standalone EBP has simple command-line options to set important options which can be displayed with the --help option:
java -jar ebp-standalone-vv.vv.vv.jar --help
Heirloom Computing Elastic Batch Platform
Usage: java -jar ebp.jar options...
--port 8080 - defines the Web service port
--home <jar location> - EBP Home directory
--logs logs - log directory
--realm realm.properties - login properties file
--context / - URL path
--help - this message
Use --port option to set the HTTP listening port. The --home option sets the home directory, if different than the location of the jar file itself. The --logs option can change the location of the message logs to other than the home directory. The --context option will change the URL path from the default of "/" (the default when running under Application Servers is typically "/ebp").
java -jar ebp-standalone-vv.vv.vv.jar --port 80 --context /ebp --home /usr/share/EBP
Elastic Batch Platform will enter a demonstration mode if no valid license is installed, it is invalid or has expired. EBP is pre-configured with a single Job Class (A) and an Initiator for that class. Other job classes and initiators are restricted from being configured as is EBP-Plex mode (multi-node scaling) and interaction with the Elastic Scheduling Platform (ESP). Web Services define/undefine/start/stop are restricted. Job state (input, output spools) do not persist between EBP restarts in Demo Mode. All other EBP features and functions are supported without restriction.
Backup and Recovery
- ebp.properties - configuration information (classlib, datalib settings)
- ebp_object_store1.xml - state information (defined job classes, started initiators, jobs and outputs)
- ebp_object_store2.xml - prior state information (defined job classes, started initiators, jobs and outputs)
- job_input_queue/ - jobs submitted but not yet executed
- job_output_queue/ - jobs finished executing
- temp_queue/ - jobs executing
To take a quick backup of configuration settings issue and save the XML retrieved from the web services:
To restore from a backup, stop EBP and replace the files/directories listed above and restart EBP. To recover from a failure requiring re-installation, use the config service XML file saved above to re-issue configuration from the home page or issue individual web service requests with one or more configuration settings:
If you with to preserve both configuration and state information between upgrades of EBP, zip or tar archive the files and folders mentioned above, uninstall EBP, install a new EBP and then use the Server's ability to "Stop" the EBP service (or shutdown the entire Java Servlet Engine, Jetty, or Java application server), unzip the archive into the new EBP's home directory, and restart the EBP service. Classes, initiators, input and output jobs will be visible to the new EBP.