To create our new Java batch application based on WildFly image, run the command:

oc new-app wildfly~https://github.com/jberet/intro-jberet.git

And the output should look something like this:

--> Found image 18929ed (5 weeks old) in image stream "openshift/wildfly" under tag "10.1" for "wildfly"

    WildFly 10.1.0.Final
    --------------------
    Platform for building and running JEE applications on WildFly 10.1.0.Final

    Tags: builder, wildfly, wildfly10

    * A source build using source code from https://github.com/jberet/intro-jberet.git will be created
      * The resulting image will be pushed to image stream "intro-jberet:latest"
      * Use 'start-build' to trigger a new build
    * This image will be deployed in deployment config "intro-jberet"
    * Port 8080/tcp will be load balanced by service "intro-jberet"
      * Other containers can access this service through the hostname "intro-jberet"

--> Creating resources ...
    imagestream "intro-jberet" created
    buildconfig "intro-jberet" created
    deploymentconfig "intro-jberet" created
    service "intro-jberet" created
--> Success
    Build scheduled, use 'oc logs -f bc/intro-jberet' to track its progress.
    Run 'oc status' to view your app.

The build of our application is now scheduled, and it will take up to 3 minutes before it's ready. To watch its status and progress, run the command:

oc rollout status dc/intro-jberet

You will see output like the following as the application is being built and deployed:

Deployment config "intro-jberet" waiting on image update
Waiting for latest deployment config spec to be observed by the controller loop...
Waiting for rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for rollout to finish: 0 of 1 updated replicas are available...
Waiting for latest deployment config spec to be observed by the controller loop...
replication controller "intro-jberet-1" successfully rolled out

This command will exit once the application is ready. Now is a great time to switch to the Dashboard and explore features in OpenShift web console including application configuration, details, logs, monitoring, resource management, etc.

To expose intro-jberet application to external clients, run the command:

oc expose svc intro-jberet

And you will see output like this:

route "intro-jberet" exposed

Once the build completes, you can visit the application test URL below to verify:

http://intro-jberet-jberet-lab.[[HOST_SUBDOMAIN]]-80-[[KATACODA_HOST]].environments.katacoda.com/

This will take you to WildFly landing page, which shows the application server is up and running. If the application is not ready yet, you will see messages like the following, which means more time is needed to build and deploy the application:

Application is not available

The application is currently not serving requests at this endpoint. It may not have been started or is still starting.

intro-jberet app uses PostgreSQL database for storing output data. To create a database which you can then connect to, run the command:

oc new-app postgresql-ephemeral --name postgresql --param POSTGRESQL_USER=jberet --param POSTGRESQL_PASSWORD=jberet

This will start up an instance of a PostgreSQL database, with the following info:

--> Deploying template "openshift/postgresql-ephemeral" to project jberet-lab

     PostgreSQL (Ephemeral)
     ---------
     PostgreSQL database service, without persistent storage. For more information about using this template, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/blob/master/9.5.

     WARNING: Any data stored will be lost upon pod destruction. Only use this template for testing

     The following service(s) have been created in your project: postgresql.

            Username: jberet
            Password: jberet
       Database Name: sampledb
      Connection URL: postgresql://postgresql:5432/

     For more information about using this template, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/blob/master/9.5.

     * With parameters:
        * Memory Limit=512Mi
        * Namespace=openshift
        * Database Service Name=postgresql
        * PostgreSQL Connection Username=jberet
        * PostgreSQL Connection Password=jberet
        * PostgreSQL Database Name=sampledb
        * Version of PostgreSQL Image=9.5

--> Creating resources ...
    secret "postgresql" created
    service "postgresql" created
    deploymentconfig "postgresql" created
--> Success
    Run 'oc status' to view your app.

Although a database would normally be paired with a persistent volume, we only want to demonstrate how to access the database in this course. The database instance we create here, will therefore only store the database in the filesystem local to the container. This means that if the database was restarted, any changes would be lost. When you deploy a database to be used with your own applications, you would want to look at using persistent volumes.

To monitor progress as the database is deployed and made ready, run the command:

oc rollout status dc/postgresql

This command will exit once the database is ready to be used:

    Waiting for rollout to finish: 0 of 1 updated replicas are available...
    Waiting for latest deployment config spec to be observed by the controller loop...
    replication controller "postgresql-1" successfully rolled out

Our sample application intro-jberet demonstrates basic concepts and programming model in batch processing, such as job, step (batchlet- and chunk-type steps), batch properties, item reader and writer.

The job structure and flow are defined in the job XML file csv2db. It contains 2 steps:

• csv2db.step1: a batchlet-type step, which performs a specific task: initializing table MOVIES.

<step id="csv2db.step1" next="csv2db.step2">
  <batchlet ref="jdbcBatchlet">
    <properties>
      <property name="url" value=
        "jdbc:postgresql://#{jobParameters['db.host']}?:postgresql;/#{jobParameters['db.name']}?:sampledb;"/>

        <property name="user" value=
          "#{jobParameters['db.user']}?:jberet;"/>

        <property name="password" value=
          "#{jobParameters['db.password']}?:jberet;"/>

        <property name="sqls" value="
          CREATE TABLE IF NOT EXISTS MOVIES (
          rank INTEGER NOT NULL PRIMARY KEY,
          tit  VARCHAR(128),
          grs  NUMERIC(12, 3),
          opn  DATE);
          DELETE FROM MOVIES"/>
    </properties>
  </batchlet>
</step>

• csv2db.step2: a chunk-type step, which reads, processes and writes data in chunks, and repeats the read-process-write cycle till the input is exhausted.

<!-- read data from online CSV resource and output
  to db, following chunk processing pattern -->

<step id="csv2db.step2">
  <chunk>
    <reader ref="csvItemReader">
      <properties>
      <property name="resource" value=
        "https://raw.githubusercontent.com/jberet/jsr352/master/jberet-support/src/test/resources/movies-2012.csv"/>

      <property name="beanType" value=
        "java.util.Map"/>

      <property name="nameMapping" value=
        "rank,tit,grs,opn"/>

      <property name="cellProcessors" value= 
        "ParseInt; 
        NotNull, StrMinMax(1, 100); 
        DMinMax(1000000, 1000000000); 
        ParseDate('yyyy-MM-dd')"/>
      </properties>
    </reader>

    <!-- processor is optional and is not used -->

    <writer ref="jdbcItemWriter">
      <properties>
      <!-- url, user & password properties are the same 
        as in csv2db.step1, and are now shown here -->

      <property name="sql" value=
        "insert into MOVIES (rank,tit,grs,opn) 
                     VALUES (?, ?, ?, ?)"/>

      <property name="parameterNames" value=
        "rank,tit,grs,opn"/>

      <property name="parameterTypes" value=
        "Int,String,Double,Date"/>

      <property name="beanType" value=
        "java.util.Map"/>
      </properties>
    </writer>
  </chunk>
</step>

In csv2db job, 3 batch artifacts from jberet-support library are used to implement the processing logic:

• jdbcBatchlet: executes SQL statements against target db.
• csvItemReader: reads from CSV input source, one line at a time.
• jdbcItemWriter: writes accumulated data in a chunk to target db.

As you can see from the above job definition, batch artifacts can be configured with batch properties to customize their behavior. Furthermore, batch properties can reference system properties, other job properties, and job parameters. Default value can also be supplied as part of batch property expression.

Now we are ready to run the batch application. We will use curl command line tool to invoke REST API to perform various batch processing operations. The JSON output is formatted with python -m json.tool

To start the job named csv2db:

curl -s -X POST -H 'Content-Type:application/json' "http://intro-jberet-jberet-lab.[[HOST_SUBDOMAIN]]-80-[[KATACODA_HOST]].environments.katacoda.com/intro-jberet/api/jobs/csv2db/start" | python -m json.tool

{
   "startTime":null,
   "endTime":null,
   "batchStatus":"STARTING",
   "exitStatus":null,
   "executionId":1,
   "href":"http://intro-jberet-jberet-lab.2886795305-80-ollie02.environments.katacoda.com/intro-jberet/api/jobexecutions/1",
   "createTime":1506284588752,
   "lastUpdatedTime":1506284588752,
   "jobParameters":null,
   "jobName":"csv2db",
   "jobInstanceId":1
}

Our application uses the following database parameters, as specified when we deployed PostgreSQL database:

• db.host: postgresql
• db.name: sampledb
• db.user: jberet
• db.password: jberet

If you've configured different values when creating PostgreSQL, you will need to include them as query parameters in test URL.

The href value in the curl command output indicates the address for the newly-created job execution resource. To get its details and status, run the following command:

curl -s http://intro-jberet-jberet-lab.[[HOST_SUBDOMAIN]]-80-[[KATACODA_HOST]].environments.katacoda.com/intro-jberet/api/jobexecutions/1 | python -m json.tool

The output is the same as above, except that the job has completed and its batch status is updated to COMPLETED

To get all step executions belonging to this job execution:

curl -s http://intro-jberet-jberet-lab.[[HOST_SUBDOMAIN]]-80-[[KATACODA_HOST]].environments.katacoda.com/intro-jberet/api/jobexecutions/1/stepexecutions/ | python -m json.tool

The output should include 2 step executions (details omitted here).

To get details for the 2nd step execution, which performs the reading and writing of batch data:

curl -s http://intro-jberet-jberet-lab.[[HOST_SUBDOMAIN]]-80-[[KATACODA_HOST]].environments.katacoda.com/intro-jberet/api/jobexecutions/1/stepexecutions/2 | python -m json.tool

{
   "startTime":1506284589019,
   "endTime":1506284591953,
   "batchStatus":"COMPLETED",
   "exitStatus":"COMPLETED",
   "stepExecutionId":2,
   "stepName":"csv2db.step2",
   "metrics":[
      {
         "type":"WRITE_COUNT",
         "value":100
      },
      {
         "type":"ROLLBACK_COUNT",
         "value":0
      },
      {
         "type":"FILTER_COUNT",
         "value":0
      },
      {
         "type":"WRITE_SKIP_COUNT",
         "value":0
      },
      {
         "type":"PROCESS_SKIP_COUNT",
         "value":0
      },
      {
         "type":"COMMIT_COUNT",
         "value":11
      },
      {
         "type":"READ_SKIP_COUNT",
         "value":0
      },
      {
         "type":"READ_COUNT",
         "value":100
      }
   ]
}

Just some notes about the step metrics:

• READ_COUNT: 100 indicates 100 items were read from CSV;
• WRITE_COUNT: 100 indicates 100 items were written to database
• COMMIT_COUNT: 11 indicates all these items were split into chunks with 11 transactions

You may want to check the output table MOVIES, just to verify that the correct data have been saved. We can do that by remotely accessing PostgreSQL POD and running psql tool.

In order to know where the database is running, run the command:

oc get pods --selector app=postgresql

This will output the details of the pod which is running the database.

NAME               READY     STATUS    RESTARTS   AGE
postgresql-1-sqf9v   1/1       Running   0          1m

To make it easier to reference the name of the pod, capture the name of the pod in an environment variable by running:

POD=`oc get pods --selector app=postgresql -o custom-columns=name:.metadata.name --no-headers`; echo $POD

To create an interactive shell within the same container running the database, you can use the oc rsh command, supplying it the name of the pod.

oc rsh $POD

To start psql tool:

psql sampledb jberet

This will present you with the prompt for running database operations via psql.

psql (9.5.7)
Type "help" for help.

sampledb=>

To check the content of MOVIES table:

SELECT * FROM MOVIES;

There should be 100 rows in MOVIES table in paginated view. Press space for next page, or q to stop.

 rank |                        tit                        |      grs      |    opn
------+---------------------------------------------------+---------------+------------
    1 | Marvel's The Avengers                             | 623357910.000 | 2012-05-04
    2 | The Dark Knight Rises                             | 448139099.000 | 2012-07-20
    3 | The Hunger Games                                  | 408010692.000 | 2012-03-23
    4 | Skyfall                                           | 304360277.000 | 2012-11-09
    5 | The Hobbit: An Unexpected Journey                 | 303003568.000 | 2012-12-14

To exit psql:

\q

To exit the PostgreSQL POD interactive shell:

exit

Restarting a job execution is a common batch job operation, when the previous job execution failed or was stopped. Let's first start a job execution with wrong configuration, expecting it to fail so we can restart it next.

To start csv2db job with wrong db.host value:

curl -s -X POST -H 'Content-Type:application/json' "http://intro-jberet-jberet-lab.[[HOST_SUBDOMAIN]]-80-[[KATACODA_HOST]].environments.katacoda.com/intro-jberet/api/jobs/csv2db/start?db.host=x" | python -m json.tool

Check the status of the above job execution, which should be FAILED when it's finished running:

curl -s http://intro-jberet-jberet-lab.[[HOST_SUBDOMAIN]]-80-[[KATACODA_HOST]].environments.katacoda.com/intro-jberet/api/jobexecutions/2 | python -m json.tool

To restart the above failed job execution, with correct configuration:

curl -s -X POST -H 'Content-Type:application/json' "http://intro-jberet-jberet-lab.[[HOST_SUBDOMAIN]]-80-[[KATACODA_HOST]].environments.katacoda.com/intro-jberet/api/jobexecutions/2/restart?db.host=postgresql" | python -m json.tool

Restarting creates a new job execution. Check the status of the restart job execution, which should be COMPLETED when it's finished running:

curl -s http://intro-jberet-jberet-lab.[[HOST_SUBDOMAIN]]-80-[[KATACODA_HOST]].environments.katacoda.com/intro-jberet/api/jobexecutions/3 | python -m json.tool

If a job execution is taking too long, you may want to stop it. Note that you can only stop a running job execution, and apply stop operation on job execution in other state will fail. Since the duration of our sample job execution is short, let's try stopping a completed job execution, expecting the stop operation to fail:

curl -s -X POST -H 'Content-Type:application/json' "http://intro-jberet-jberet-lab.[[HOST_SUBDOMAIN]]-80-[[KATACODA_HOST]].environments.katacoda.com/intro-jberet/api/jobexecutions/2/stop" | python -m json.tool

You should see output like this with long stacktrace:

{
  "type":"javax.batch.operations.JobExecutionNotRunningException",
  "message":"JBERET000612: Job execution 2 has batch status FAILED, and is not running.",
  "stackTrace":"javax.batch.operations.JobExecutionNotRunningException: ..."
}

To abandon a finished job execution:

curl -s -X POST -H 'Content-Type:application/json' "http://intro-jberet-jberet-lab.[[HOST_SUBDOMAIN]]-80-[[KATACODA_HOST]].environments.katacoda.com/intro-jberet/api/jobexecutions/2/abandon"

Note that you can only abandon a finished job execution, not running ones. Once a job execution is abandoned, it cannot be restarted.

To schedule a job execution:

curl -s -X POST -H 'Content-Type:application/json' -d '{"jobName":"csv2db", "initialDelay":1, "interval":60}' "http://intro-jberet-jberet-lab.[[HOST_SUBDOMAIN]]-80-[[KATACODA_HOST]].environments.katacoda.com/intro-jberet/api/jobs/csv2db/schedule" | python -m json.tool

And you will see the resultant job schedule information as output:

{
  "id":"1",
  "jobScheduleConfig":{
    "jobName":"csv2db",
    "jobExecutionId":0,
    "jobParameters":null,
    "scheduleExpression":null,
    "initialDelay":1,
    "afterDelay":0,
    "interval":60,
    "persistent":false
  },
  "createTime":1508094664385,
  "status":"SCHEDULED",
  "jobExecutionIds":[]
}

The above command schedules to run job csv2db after 1 minute, and every 60 minutes. Note that its status is SCHEDULED, and its associated job executions are an empty array, since no job execution has been started for this schedule. More advanced scheduling are also supported by JBeret such as repeated execution, interval between executions, and calendar-based cron-like schedules.

To list all job schedules:

curl -s "http://intro-jberet-jberet-lab.[[HOST_SUBDOMAIN]]-80-[[KATACODA_HOST]].environments.katacoda.com/intro-jberet/api/schedules" | python -m json.tool

Your output may be like this:

[
  {
    "id":"1",
    "jobScheduleConfig":{
      "jobName":"csv2db",
      "jobExecutionId":0,
      "jobParameters":null,
      "scheduleExpression":null,
      "initialDelay":1,
      "afterDelay":0,
      "interval":60,
      "persistent":false
    },
    "createTime":1508094664385,
    "status":"SCHEDULED",
    "jobExecutionIds":[
      4
    ]
  }
]

From the above output, we can see there is 1 job schedule, which has been fulfilled once with job execution id 4. Its status remains SCHEDULED, since it's a recurring job schedule with more job executions to come.

To cancel the above recurring job schedule:

curl -s -X POST -H 'Content-Type:application/json' "http://intro-jberet-jberet-lab.[[HOST_SUBDOMAIN]]-80-[[KATACODA_HOST]].environments.katacoda.com/intro-jberet/api/schedules/1/cancel"

Then re-run the previous command to list job schedules, to verify their status:

curl -s "http://intro-jberet-jberet-lab.[[HOST_SUBDOMAIN]]-80-[[KATACODA_HOST]].environments.katacoda.com/intro-jberet/api/schedules" | python -m json.tool

The output should show that the job schedule 1 is in CANCELLED status:

[
  {
    "id":"1",
    "jobScheduleConfig":{
      "jobName":"csv2db",
      "jobExecutionId":0,
      "jobParameters":null,
      "scheduleExpression":null,
      "initialDelay":1,
      "afterDelay":0,
      "interval":60,
      "persistent":false
    },
    "createTime":1508094664385,
    "status":"CANCELLED",
    "jobExecutionIds":[
      4
    ]
  }
]