For upgrading the image, we will be using a bin file of the existing image already developed. Let us download a .bin file to the directory we have created.
Before that, let us install wget package. For doing so, execute the following command
yum install wget -y

Now, navigate to the Source directory and download the bin file by executing the following command(Due to space constraint in Katacoda we have uploaded the required files into dropbox)
cd ~/Source
wget https://www.dropbox.com/s/r4m3bba2qtueu5q/bdcatalog-centos7-bluedata-spark221-1.0.bin.tar


Extract the content of the tar file by executing the following command
tar xvf bdcatalog-centos7-bluedata-spark221-1.0.bin.tar

Now, Make the .bin file executable by executing the following command
chmod +x bdcatalog-centos7-bluedata-spark221-1.0.bin

In this step we will extract the contents of the .bin file. For this execute the following command.
./bdcatalog-centos7-bluedata-spark221-1.0.bin --payload

This will create two files in the directory. (to view the files execute the following command)
ls

Untar the payload.tar file by executing the following command
tar xvf payload.tar

This will create new directories. (to view the files execute the following command)
ls

Remove the bin and tar file to save space on the device
rm -rf bdcatalog-centos7-bluedata-spark221-1.0.bin
rm -rf bdcatalog-centos7-bluedata-spark221-1.0.bin.tar

Navigate to the new directory, and untar the file by executing the following command
cd bdcatalog-centos7-bluedata-spark221-1.0
tar xvf bdcatalog-centos7-bluedata-spark221-1.0-src.tgz

Now navigate to the directory created after the above step
cd bdcatalog-centos7-bluedata-spark221-1.0-src

Copy the content of this directory to the directory you created initially
cp -R * ~/Source/Spark

Now, to upgrade an existing image to a new one, we should make changes in some of the files like:

-.wb file

Changes that needs to be done in wb file is: Name and distro_id needs to be updated based on the version of the Spark image to reflect the changes in the App Store for the upgraded image.

-Dockerfile

EPIC always starts with a Dockerfile, here we need to explicitly list all of the software used for building the Spark image and also add all the required dependencies as for the Spark version that we are going to build and the commands to configure the application.

In the next few steps we will be modifying these files to upgrade our application image.

Now, we will make changes in .wb file first.

.wb file contains a series of App Workbench commands that control the creation of the Catalog Image
Please review the link before proceeding for wb file: Link

For your reference, we have already created a ready .wb file in the ~/test directory. We will copy that file into the Spark directory using the following command:
cd ~/Source/Spark

Delete the existing .wb file and json file before you copy the new .wb file by executing the below command
rm -rf spark221.wb spark221e1.json

Copy the new .wb file into Spark directory using the following command:
cp ~/test/spark_notebook.wb ~/Source/Spark

To view the content of the file, execute the following
cat ~/Source/Spark/spark_notebook.wb

The next step is to create a Dockerfile.
"A docker file is a text file that the Docker engine understands to automatically build an image by reading the file. The Dockerfile consists of all the commands a user would call to assemble the desired image."

Let’s go ahead and create a Dockerfile inside the newly created centos folder.
For your reference, we have already created a ready Dockerfile in the ~/test directory.We will copy that file into the centos folder using the following command:

Delete the existing Dockerfile by executing the below command:
rm -rf ~/Source/Spark/image/centos/Dockerfile

Copy the new Dockerfile into the centos folder using the following command:
cp ~/test/Dockerfile ~/Source/Spark/image/centos/

To view the contents of the Dockerfile, you can use vi, vim, or cat out the contents. To view the contents in the terminal console execute the following command:
cat ~/Source/Spark/image/centos/Dockerfile

You will now see many commands populate your terminal. These are the commands you would use if you were to install your application manually on a host.
The first line of the Dockerfile determines what is the “base” image you will be using to install your application on.

Blue Data provides their own base image, which you can use by simply adding the following command at the top of your Dockerfile:
FROM bluedata/centos7:latest
You have ability to create your own base image.
e.g: FROM ubuntu:12.04, ubuntu:12.04 is the base image used here.

All the commands proceeding the base image, are the commands used to setup the application.
These files or commands will be setup on top of the base image from BlueData and will eventually compile into a .Bin file for use on the EPIC platform.

Copy java8.sh file to centos directory
cp ~/Source/Spark/image/centos/spark/configure_java8.sh ~/Source/Spark/image/centos/

Copy jupyter file to centos directory
cp ~/Source/Spark/image/centos/spark/configure_jupyter.sh ~/Source/Spark/image/centos/

We need to add additional configuration files under appconfig directory, We already have a reference configuration files, To add them follow the below steps

Remove the existing appconfig folder from the Spark folder
rm -rf appconfig

Add the appconfig reference file using the below command from stored location(Due to space constraint in Katacoda we have uploaded the required files into dropbox)
wget https://www.dropbox.com/s/f8cjkwzz6qdov0d/appconfig.zip
yum install unzip -y

Unzip the file
unzip appconfig.zip

Check files under appconfig directory
ls appconfig

Make sure you see following files unders appconfig folder:

Startscript is a script file which contains code to start all service(s).

spark-slave file is a service script to start/stop/get-status of the Spark-slave service

spark-master file is a service script to start/stop/get-status of the Spark-slave service.

spark-defaults.conf is a default system properties included when running spark-submit.

spark-env.sh is a file to setup the spark environment.

start_jupyterhub.sh file is a service script to start/stop/get-status of the jupyterhub service.

start_jupyter.sh file is a service script to start/stop/get-status of the jupyterhub service.

total_vcores.sh file which is used to obtain the total number of virtual CPU cores assigned to the nodes.

Macros.sh contains all the built in macros of BlueData that would be executed during image creation.

Logging.sh This provides the logging facilities for a catalog configuration bundle.

Utils.sh contains utility functions defined which provides information on docker id, cpu share, memory status and fqdn of the current container.

p_kernel.json file provides interactive python development for Jupyter.

sq_kernel.json file provides interactive SQL interpreter for Jupyter.

core-site.xml,hadoop these files are used to setup Hadoop related configurations.

appjob provides the information on the type of job to be launched and we can also add application specific jobs.

systemd.service It is init system used to bootstrap the user space and to manage system processes after booting.

jupyter-server is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text.

jupyterhub_config.py and jupyter_notebook_config.py contains required configuration to setup jupyterhub and jupyter notebook.

jupyter and jupyterhub contains all the required configuration to run jupyter and jupyterhub.

rstudio is a free and open-source integrated development environment for R, a programming language for statistical computing and graphics.

rstudioserver is a Linux server application that provides a web browser based interface to the version of R running on the server.

start_rstudioserver.sh file is a service script to start/stop/get-status of the rstudio service.

shinyserver is a back end program, it builds a web server specifically designed to host Shiny apps.

start_shinyserver.sh file is a service script to start/stop/get-status of the shiny service.

pam_mkhomedir.sh will create a users home directory if it does not exist when the session begins.

shiro.ini is a config file which defines the user credentials that are recognized by the SecurityManager.

Remove the zip file from the folder
rm -rf appconfig.zip

Now, for creating the bin file, let us execute the .wb file.
./spark_notebook.wb

After executing the command, wait for sometime as the bin file will take time to create!!!!

The newly built application package (or bundle) is saved in the deliverables directory.
cd deliverables
ls

Make it an executable using the below command
chmod +x bdcatalog-centos7-bluedata-spark231mjuphub7x-2.1.bin

Congratulations you have successfully upgraded spark with notebook image.

To make the new image appear in App Store Copy the bin file to /srv/bluedata/catalog

Go to App Store in the EPIC GUI and click on Refresh button to bring the image in to App Store.

Once the image appears on the App Store click on Install button to install the image.