To begin the application development, we will first need to create a directory called “Spark” or any directory name of your choice. This directory will house all the files and components necessary to create the application image.

To create a directory, execute the following command:
mkdir ~/Spark

Navigate to the newly created directory:
cd ~/Spark

Next, we need to create a skeleton file structure. To do so execute the following command:
bdwb --init

bdwb --init command will setup all the base directories and files you need in order to develop any app image

Execute the below command to list all the files and folders created by bdwb command:
ls -R

Make sure you will see the following files and folders:

The next step is to create a Dockerfile.

"A docker file is a text file that the Docker engine understands to automatically build an image by reading the file. The Dockerfile consists of all the commands a user would call to assemble the desired image."

Let’s go ahead and create a Dockerfile inside the newly created centos folder.
For your reference, we have already created a ready Dockerfile in the ~/test directory. We will copy that file into the centos folder using the following command:

copy that file into the centos folder using the following command:
cp ~/test/Dockerfile ~/Spark/image/centos

Execute the following command to view the content of the Dockerfile:
cat ~/Spark/image/centos/Dockerfile

Setup additional dependencies which are needed for Spark.

In this step, we will be showing you the additional steps needed to create a successful Spark image.
We have already referenced the files that are needed, all you would need to do is copy them into the appropriate location.

The dependencies that need to be configured is Jupyter, which we will provide a shell script , and configuring the correct version of Java for Spark.

It is always good to check what version of Spark you are using to understand the dependencies you may need setup in your base image.

Task1:
Copy the pre-made jupyter script to the centos folder:
cp ~/test/configure_jupyter.sh ~/Spark/image/centos

Feel free to run the following command to see what is being executed in the shell script:
cat ~/Spark/image/centos/configure_jupyter.sh

You will see the steps required to setup jupyter listed.

Next, copy over the java script:.
cp ~/test/configure_java8.sh ~/Spark/image/centos

Task2:
We need to add additional configuration files under the appconfig directory. We have already made these files for you, to add them in, please execute the following commands:
Remove the appconfig folder from the Spark folder
rm -rf appconfig
yum install wget -y

Add the appconfig reference files using the below command(Due to space constraint in Katacoda we have uploaded the required appconfig files into dropbox)
wget https://www.dropbox.com/s/wbnr83q26przbs6/appconfig.zip
yum install unzip -y

Unzip the file
unzip appconfig.zip

Check the files under appconfig directory
ls appconfig

Make sure you see following files unders appconfig folder:

In this step we will be creating the bin using BlueData App Workbench by executing the following commands.

.wb file contains a series of App Workbench commands that control the creation of the Catalog Image
Please review the link before proceeding for wb file: Link

Task 1:
For your reference, we have already created a ready .wb file in the ~/test directory. We will copy that file into the Spark directory using the following command:
cp ~/test/Spark.wb ~/Spark

To check the files in Spark folder
ls

Task 2:
To view the content of the file, execute the following
cat ~/Spark/Spark.wb

Task 3:
Execute the .wb file
./Spark.wb

After executing the above command wait for sometime until bin gets created!!

Checking the Bin File For Spark.

Let's see what you have built. The newly built application package (or bundle) is saved in the deliverables directory.
cd deliverables
ls

Make it an executable using the below command
chmod +x bdcatalog-centos7-bluedata-spark221-1.0.bin

To make the new image appear in App Store Copy the bin file to /srv/bluedata/catalog

Go to App Store in the EPIC GUI and click on Refresh button to bring the image in to App Store.

Once the image appears on the App Store click on Install button to install the image.