Clone a data science project into Jupyter from Git

Inside Jupyter, open a terminal. Then click the following text, and paste it into the terminal (cmd+v on Mac or ctrl+v on Windows):

git clone https://github.com/dotmesh-io/dotscience-demo dotscience-demo/setup.sh rm -rf dotscience-demo

This uses Git to clone a public repo from GitHub which contains sample data files and notebooks. It then runs the setup.sh script which simply copies the notebooks into your Dotscience workspace and the data into your dataset folders.

Open the Files view (you may need to click the reload button on the files view) and you should notice that there is now a CSV file in each of 'agent1' and 'agent2' folders, and two notebooks in the main project folder.

This is Bay Area housing data. We're going to build some models to predict the price of the housing data as a function of the features of the houses, such as number of bedrooms and square footage.

Let's start by doing some data wrangling!

Merge and validate some data

The first step is to combine the property data from the first two real estate agents.

In Jupyter, open the Files view and click on 01-data-wrangling.ipynb.

You can read through the notebook and run each cell at a time, or if you're impatient you can click Run -> All Cells. This notebook wrangles the data from both CSV files into a single CSV file, dropping some columns that we don't need and checking the data has no nulls in it.

Once you've run the notebook click Save in Jupyter (the floppy disk icon, or File -> Save) and then go to the Dotscience tab. You should notice that Dotscience has automatically recorded the change to the combined-houses data folder.

What's more, if you go to the Experiment Tracker you should see that the provenance graph is starting to form, dynamically as you do work.

Now it's time to build a model!

Let's build a model!

In Jupyter, open the Files view and click on 02-linear-regression.ipynb.

You can read through the notebook and run each cell at a time, or if you're impatient you can click Run -> All Cells.

This notebook trains a linear regression on the combined house data, using just the finishedsqft feature to begin with.

Once you've run the notebook click Save in Jupyter (the floppy disk icon, or File -> Save) and then go to the Dotscience tab. You should notice that Dotscience has automatically recorded the new model, and also the parameters and summary statistics (in this case, error rate) from testing the trained model.

On the Experiment Tracker, you should now be able to see the first result in the results table and on the scatter plot. You can click on the point on the scatter plot to view exactly the notebook which generated this result, as well as the input & output data files.

Optimise the linear regression

Now try optimising the linear regression, and do another run. You could try putting more features in the model training and see if that reduces the error rate.

When you do a run which changes the results (parameters and summary stats), you should see a new run automatically show up in the tracker.

Remember to save the notebook file, as dotscience doesn't record the run until the file is saved!

Who can get the lowest error rate?

Use whichever libraries or tools are available to you in this docker image (tensorflow, sklearn) to build the most accurate model.

And remember to track your work by using the python print statements like the other examples. The print statements give Dotscience the information it needs to track your work:

• DOTSCIENCE_INPUTS - a list of strings: Which input datasets are used by the notebook/script. Required to build a sensible provenance graph.
• DOTSCIENCE_OUTPUTS - a list of strings: Which output datasets are used by the notebook/script. Required to build a sensible provenance graph.
• DOTSCIENCE_LABELS - a dict of strings -> strings: A set of key-value pair labels which can later be queried over. A label of "ignore" means not to plot the run on the run table or the scatter plot.
• DOTSCIENCE_PARAMETERS - a dict of strings -> strings: The "input parameters" to e.g. training a model. Examples include which features were selected, or how many hidden layers of a neural network to include in the model.
• DOTSCIENCE_SUMMARY - a dict of strings -> strings: Resulting scalar values which you - the data scientist - cares about and are trying to optimise. Examples include accuracy, f-score or error rate (RMSE).

There will be chocolate for the attendee who gets the lowest RMSE!

Show the power of the provenance graph

Try changing how we filter the data in the 01-data-wrangling.ipynb notebook, and observe that on re-running and saving the notebook, the provenance graph is updated to refer to the new data. Later runs now reference the newly updated data, saving you from confusion as data changes "under your feet"!

Now you can track exactly which data each run used, and understand how different datasets impact the different results you're seeing.

And you're done!

Let's move onto the next workshop - a deep dive into Docker, and then Git...

Sign up for the beta

If you liked what you saw and you think it will be useful in your work, please sign up for the beta at:

beta.dotscience.io

Please also join our Slack channel to chat to us and give us feedback!

slack.dotscience.io