Reading and writing lakeFS objects

Now that we have a repository, we can add an object to lakeFS by using the fs command.

Add an object from an existing source

The following command will add the object title.episode.tsv.gz to lakeFS under the path /imdb/title.episode.tsv.gz in branch master: lakectl fs upload lakefs://[email protected]/imdb/title.episode.tsv.gz --source title.episode.tsv.gz

Add an object using stdin

The following command will add the object object.txt under path /sample in branch master: echo "this is a sample" | lakectl fs upload lakefs://[email protected]/sample/object.txt --source -

You can use the ls command to list the objects in your repository: lakectl fs ls lakefs://[email protected]/

If you'd like to see the objects and paths under sample: lakectl fs ls lakefs://[email protected]/sample/

To view the metadata of an object you can use the stat command: lakectl fs stat lakefs://[email protected]/sample/object.txt

To dump the content of an object use the cat command: lakectl fs cat lakefs://[email protected]/sample/object.txt

Remove the object with the rm command: lakectl fs rm lakefs://[email protected]/sample/object.txt

Commit commands

We’ve successfully created our first repositroy and added a sample dataset to it.

You can now commit your changes using the commit command

The next command will commit all changes under master branch in th repository my-repo with the message first commit: lakectl commit lakefs://[email protected] --message "first commit"

To list the commits with the log command: lakectl log lakefs://[email protected]

You'll now see there is two commits:

• Your first commit
• An auto generated commit, Repository created

Creating a branch

Create a new branch out of master using the branch command

The next command will create a new branch called my-branch the source branch will be master lakectl branch create lakefs://[email protected]/ --source lakefs://[email protected]

After succesfully creating a branch, you can use the list command to list the branches on owr repository: lakectl branch list lakefs://my-repo

Run a Spark job

Let's use spark shell and make some changes on the new branch.

Open spark shell: spark-shell

This is a standard installation of Spark - since lakeFS is API compatible with S3, all we need to do is simply add the branch name as a prefix for the path

Next, we will run the next commands in sprak shell:

val repo = "my-repo" val branch = "my-branch" val workDir = s"s3a://${repo}/${branch}/imdb" val dataPath = s"$workDir/title.episode.tsv.gz" val outputPath = s"$workDir/episodes_by_season" import org.apache.spark.sql.SparkSession val spark = SparkSession.builder.appName("IMDB Loader").getOrCreate() spark.sparkContext.hadoopConfiguration.set("fs.s3a.aws.credentials.provider", "com.amazonaws.auth.DefaultAWSCredentialsProviderChain") spark.sparkContext.hadoopConfiguration.set("fs.s3a.endpoint", "http://s3.lakefs:8000") spark.sparkContext.hadoopConfiguration.set("fs.s3a.path.style.access", "true") // convert our TSV input into Parquet, partitioned by the "startYear" field import spark.implicits._ val basics = spark.read.format("csv").option("header", "true").option("delimiter", "\t").load(dataPath) basics.toDF().write.partitionBy("seasonNumber").parquet(outputPath) spark.stop()

It might take a few minutes.

Once the commands end running, quit with the command: :q

Commit Spark changes

Before we commit let's review the changes we are about to commit: lakectl diff lakefs://[email protected]

Looks good, let's commit: lakectl commit lakefs://[email protected] --message "after spark job"

Before merging your changes back to master - let's see the changes we are about to push by:

Running the diff command on my-branch and master: lakectl diff lakefs://[email protected] lakefs://[email protected]

After verifying, you can merge back: lakectl merge lakefs://[email protected] lakefs://[email protected]

That's it :) You've successfully merged your changes to the repository.

Feel free to stay and play around with the environment. Installing lakeFS is as easy - see the quickstart for further instructions.