Sparklyr coding vignettes
I have uploaded two coding vignette videos. Both are connected to a shared databricks notebook in our Azure databricks workspace. All of you should have the ability to run the workspace (I didn’t allow for editing but you can replicate it to experiment). Find that notebook here (there are two versions, the original which I made editable and a “live” version which is just executable):
Editable Notebook Executable Notebook
The first video focuses on loading data and shows the code for mounting the blob storage object where the data is stored. I believe this is not going to be necessary to use but just in case I have shared it. In the first video I also talk about how to make plots, which is much trickier on a spark cluster than on a regular computer. To facilitate this, I go through some examples with the dbplot package.
In the second video, I go through some examples of using group_by and summarize on a dataframe in spark, using the microbenchmark package to compare performance and the spark UI and the transformation plans to understand when shuffles and broadcast joins are occurring. We look specifically at two types of aggregation operations, window functions and regular functions, and observe the different behaviors they cause.