Interactive Computing with Jupyter and Almond

space/shift+space to navigate slides

ctrl+enter to run code

up/down to select the previous/next code cell

optimized for full-screen mode

Sören Brunk
@soebrunk

How Do We learn?¶

Exploration is a Feedback Loop¶

Interactive Computing¶

An Interactive computation is a persistent computer program that runs with a "human in the loop," where the primary mode of interaction is through the human interactively writing/running blocks of code and looking at the result.

— Brian Granger (co-founder of Project Jupyter)

The Scala 2 REPL

Ammonite

Limitations of the REPL Design¶

Optimized for exploration from scratch
Write code and see outputs fast (feedback loop)
Not ideal for reading, copying or changing code
- Interleaved inputs and outputs
- History cluttered with refinement attempts
- Dependent expressions
Sharing is difficult
- Not persistent except history
Observe and replicate phases not part the of REPL design

Learning with the REPL

Worksheets¶

A worksheet is a Scala file that is evaluated on save, and the result of each expression is shown in a column to the right of your program. Worksheets are like a REPL session on steroids, and enjoy 1st class editor support: completion, hyperlinking, interactive errors-as-you-type, etc.

— Source: Dotty documentation

Online Worksheets¶

Learning with Worksheets¶

A More Integrated Experience?¶

Tour of Scala

Higher-Order-Functions

Higher order functions take other functions as parameters or return other functions ...

One of the most common examples is the higher-order function map which is available for collections in Scala.

In [ ]:

// Use ctrl+enter to run the cell
val salaries = Seq(20000, 70000, 40000)
val doubleSalary = (x: Int) => x * 2
val newSalaries = salaries.map(doubleSalary)
salaries.map(x => x *3)

doubleSalary is a function which takes a single Int, x, and returns x * 2. In general, the tuple on the left of the arrow => is a parameter list and the value of the expression on the right is what gets returned. On line 3, the function doubleSalary gets applied to each element in the list of salaries.

How?

We're actually in a Jupyter notebook!
It's just shown as a slideshow (using the RISE plugin)
Usually, it looks more like this:

Notebook Basics

Jupyter notebooks are interactive, web based documents
They are made of cells
Two cell types
Documentation
Code

Documentation Cells¶

All the usual Markdown formatting options
Images
Code fences

val x = 23

LaTeX equations $$e^x=\sum_{i=0}^\infty \frac{1}{i!}x^i$$
We can edit them right in the browser!

Double click to enter edit mode, then hit ctrl+enter to return to the rendered view

Code Cells¶

Write code
Evaluate code

In [ ]:

case class Person(name: String, age: Int)

val alice = Person("alice", 42)
val bob = Person("bob", 43)

Seq(alice, bob).map(_.name.capitalize)

Mix and Match¶

We can have any number of cells in a notebook
We can freely mix code cells with documentations cells

In [ ]:

"I'm a code cell, evaluate me!"

In [ ]:

println("I'm so effectful!")

We can even put cells side-by-side
(with the help of a plugin)

In [ ]:

"I'm smaller, but you can still evaluate me."

Notebooks Integrate Rich Documentation with Runnable Code¶

⬇

Notebooks are Worksheets in a REPL¶

⬇

Notebooks are Worksheets in a REPL¶

→ DEMO ←¶

Exploring Data with the REPL¶

Exploring Data with Notebooks¶

In [ ]:

interp.repositories() ++= Seq(coursier.maven.MavenRepository("https://jitpack.io"))
import $ivy.`org.apache.spark::spark-sql:2.4.3` // Or use any other 2.x version here
import $ivy.`sh.almond::almond-spark:0.5.0`
import org.apache.spark.sql._, org.apache.log4j.{Level, Logger}
Logger.getLogger("org").setLevel(Level.OFF)

val spark = {
  NotebookSparkSession.builder()
    .progress(false)
    .master("local[*]")
    .getOrCreate()
}
def sc = spark.sparkContext

Exploring Data with Notebooks¶

In [ ]:

import spark.implicits._
import $file.SparkHelpers, SparkHelpers._

val titanic = spark.read
  .format("csv")
  .option("inferSchema", "true").option("header", "true")
  .load("titanic.csv")

titanic.showHTML()

Exploring Data with Notebooks¶

In [ ]:

import $ivy.`org.vegas-viz::vegas-spark:0.3.12-SNAPSHOT`
import vegas._, vegas.data.External._, vegas.sparkExt._
Vegas().
  withDataFrame(titanic).
  mark(vegas.Bar).
  encodeY("*", aggregate=AggOps.Count, axis=vegas.Axis(title="Number of People", grid=false)).
  encodeColumn("Pclass", Ord, scale=Scale(padding=16.0), axis=vegas.Axis(orient=Orient.Bottom, axisWidth=1.0, offset= -8.0)).
  encodeX("Survived", Nominal, scale=Scale(bandSize = 20.0), hideAxis=true).
  encodeColor("Survived", Nominal, scale=Scale(rangeNominals=List("red", "green"))).
  configFacet(cell=CellConfig(strokeWidth = 0)).configCell(height=400).show

Exploring Data with Notebooks¶

In [ ]:

import $ivy.`org.plotly-scala::plotly-almond:0.7.0`
import plotly._, plotly.element._, plotly.layout._, plotly.Almond._

val trace1 = plotly.Bar(
  Seq("giraffes", "orangutans", "monkeys"), Seq(20, 14, 23), name = "SF Zoo"
)
val trace2 = plotly.Bar(
  Seq("giraffes", "orangutans", "monkeys"), Seq(15, 18, 29), name = "LA Zoo"
)
val data = Seq(trace1, trace2)
val layout = Layout(barmode = BarMode.Group)

plot(data, layout)

But we aren't restricted to visualizing data. We can also visualize code.

Visualizing Code with Notebooks¶

In [ ]:

import $ivy.`io.github.stanch::reftree:1.4.0`
import reftree.render._, reftree.diagram._, reftree.contrib.SimplifiedInstances.string

case class Person(name: String, age: Int)
val people = List(Person("Alice", 29), Person("Bob", 25))

val renderer = Renderer(renderingOptions = RenderingOptions(density = 80))
renderer.render("example", Diagram.sourceCodeCaption(people))
Image.fromFile("example.png")

Notebooks Enable Rich Embedded Output¶

➡

Project Jupyter

Jupyter History¶

2001: IPython Shell

2011: IPython Notebook

2014: Project Jupyter (support for other languages besides Python)

2018: JupyterLab (next generation UI)

High-Level Jupyter Architecture

(source: Jupyter documentation)

Jupyter Frontends¶

Classic Notebook - first Jupyter UI, still widely used
JupyterLab - modern frontend implementation
nteract - minimalistic desktop based frontend
Hydrogen atom plugin - worksheet like with Jupyter backend
IntelliJ and VS Code also have limited notebook support
Currently restricted to Python

JupyterLab

Modernized UI
Modular, component based, modern web technologies
More than notebooks:
File manager
Editor
Terminal
Plugins
Close to a 1.0 release

JupyterLab

The Jupyter Protocol¶

Kernels¶

Responsible for running code
Don't know anything about notebooks
Can be implemented in any language
- Have to support the Jupyter protocol
~ 100 different Kernels available

The almond Scala Kernel

Ammonite Niceties¶

Dynamic dependency resolution via Coursier

In [ ]:

import $ivy.`org.typelevel::squants:1.4.0`
import squants.energy.Kilowatts, squants.time.Hours

val load = Kilowatts(1.2)
val time = Hours(2)
val energyUsed = load * time

Syntax highlighting and pretty printing

In [ ]:

Seq.fill(3)(Seq.fill(5)(f"${scala.util.Random.nextFloat}%1.4f"))

Not A Full Blown IDE - But¶

Autocompletion for regular code and ivy imports
Type hints & metabrowse via shift + tab (work in progress)

In [ ]:

import $ivy.`org.typelevel::squants:` // put the cursor after the last colon and press shift, select 1.4.0

Seq("alice", "bob", "charlie").map(name => name.) // put the cursor after name. and press shift, select capitalize 
// Run the cell, then put the cursor on map and press shift+tab multiple times until you get a type hint at the bottom
// Click on the link int the type hint for a metabrowse window

Using the Ammonite API¶

In [ ]:

repl.history.takeRight(3).toList

In [ ]:

interp.repositories().last

Rich Outputs with the almond Jupyter API¶

In [ ]:

import almond.display._

Html("Some <b>bold</b> text")

In [ ]:

Svg("""<svg height="250" xmlns="http://www.w3.org/2000/svg">
        <rect x="25" y="25" width="200" height="200" fill="lime" stroke-width="4" stroke="pink" />
        <circle cx="125" cy="125" r="75" fill="orange" />
       </svg>""")

In [ ]:

Image.from(url = "http://localhost:8888/static/base/images/logo.png")

In [ ]:

Latex("""$$e^x=\sum_{i=0}^\infty \frac{1}{i!}x^i$$""")

In [ ]:

Javascript("alert('Hello')") // JS execution is restricted in newer frontends

Finally some Progress...¶

In [ ]:

val handle = ProgressBar(10,100)
  .withHtmlWidth("100%")
  .withLabel("Overall Progress")

In [ ]:

for {i <- 1 to 100} {
  handle.withProgress(i).update()
  Thread.sleep(30)
}

Reading User Input¶

In [ ]:

val result = Input().withPrompt(">>> ").request()

Updating Vars¶

In [ ]:

import scala.concurrent.Future
implicit val ec = scala.concurrent.ExecutionContext.global

val f = Future {
  Thread.sleep(10000)
  "finished"
}

val v = "a val"
var x = "a var that we'll update"
var y = "another var"

In [ ]:

x = "updated x"

Using the almond APIs for Library Integrations¶

build.sbt of your library:

libraryDependencies += ("sh.almond" % "scala-kernel-api" % "0.5.0" % Provided)
  .cross(CrossVersion.full))

Your library code:

In [ ]:

def randomEmoji()(implicit kernel: almond.api.JupyterApi): Unit = {
  val emoji = Character.toChars(scala.util.Random.nextInt(0x1F64F - 0x1F600) + 0x1F600).mkString
  kernel.publish.html(s"""<p style="font-size:4em; text-align: center">$emoji</p>""")}

Some notebook:

In [ ]:

// import $ivy.`my:library:1.0`
randomEmoji()

Teaching with Notebooks¶

How To Make Your Existing Documentation Interactive¶

Many tutorials are already Markdown with examples in code fences
- Often even checked via tut or mdoc
Notebook contents are very similar, just encoded differently (JSON)
- Tools like https://github.com/aaren/notedown let us convert markdown with code fences to notebooks

→ DEMO ←¶

Ok this looks cool, where do I find more Info?¶

https://jupyter.org

https://almond.sh

https://github.com/almond-sh/examples

Contributors Welcome!¶

We'd love to see
- More library/framework integrations
- More example notebooks
- Better IDE integration
- A logo :)

Interested?
- ⇒ Talk to Alex (@alxarchambault) or me (@soebrunk)
- ⇒ Join our Gitter channel: https://gitter.im/almond-sh/almond
- ⇒ Or check out the almond repo on https://github.com/almond-sh/almond

Recap¶

Jupyter notebooks
Integrate documentation, runnable code and rich output in a single document
Combine REPLs and worksheets
almond combines the power of Jupyter and Ammonite
Giving us first class interactive computing support in Scala

Questions?

Sören Brunk
@soebrunk

Interactive Computing with Jupyter and Almond

How Do We learn?¶

Exploration is a Feedback Loop¶

Interactive Computing¶

The Scala 2 REPL

Ammonite

Limitations of the REPL Design¶

Learning with the REPL

Worksheets¶

Online Worksheets¶

Learning with Worksheets¶

A More Integrated Experience?¶

How?

Notebook Basics

Documentation Cells¶

Code Cells¶

Mix and Match¶

Notebooks Integrate Rich Documentation with Runnable Code¶

Notebooks are Worksheets in a REPL¶

Notebooks are Worksheets in a REPL¶

→ DEMO ←¶

Exploring Data with the REPL¶

Exploring Data with Notebooks¶

Exploring Data with Notebooks¶

Exploring Data with Notebooks¶

Exploring Data with Notebooks¶

Visualizing Code with Notebooks¶

Notebooks Enable Rich Embedded Output¶

Project Jupyter

Jupyter History¶

High-Level Jupyter Architecture

Jupyter Frontends¶

JupyterLab

The Jupyter Protocol¶

Kernels¶

The almond Scala Kernel

Ammonite Niceties¶

Not A Full Blown IDE - But¶

Using the Ammonite API¶

Rich Outputs with the almond Jupyter API¶

Finally some Progress...¶

Reading User Input¶

Updating Vars¶

Using the almond APIs for Library Integrations¶

Teaching with Notebooks¶

How To Make Your Existing Documentation Interactive¶

Share Your Runnable Notebooks With Binder¶

→ DEMO ←¶

Ok this looks cool, where do I find more Info?¶

https://jupyter.org

https://almond.sh

https://github.com/almond-sh/examples

Contributors Welcome!¶

Recap¶

Questions?