Libraries API Guide¶

In [1]:

notebook.kernelVersion

0.11.0.365

Dependencies and imports¶

Assume we want to add binary dependency on some library. In this case let's add JAR coordinates of this library to the dependencies array of our descriptor. If it is not on Maven Central, we add repository (or repositories) where this dependency is located. Also, we can add default imports

In [ ]:

val dollar = "$"
loadLibraryDescriptor(
    """
    {
        "dependencies": [
            "org.ileasile:kformat:${dollar}libVersion"
        ],
        "repositories": [
            "https://maven.pkg.github.com/ileasile/kformat"
        ],
        "imports": [
            "org.ileasile.kformat.*"
        ],
        "properties": {
            "libVersion": "0.0.1"
        }
    }
    """,
    mapOf("libVersion" to "0.0.2")
)

Huh, we've got 401(Unauthorized). That's because reading from GitHub Packages requires authorization. We can add authorization for our repository in descriptor.

In [ ]:

val readToken = "ghp_<your read token>"

In [ ]:

loadLibraryDescriptor("""
    {
        "dependencies": [
            "org.ileasile:kformat:0.0.2"
        ],
        "repositories": [
            {
                "path": "https://maven.pkg.github.com/ileasile/kformat",
                "username": "ileasile",
                "password": "$readToken"
            }
        ],
        "imports": [
            "org.ileasile.kformat.*"
        ]
    }
""")

In [ ]:

kFormat {
    color(BasicColor.Red) {
        italic {
            + "Red italic text"
        }
    }
}.asAnsiEscaped()

Red italic text

Another way to do the same thing using Kotlin API:

In [ ]:

USE {
    dependencies("org.ileasile:kformat:0.0.2")
    repository(
        "https://maven.pkg.github.com/ileasile/kformat",
        "ileasile",
        readToken
    )
    import("org.ileasile.kformat.*")
}

Also, Kotlin API allows to define dependencies in Gradle style:

In [ ]:

USE {
    dependencies {
        implementation("org.ileasile:kformat:0.0.2")
    }
    repositories {
        maven {
            url = "https://maven.pkg.github.com/ileasile/kformat"
            credentials {
                username = "ileasile"
                password = readToken
            }
        }
    }
    import("org.ileasile.kformat.*")
}

In [ ]:

//@file:DependsOn
//@file:Repository

Initialization callbacks¶

You can define the code that should be run after library initialization. In descriptors, only code executions are available for this purpose. In Kotlin API you can do a bit more.

In [ ]:

loadLibraryDescriptor("""
    {
        "init": [
            "val myVar1 = 42",
            "fun doInit1() {}",
            "println(\"Library 1 initialized\")"
        ]
    }
""")

Library 1 initialized

In [ ]:

class Config(
    var bufferSize: Int = 100
)

val config1 = Config()

USE {
    onLoaded {
        declare("myVar3" to config1)
    }
}

In [ ]:

myVar3.bufferSize = myVar1

In [ ]:

myVar3.bufferSize

In [ ]:

myVar3.bufferSize = 10

In [ ]:

config1.bufferSize

Before- and after-cell callbacks¶

Sometimes you may want to execute some code before cell execution. That's how you can achieve it with library descriptor

In [ ]:

loadLibraryDescriptor("""
    {
        "initCell": [
            "println(\"Before cell execution - 1\")"
        ]
    }
""")

In [ ]:

2 + 2 + 7

Before cell execution - 1

You can achieve the same thing using Kotlin API

In [ ]:

USE {
    beforeCellExecution {
        println("Before cell execution - 2")
    }
}

Before cell execution - 1

In [ ]:

2 + 2

Before cell execution - 1
Before cell execution - 2

You can manage added callbacks. Let's clear them.

In [ ]:

notebook.beforeCellExecutionsProcessor.unregisterAll()

Before cell execution - 1
Before cell execution - 2

In [ ]:

2 + 2

With Kotlin API, you can also create callbacks that are run after cells execution. Execution result is passed to them.

In [ ]:

USE {
    afterCellExecution { snippetInstance, result ->
        val maybeDoubledResult = (result.value as? Int)?.let { it * 2 }
        println("After execution of $snippetInstance: $maybeDoubledResult (${result.name})")
    }
}

In [ ]:

2 + 2

After execution of Line_34_jupyter@4306c59c: 8 (res34)

Let's remove this callback as well, but do it in a bit more complicated manner

In [ ]:

notebook.afterCellExecutionsProcessor.let { processor ->
    processor.unregister(processor.registeredExtensions().first())
}

Shutdown and interruption callbacks¶

Kernel users have the ability to interrupt, restart and stop the kernel. What does it mean?

Interruption just interrupts current cell execution. For example, if the user has run very long or infinite computation, they might want to interrupt it. After interruption kernel state remains untouched.

Stopping kernel means just stopping the session. Kernel process is aborted, kernel state is aborted too.

Restarting kernel means stopping followed by starting. Kernel process is aborted and then started again, kernel state is also cleared.

Sometimes you may want to write a callback on kernel interruption or shutdown (that happens on stopping or restarting). And you can, using descriptor or Kotlin API.

In [ ]:

// Interruption callbacks aren't supported by descriptor API
loadLibraryDescriptor("""
    {
        "shutdown": [
            "println(\"Bye!\")"
        ]
    }
""")

In [ ]:

var interruptionCount = 0
USE {
    onShutdown {
        println("Bye! (2)")
    }

    onInterrupt {
        ++interruptionCount
    }
}

In [ ]:

while(true) {}

The execution was interrupted

Shutdown callbacks can be managed by corresponding processor. API for extensions management will be most likely reworked in the future versions of kernel.

In [ ]:

interruptionCount

In [ ]:

notebook.shutdownExecutionsProcessor.unregisterAll()
notebook.interruptionCallbacksProcessor.unregisterAll()

Color scheme change callbacks¶

Sometimes you may want to do something when the user changes current color scheme in their client. For example, you might want to change color scheme for newly created outputs or even update already created ones. You definitely can, but only in Kotlin API, and only if the user uses Kotlin Notebook plugin

In [ ]:

var isDark = notebook.currentColorScheme == ColorScheme.DARK
USE {
    println("Dark? - $isDark")

    onColorSchemeChange { colorScheme ->
        isDark = colorScheme == ColorScheme.DARK
        println("Color scheme is changed")
    }
}

Dark? - false
Color scheme is changed

In [ ]:

isDark

false

In [ ]:

notebook.colorSchemeChangeCallbacksProcessor.unregisterAll()

Rendering¶

Imagine you're developing a model for TicTacToe game and want to experiment with it in the notebook. You want to have some visual representation of the model. Now all what you have is a following string representation of the game state:

In [ ]:

val gameState = "x_o|_x_|___"
gameState

x_o|_x_|___

First thing you might want to do is to create a matrix containing the state in easier-to-use form

In [ ]:

enum class TTTCell {
    X, O, Empty
}

fun getGameState(state: String) = state.split('|')
    .map { line ->
        line.map {
            when (it) {
                'x' -> TTTCell.X
                'o' -> TTTCell.O
                '_' -> TTTCell.Empty
                else -> error("Not possible")
            }
        }
    }

In [ ]:

val stateMatrix = getGameState(gameState)
stateMatrix

[[X, Empty, O], [Empty, X, Empty], [Empty, Empty, Empty]]

In [ ]:

class TTTState(val stateMatrix: List<List<TTTCell>>)

In [ ]:

val state = TTTState(stateMatrix)

To render it, we may add a text renderer. Text renderers are supported only by Kotlin API and can only render values to string. Their benefit is that they can easily stack together: string are addable.

In [ ]:

USE {
    addTextRenderer(TextRenderer { _, value ->
        (value as? TTTState)?.let { state ->
            state.stateMatrix.joinToString("\n") { line ->
                line.joinToString(" ") {
                    when (it) {
                        TTTCell.X -> "X"
                        TTTCell.O -> "O"
                        TTTCell.Empty -> "_"
                    }
                }
            }
        }
    })
}

In [ ]:

state

X _ O
_ X _
_ _ _

It's much better now, but we still might wish some graphic representation of the state. Let's create a couple of renderers. Renderers accept some values and return some other values. Rendering logic is described in kernel documentation. Here, we register created renderers with RenderersProcessor and specify priority for them to ensure the order of probation

In [ ]:

val htmlStateRenderer = createRenderer<TTTState> {
    val style = """
        border: 1px solid black;border-collapse: collapse;
    """.trimIndent()
    val html =
        it.stateMatrix.joinToString("\n", prefix = "<div><table style='$style'>", postfix = "</table></div>") { line ->
            line.joinToString("", prefix = "<tr style='$style'>", postfix = "</tr>") { cell ->
                buildString {
                    append("<td style='$style'>")
                    append(
                        when (cell) {
                            TTTCell.X -> "X"
                            TTTCell.O -> "O"
                            TTTCell.Empty -> "\u200D"
                        }
                    )
                    append("</td>")
                }
            }
        }
    mimeResult(
        MimeTypes.HTML to html,
        MimeTypes.PLAIN_TEXT to it.stateMatrix.toString()
    )
}
notebook.renderersProcessor.registerWithoutOptimizing(htmlStateRenderer, ProcessingPriority.DEFAULT + 5)

In [ ]:

state

X	‍	O
‍	X	‍
‍	‍	‍

[[X, Empty, O], [Empty, X, Empty], [Empty, Empty, Empty]]

The following renderer will render not to DisplayResult, but to a BufferedImage. Kernel knows how to render BufferedImage, that's because it works fine.

In [ ]:

import java.awt.Color
import java.awt.geom.Ellipse2D
import java.awt.image.BufferedImage

@OptIn(ExperimentalStdlibApi::class)
val imageStateRenderer = createRenderer<TTTState> { state ->
    val cellSize = 100
    val padding = 5
    val mtx = state.stateMatrix
    val rowsCount = mtx.size
    val colsCount = mtx.first().size
    val width = 2 * padding + cellSize * colsCount
    val height = 2 * padding + cellSize * rowsCount

    val img = BufferedImage(width, height, BufferedImage.TYPE_INT_ARGB)
    val g = img.createGraphics()
    g.color = Color.BLACK

    val figPadding = cellSize / 10
    fun drawCell(xOffset: Int, yOffset: Int, cell: TTTCell) {
        when (cell) {
            TTTCell.X -> {
                g.drawLine(
                    xOffset + figPadding,
                    yOffset + figPadding,
                    xOffset + cellSize - figPadding,
                    yOffset + cellSize - figPadding
                )
                g.drawLine(
                    xOffset + cellSize - figPadding,
                    yOffset + figPadding,
                    xOffset + figPadding,
                    yOffset + cellSize - figPadding
                )
            }
            TTTCell.O -> {
                g.draw(Ellipse2D.Double(
                    (xOffset + figPadding).toDouble(),
                    (yOffset + figPadding).toDouble(),
                    (cellSize - 2 * figPadding).toDouble(),
                    (cellSize - 2 * figPadding).toDouble(),
                ))
            }

            TTTCell.Empty -> {}
        }
    }

    fun drawGrid() {
        for (i in 0..rowsCount) {
            g.drawLine(
                padding,
                padding + i * cellSize,
                padding + colsCount * cellSize,
                padding + i * cellSize,
            )
        }
        for (j in 0..colsCount) {
            g.drawLine(
                padding + j * cellSize,
                padding,
                padding + j * cellSize,
                padding + rowsCount * cellSize,
            )
        }
    }

    drawGrid()
    for (i in 0..< rowsCount) {
        for (j in 0..< colsCount) {
            drawCell(padding + j * cellSize, padding + i * cellSize, mtx[i][j])
        }
    }

    img
}

notebook.renderersProcessor.registerWithoutOptimizing(imageStateRenderer, ProcessingPriority.HIGHER)

In [ ]:

state

class java.awt.image.BufferedImage: 310x310 px

Renderer for BufferedImage is already registered!

In [ ]:

USE {
    render<TTTState> { it }
}

In [ ]:

notebook.renderersProcessor.registeredRenderers()
    .map { it.renderer }
    .filter { (it as? RendererTypeHandler)?.acceptsType(BufferedImage::class) == true }

[Renderer of subtypes of class java.awt.image.BufferedImage]

You can add a renderer in the library descriptor as well, but functionality isn't so rich there. In fact, you can only write code-based renderers, and renderers are matched by exact FQN match. Subtypes aren't accepted.

In [ ]:

val dollar = "$"
val dollarIt = "\$it"

class Customer2(val name: String, val age: Int)

loadLibraryDescriptor(
    """
        {
            "renderers": {
                "${Customer2::class.qualifiedName}": "\"Customer $dollar{${dollarIt}.name} aged $dollar{${dollarIt}.age}\""
            }
        }
    """
)

In [ ]:

Customer2("Ada", 42)

Customer Ada aged 42

One more thing that you can render are exceptions! Let's add a renderer for NullPointerException

In [ ]:

USE {
    renderThrowable<NullPointerException> {
        HTML("<b>Kotlin is a null-safe language</b>")
    }
}

In [ ]:

System.getProperty("some_unexistent_property")!!

Kotlin is a null-safe language

You can also render having current execution host in the context

In [ ]:

USE {
    fun TTTCell.toHTML() = HTML("<TODO>")

    renderWithHost<TTTCell> { executionHost, value ->
        executionHost.execute {
            display(value.toHTML(), "12")
            display(textResult(value.toString()), "")
        }
    }
}

Code preprocessing¶

Kotlin kernel gives you an ability to preprocess the code before execution. One of the examples of preprocessing is magics. They are preprocessed with a built-in processor. But sometimes you have to write your own one.

For now, you can't run suspend functions directly in the notebook cells. Let's try to improve the situation writing a workaround for this problem. We start with creating a function

In [ ]:

%use coroutines

In [ ]:

private class Decl(val varMod: String, val name: String)

fun generateRunBlocking(code: String): String {
    val declarationRegex = """^(va[lr]) +(\w+) """.toRegex(RegexOption.MULTILINE)
    val matches = declarationRegex.findAll(code).toList()
    val newCode = code.toCharArray()
    val declarations = matches.map { m ->
        val modGroup = m.groups[1]!!
        val modifier = modGroup.value
        for (i in modGroup.range) {
            newCode[i] = ' '
        }
        val varName = m.groups[2]!!.value
        Decl(modifier, varName)
    }
    return buildString {
        for (decl in declarations) {
            append("var ${decl.name}: Any? = null\n")
        }
        append("runBlocking {\n")
        append(String(newCode))
        append("\n}")
    }
}

Let's test it

In [ ]:

generateRunBlocking("""
    val xyz = 42
    var t = 12
""".trimIndent())

var xyz: Any? = null
var t: Any? = null
runBlocking {
    xyz = 42
    t = 12
}

In [ ]:

USE {
    preprocessCode { code -> generateRunBlocking(code) }
}

In [ ]:

suspend fun susF(): Int = 42

val a = susF()

In [ ]:

val processors = notebook.codePreprocessorsProcessor.registeredExtensions().toList()
println(processors)

[org.jetbrains.kotlinx.jupyter.api.libraries.JupyterIntegration$Builder$preprocessCodeWithLibraries$1@599b664e, org.jetbrains.kotlinx.jupyter.magics.MagicsProcessor@6b1106a5]

In [ ]:

notebook.codePreprocessorsProcessor.unregister((processors!! as List<CodePreprocessor>)[0])

In [ ]:

val processors2 = notebook.codePreprocessorsProcessor.registeredExtensions().toList()
println(processors2)

[org.jetbrains.kotlinx.jupyter.magics.MagicsProcessor@6b1106a5]

Reflection- and compile-based postprocessing¶

As we discussed above, there is possibility to add after-evaluation callbacks. There is a possibility to consider script instance as well. Script instance contains all top-level variables and functions you define as members. Let's take a look

In [ ]:

import kotlin.reflect.KProperty
import kotlin.reflect.KProperty1
import kotlin.reflect.full.memberFunctions
import kotlin.reflect.full.memberProperties

USE {
    afterCellExecution { snippetInstance, _ ->
        val kClass = snippetInstance::class
        println(
            """
Variables:
${
    kClass.memberProperties.joinToString("\n") { prop ->
        "${prop.name} = ${
            (prop as KProperty1<Any, Any?>).get(
                snippetInstance
            )
        }"
    }
}

Functions:
${kClass.memberFunctions.joinToString("\n") { f -> f.name }}
        """
        )
    }
}

In [ ]:

val myVar1 = 42
val myVar2 = "str"

fun myFun1() { println("fun called") }

Variables:
myVar1 = 42
myVar2 = str
userHandlesProvider = org.jetbrains.kotlinx.jupyter.ReplForJupyterImpl@4d7b2340

Functions:
myFun1
equals
hashCode
toString

In [ ]:

notebook.afterCellExecutionsProcessor.unregisterAll()

Let's now try to do something compile-time based. We start with file annotations processing

In [ ]:

USE {
    onFileAnnotation<Suppress> {
        val myAnno = it.first() as Suppress
        println("Warnings suppressed: " + myAnno.names.toList())
    }
}

In [ ]:

@file:Suppress("UNUSED_PARAMETER")

fun u(unusedArg: Int) = 44

Warnings suppressed: [UNUSED_PARAMETER]

We can also handle annotations for classes. They are runtime-based.

In [ ]:

annotation class MyAnnotation

In [ ]:

USE {
    onClassAnnotation<MyAnnotation> { println("Annotated classes: $it") }
}

In [ ]:

@MyAnnotation
class MyClass

class MyClass2

@MyAnnotation
interface MyInterface3

3 + 3

Annotated classes: [class Line_83_jupyter$MyClass, class Line_83_jupyter$MyInterface3]

In [ ]:

class MyData

USE {
    onVariable<MyData> { t, kProperty ->
        execute("val gen_${kProperty.name} = 1")
    }
//    updateVariable<MyData> { t, kProperty ->
//        execute("")
//
//    }
}

In [ ]:

val d = MyData()

In [ ]:

gen_d

Variables reports¶

In [ ]:

notebook.variablesReport

Visible vars: 
	dollar : $
	readToken : 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
	myVar1 : 42
	config1 : Line_16_jupyter$Config@45d5ee1e
	myVar3 : Line_16_jupyter$Config@45d5ee1e
	interruptionCount : 1
	isDark : false
	gameState : x_o|_x_|___
	stateMatrix : [[X, Empty, O], [Empty, X, Empty], [Empty, Empty, Empty]]
	state : Line_47_jupyter$TTTState@1f8e46f4
	htmlStateRenderer : Renderer of subtypes of class Line_47_jupyter$TTTState
	imageStateRenderer : Renderer of subtypes of class Line_47_jupyter$TTTState
	dollarIt : $it
	___a : 42
	a : 42
	___processors : [org.jetbrains.kotlinx.jupyter.api.libraries.JupyterIntegration$Builder$preprocessCodeWithLibraries$1@599b664e, org.jetbrains.kotlinx.jupyter.magics.MagicsProcessor@6b1106a5]
	processors : [org.jetbrains.kotlinx.jupyter.api.libraries.JupyterIntegration$Builder$preprocessCodeWithLibraries$1@599b664e, org.jetbrains.kotlinx.jupyter.magics.MagicsProcessor@6b1106a5]
	processors2 : [org.jetbrains.kotlinx.jupyter.magics.MagicsProcessor@6b1106a5]
	myVar2 : str
	d : Line_84_jupyter$MyData@9b17514
	gen_d : 1

In [ ]:

USE {
    markVariableInternal { prop ->
        prop.name.startsWith("___")
    }
}

In [ ]:

val ___b = 8
val b = 9

In [ ]:

notebook.variablesReport

Visible vars: 
	dollar : $
	readToken : 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
	myVar1 : 42
	config1 : Line_16_jupyter$Config@45d5ee1e
	myVar3 : Line_16_jupyter$Config@45d5ee1e
	interruptionCount : 1
	isDark : false
	gameState : x_o|_x_|___
	stateMatrix : [[X, Empty, O], [Empty, X, Empty], [Empty, Empty, Empty]]
	state : Line_47_jupyter$TTTState@1f8e46f4
	htmlStateRenderer : Renderer of subtypes of class Line_47_jupyter$TTTState
	imageStateRenderer : Renderer of subtypes of class Line_47_jupyter$TTTState
	dollarIt : $it
	___a : 42
	a : 42
	___processors : [org.jetbrains.kotlinx.jupyter.api.libraries.JupyterIntegration$Builder$preprocessCodeWithLibraries$1@599b664e, org.jetbrains.kotlinx.jupyter.magics.MagicsProcessor@6b1106a5]
	processors : [org.jetbrains.kotlinx.jupyter.api.libraries.JupyterIntegration$Builder$preprocessCodeWithLibraries$1@599b664e, org.jetbrains.kotlinx.jupyter.magics.MagicsProcessor@6b1106a5]
	processors2 : [org.jetbrains.kotlinx.jupyter.magics.MagicsProcessor@6b1106a5]
	myVar2 : str
	d : Line_84_jupyter$MyData@9b17514
	gen_d : 1
	b : 9

Auxiliary¶

In [ ]:

// You can set minimal supported version of kernel
USE {
    setMinimalKernelVersion("0.11.0.365")
}

In [ ]:

USE {
    setDescription("Very good library")
    setWebsite("https://cool.site")
}