Notebook

In [ ]:

from yggdrasil import tools
from yggdrasil.runner import run
import trimesh

One way model-to-model connection¶

To instead direct the heights to the light model from above, we then use an updated version of the yamls/connections_v0.yml YAML (diff displayed below). This can be in same file or a new one as the model, but is separate here to keep output modular.

In [ ]:

tools.display_source_diff('yamls/connections_v0.yml', 'yamls/connections_v1.yml', number_lines=True)

To run this integration, we then pass 3 YAMLs: the light model YAML, the shoot model YAML, and the shoot-to-light connection YAML.

In [ ]:

run(['yamls/light_v0_python.yml', 'yamls/shoot_v1.yml', 'yamls/connections_v1.yml'], production_run=True)

The next two cells display the mesh from the final time step and the contents of the resulting output/light_v0.txt file.

In [ ]:

mesh = trimesh.load_mesh('output/mesh_008.obj')
mesh.show()

In [ ]:

tools.display_source('output/light_v0.txt')

Test your knowledge #3¶

Write a YAML to connect the models/light_v0.py model to the models/weather.py model from "Test your knowledge #1" and run the integration in the cell below.
Write a YAML to connect the models/shoot_v1.py model to the models/co2.py model from "Test your knowledge #2" and run the integration in the cell below.
Write a YAML to connect the models/shoot_v1.py model to both the models/light_v0.py model and the models/co2.py model and run the integration in the cell below.

Tip: Connections can have more than one input and/or output¶

In [ ]:

Remote Procedure Call (RPC)¶

The previous example achieves the goal of connecting the two models, but it is not particularly powerful in this case given that the same could be received by saving the heights from the shoot to a file that can be read by the light model. However, if we want the shoot model to incorporate the light intensity into the calculation of how fast the shoot should grow, then we need to be able to dynamically access the output from the light model at each time step which cannot be accomplished via files. Instead we need the plant model to call the light model at each timestep via a remote procedure call (RPC). The client model (shoot model) sends requests to the server model (light model) and then receives the response via the call method.

The diff displayed by the cell below shows the updated version of the shoot model where

The YggRpcClient interface function is used instead of YggOutput
The channel name used to complete the connection is light_shoot which comes from the name of the two models (<server>_<client>)
The time and height are sent to the light model via the call method which returns the output from the light model (intensity).
The scale factor used to grow the shoot mesh is computed based on the light intensity return by the light model

While this example uses the Python interface as it is written in Python, the server/client interfaces in the other languages are very similar. This table shows the interface classes/functions and the call method that can be used on the client side:

Language	Server	Client	Call
python	`YggRpcServer("{channel_name}")`	`YggRpcClient("{channel_name}")`	`flag, {inputs} = {channel_obj}.call({outputs})`
R	`YggInterface('YggRpcServer', '{channel_name}')`	`YggInterface('YggRpcClient', '{channel_name}')`	`c(flag, {inputs}) %<-% {channel_obj}$call({outputs})`
matlab	`YggInterface('YggRpcServer', '{channel_name}')`	`YggInterface('YggRpcClient', '{channel_name}')`	`[flag, {inputs}] = {channel_obj}.call({outputs})`
c	`yggRpcServerType("{channel_name}", {datatype_in}, {datatype_out})`	`yggRpcClientType("{channel_name}", {datatype_out}, {datatype_in})`	`flag = rpcCall({channel_obj}, {outputs}, {inputs})`
c++	`YggRpcServer {channel_obj}("{channel_name}", {datatype_in}, {datatype_out})`	`YggRpcClient {channel_obj}("{channel_name}", {datatype_out}, {datatype_in})`	`flag = {channel_obj}.call({nargs}, {outputs}, {input_refs})`
fortran	`ygg_rpc_server_type("{channel_name}", {datatype_in}, {datatype_out})`	`ygg_rpc_client_type("{channel_name}", {datatype_out}, {datatype_in})`	`flag = ygg_rpc_call({channel_obj}, [yggarg({outputs})], [yggarg({inputs})])`

In [ ]:

tools.display_source_diff('models/shoot_v1.py', 'models/shoot_v2.py', number_lines=True)

To run this integration, two key modification need to be made to the model YAMLs as shown in the diffs below:

Add the is_server: true parameter to the light model YAML to declare that it is a server
Add the client_of: [light] parameter to the shoot model YAML to tell yggdrasil that it will be calling the light model server as a client

Since yggdrasil handles the server-client connections, no connection YAML is required.

In [ ]:

tools.display_source_diff('yamls/shoot_v1.yml', 'yamls/shoot_v2.yml', number_lines=True)
tools.display_source_diff('yamls/light_v0_python.yml', 'yamls/light_v1_python.yml', number_lines=True)

The cell below runs the shoot-light RPC integration using these YAML files and displays the resulting mesh.

In [ ]:

run(['yamls/light_v1_python.yml', 'yamls/shoot_v2.yml'], production_run=True)
mesh = trimesh.load_mesh('output/mesh_008.obj')
mesh.show()

Splitting Calls¶

Client (and timesync as we will discuss later) "calls" are really a combination of a send and receive (sending the request and receiving the response). If call's take a long amount of time, we can split the call into its components to take advantage of the parallelism yggdrasil offers. A client (or timesync comm) can call send a request, finish another unrelated task while the server/sync operation takes place, and then call receive to get the response. For example, the call to the light model in the previous example could be split between a send and receive call:

In [ ]:

tools.display_source_diff('models/shoot_v2.py', 'models/shoot_v2_split.py', number_lines=True)

In [ ]:

run(['yamls/light_v1_python.yml', 'yamls/shoot_v2_split.yml'], production_run=True)
mesh = trimesh.load_mesh('output/mesh_008.obj')
mesh.show()

Test your knowledge #4¶

Write a YAML to have the models/shoot_v2.py model call both the models/light_v0.py model and the models/weather.py model.
Update the YAMLs and source code for the models/light_v0.py and models/weather.py models so that the light model acts as a client and calls the weather model as a server and run the integration in the cell below.
Write a YAML to have the models/shoot_v2.py model call the models/light_v0.py model and have the models/light_v0.py model call the models/weather.py model (nested servers).

Tip: Models that are wrapped functions need to pass the `global_scope=True` keyword to interface functions (e.g. `YggRpcClient`) if they will be called more than once¶

In [ ]:

Duplicating Models¶

Splitting calls can be particularly powerful if you are making many calls to a server model and the server model evaluation does not depend on previous calls. In such cases, multiple copies of the server model can be run in parallel and respond to requests in parallel.

The diff displayed in cell below modifies the models/shoot_v2_split.py version of the shoot model so that it calculates an intensity for every vertex in the shoot mesh and saves the intensity array to a file as a Python pickle. Requests are sent for every vertex before beginning to receive any responses.

In [ ]:

tools.display_source_diff('models/shoot_v2_split.py', 'models/shoot_v2_copies.py', number_lines=True)

To run 2 copies of the light model, the copies: 2 line can be added to the light YAML from above as shown in the diff the cell below displays.

In [ ]:

tools.display_source_diff('yamls/light_v1_python.yml', 'yamls/light_v2_python.yml', number_lines=True)

To run this integration, you would pass 2 YAMLS to yggdrasil:

The plant model YAML with that outputs heights for every vertex
The light model YAML with the server and copies parameters

The next shell runs this integration and displays the resulting mesh.

In [ ]:

run(['yamls/light_v1_python.yml', 'yamls/shoot_v2_copies.yml'], production_run=True)

# Plot results w/ light intensity mapped to color
import pickle
with open('output/light_008.pkl', 'rb') as fd:
    light = pickle.load(fd)
mesh = trimesh.load_mesh('output/mesh_008.obj')
mesh.visual.vertex_colors = trimesh.visual.interpolate(light/max(light))
mesh.show()

Parallelism inside MyBinder¶

Unless you install the materials locally, your notebook is being hosted by the MyBinder service. Although yggdrasil attempts to run the models in parallel, there is essentially only 1 core available on the instances provided (for free) by MyBinder so integrations run on MyBinder instances are not really running in parallel (they are running concurrently so one can run while the other is waiting for an asynchronous call to return, but the models must share the CPU). If you were to download the demo repository and run the notebook on your machine, you would see much better performance from all of the integrations involving more than one model as they would actually run in parallel.

Test your knowledge #5¶

Increase the number of copies of the in the yamls/shoot_v1.yml model. What do you think the output file (output/height.txt) will look like?
Run the version with multiple copies. What does the output file (output/height.txt) look like? Why?

Tip: There is a bug(!) When duplicating a wrapped function model (e.g. the light model) that is not a server, the duplicates won't be shut down cleanly. If you run into this (the notebook will hang), you will need to interupt the kernel (`Ctrl+C` on the command line)¶

In [ ]:

One way model-to-model connection¶

Test your knowledge #3¶

Tip: Connections can have more than one input and/or output¶

Remote Procedure Call (RPC)¶

Splitting Calls¶

Test your knowledge #4¶

Tip: Models that are wrapped functions need to pass the global_scope=True keyword to interface functions (e.g. YggRpcClient) if they will be called more than once¶

Duplicating Models¶

Parallelism inside MyBinder¶

Test your knowledge #5¶

Tip: There is a bug(!) When duplicating a wrapped function model (e.g. the light model) that is not a server, the duplicates won't be shut down cleanly. If you run into this (the notebook will hang), you will need to interupt the kernel (Ctrl+C on the command line)¶

Tip: Models that are wrapped functions need to pass the `global_scope=True` keyword to interface functions (e.g. `YggRpcClient`) if they will be called more than once¶

Tip: There is a bug(!) When duplicating a wrapped function model (e.g. the light model) that is not a server, the duplicates won't be shut down cleanly. If you run into this (the notebook will hang), you will need to interupt the kernel (`Ctrl+C` on the command line)¶