SciUnit is a framework for validating scientific models by creating experimental-data-driven unit tests.¶

Chapter 4. Example of RunnableModel and Backend¶

If you are using this file in Google Colab, this block of code can help you install sciunit from PyPI in Colab environment.¶

In [ ]:

try:
    import google.colab
    IN_COLAB = True
except:
    IN_COLAB = False
if IN_COLAB:
    !pip install -q sciunit

Beside the usual model in previous sections, let’s create a model that run a Backend instance to simulate and obtain results.

Firstly, import necessary components from SciUnit package.

In [ ]:

import sciunit, random
from sciunit import Test
from sciunit.capabilities import Runnable
from sciunit.scores import BooleanScore
from sciunit.models import RunnableModel
from sciunit.models.backends import register_backends, Backend

Let’s define subclasses of SciUnit Backend, Test, and Model.

Note that:

A SciUnit Backend subclass should implement _backend_run method.
A SciUnit Backend subclass should implement run method.

In [ ]:

class RandomNumBackend(Backend):
    '''generate a random integer between min and max'''

    def set_run_params(self, **run_params):

        # get min from run_params, if not exist, then 0.
        self.min = run_params.get('min', 0)

        # get max from run_params, if not exist, then self.min + 100.
        self.max = run_params.get('max', self.min + 100)

    def _backend_run(self):
        # generate and return random integer between min and max.
        return random.randint(self.min, self.max)

class RandomNumModel(RunnableModel):
    """A model that always produces a constant number as output."""

    def run(self):
        self.results = self._backend.backend_run()


class RangeTest(Test):
    """Tests if the model predicts the same number as the observation."""

    # Default Runnable Capability for RunnableModel
    required_capabilities = (Runnable,)

    # This test's 'judge' method will return a BooleanScore.
    score_type = BooleanScore

    def generate_prediction(self, model):
        model.run()
        return model.results

    def compute_score(self, observation, prediction):
        score = BooleanScore(
            observation['min'] <= prediction and observation['max'] >= prediction
        )
        return score

Let’s define the model instance named model 1.

In [ ]:

model = RandomNumModel("model 1")

We must register any backend isntance in order to use it in model instances.

set_backend and set_run_params methods can help us to set the run-parameters in the model and its backend.

In [ ]:

register_backends({"Random Number": RandomNumBackend})
model.set_backend("Random Number")
model.set_run_params(min=1, max=10)

Next, create an observation that requires the generated random integer between 1 and 10 and a test instance that use the observation and against the model

Then we get a more quantitative summary of the results:

In [ ]:

observation = {'min': 1, 'max': 10}
oneToTenTest = RangeTest(observation, "test 1")
score = oneToTenTest.judge(model)

print the score, and we can see the result.

In [ ]:

print(score)