Run Inference on Multi-Region S3 Buckets¶

Prerequisites¶

terraform
python
aws cli (configured)
bacalhau cli

In [ ]:

%cd multi-region

Add your regions to regions.md¶

Note: The Bootstrap region should be first¶

In [2]:

%cat regions.md

# aws ec2 describe-regions | jq -r '.Regions[].RegionName'
# Bootstrap region must be first
us-east-1
us-west-2

In [5]:

#Replace the value of the name parameter with the name of your AMI
%python3 get_instance_amis.py --name "Deep Learning AMI Graviton GPU PyTorch 1.10.0 (Ubuntu 20.04) 20211122"

{
    "us-east-1": {
        "region": "us-east-1",
        "zone": "us-east-1a",
        "instance_ami": "ami-09ad94ebabb1d9472"
    },
    "us-west-1": {
        "region": "us-west-2",
        "zone": "us-west-2a",
        "instance_ami": "ami-09901fdae1bac6fe0"
    }
}

Replace the Values of "locations" with the values outputted from the above command¶

In [10]:

%cat ./tf/.env.example

{
    "us-east-1": {
        "region": "us-east-1",
        "zone": "us-east-1a",
        "instance_ami": "ami-09ad94ebabb1d9472"
    },
    "us-west-1": {
        "region": "us-west-2",
        "zone": "us-west-2a",
        "instance_ami": "ami-09901fdae1bac6fe0"
    }
}
  "app_tag": "bacalhau-edge-inference",
  "instance_type": "g5g.xlarge",
  "bacalhau_run_file": "bacalhau.run",
  "tailscale_key": "<your_tailscale_key>",
  "shelluser": "bacalhau-installer",
  "public_key": "~/.ssh/id_rsa.pub",
  "private_key": "~/.ssh/id_rsa"
}

Replace The values of the keys with your own key values and also the value of instance_type if you want¶

In [ ]:

%%bash
cp -r .env.example .env.json

Writting the Python Inference Script¶

For this example we use the jetson emulator SDK To install the package run

 pip install jetson-emulator

In this script we will use the googlenet model to run inference on sample images

import jetson_emulator.inference as inference
import jetson_emulator.utils as utils

# load the recognition network
net = inference.imageNet("googlenet")
for x in range(1,6):
    # emulator API to generate sample images for imageNet
    filename = net.emulatorGetImageFile()      
    img = utils.loadImage(filename) 
    class_idx, confidence = net.Classify(img)            
    class_desc = net.GetClassDesc(class_idx)            
    print("image "+str(x)+" is recognized as '{:s}' (class #{:d}) with {:f}% confidence".
    format(class_desc, class_idx, confidence*100))

Building the container¶

You can use a base image of our choice and install the dependencies we need In this case we will use the balenalib/jetson-tx2-ubuntu-python for simulating a Jetson-TX2 device

FROM balenalib/jetson-tx2-ubuntu-python

RUN pip install jetson-emulator

To Build and Push the container run this command

 sudo docker buildx build --push --platform linux/amd64,linux/arm64,linux/arm/v7 -t expanso/jetson .

Deploying the infra¶

In [ ]:

%%bash
./bulk-deploy.sh create

Running the job¶

In [ ]:

!bacalhau docker run --concurrency 2 -i https://gist.githubusercontent.com/js-ts/0474dc29b091dc6cf66d9061b2fd5838/raw/bef847a676f1978f5698e38103a355dce00af61e/script.py expanso/jetson -- python inputs/script.py

Viewing the outputs¶

In [ ]:

!bacalhau logs <YOUR-JOB-ID>

image 1 is recognized as 'pole' (class #733) with 16.345388% confidence
image 2 is recognized as 'quill, quill pen' (class #749) with 34.726549% confidence
image 3 is recognized as 'earthstar' (class #995) with 40.020620% confidence
image 4 is recognized as 'garden spider, Aranea diademata' (class #74) with 61.250917% confidence
image 5 is recognized as 'throne' (class #857) with 24.096852% confidence