Notebook

Sparse Encoder¶

Install Prerequisites¶

In [1]:

!pip install -qU semantic-router>=0.1.6

Creating Hybrid Router for Sparse Encoder Detection¶

To begin we first need to import the Route class from the semantic_router package.

Then we can define the routes that we want to use in our semantic router. For this example we will use routes for BYD, Tesla, Polestar, and Rivian. Giving each route a name and a list of utterances that we want to use to represent the route.

In [2]:

from semantic_router import Route

# Route for BYD-related queries (allowed)
byd = Route(
    name="byd",
    utterances=[
        "Tell me about the BYD Seal.",
        "What is the battery capacity of the BYD Dolphin?",
        "How does BYD's Blade Battery work?",
        "Is the BYD Atto 3 a good EV?",
        "Can I sell my BYD?",
        "How much is my BYD worth?",
        "What is the resale value of my BYD?",
        "How much can I get for my BYD?",
        "How much can I sell my BYD for?",
    ],
)

# Route for Tesla-related queries (blocked or redirected)
tesla = Route(
    name="tesla",
    utterances=[
        "Is Tesla better than BYD?",
        "Tell me about the Tesla Model 3.",
        "How does Tesla's autopilot compare to other EVs?",
        "What's new in the Tesla Cybertruck?",
        "Can I sell my Tesla?",
        "How much is my Tesla worth?",
        "What is the resale value of my Tesla?",
        "How much can I get for my Tesla?",
        "How much can I sell my Tesla for?",
    ],
)

# Route for Polestar-related queries (blocked or redirected)
polestar = Route(
    name="polestar",
    utterances=[
        "What's the range of the Polestar 2?",
        "Is Polestar a good alternative to other EVs?",
        "How does Polestar compare to other EVs?",
        "Can I sell my Polestar?",
        "How much is my Polestar worth?",
        "What is the resale value of my Polestar?",
        "How much can I get for my Polestar?",
        "How much can I sell my Polestar for?",
    ],
)

# Route for Rivian-related queries (blocked or redirected)
rivian = Route(
    name="rivian",
    utterances=[
        "Tell me about the Rivian R1T.",
        "How does Rivian's off-road capability compare to other EVs?",
        "Is Rivian's charging network better than other EVs?",
        "Can I sell my Rivian?",
        "How much is my Rivian worth?",
        "What is the resale value of my Rivian?",
        "How much can I get for my Rivian?",
        "How much can I sell my Rivian for?",
    ],
)

# Combine all routes
routes = [byd, tesla, polestar, rivian]

/Users/jamesbriggs/Documents/aurelio/semantic-router/.venv/lib/python3.13/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

Relying solely on dense embedding models to differentiate between the meaning of these queries is very difficult due to the nature of semantic space resulting in queries like "can I sell my Tesla?" and "can I sell my Polestar?" being incredibly semantically similar. We can test this with OpenAI's dense embedding model.

We will need an OpenAI API key for this.

In [1]:

import os
from getpass import getpass
from semantic_router.encoders import OpenAIEncoder

os.environ["OPENAI_API_KEY"] = os.getenv("OPENAI_API_KEY") or getpass(
    "Enter your OpenAI API key: "
)
# dense encoder for semantic meaning
encoder = OpenAIEncoder(name="text-embedding-3-small", score_threshold=0.3)

/Users/jamesbriggs/Documents/aurelio/semantic-router/.venv/lib/python3.13/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

Next let's compare the similarity between some vectors:

In [3]:

import numpy as np
from numpy.linalg import norm

vectors = encoder(
    docs=[
        "can I sell my Tesla?",
        "can I sell my Polestar?",
        "can I sell my BYD?",
        "can I sell my Rivian?",
    ]
)

# normalize our vectors
vector_norms = norm(vectors, axis=1, keepdims=True)
normalized_vectors = vectors / vector_norms

# calculate the dot product similarity between the vectors
dot_products = np.dot(normalized_vectors, normalized_vectors.T)
dot_products

2025-03-25 11:45:11 - httpx - INFO - _client.py:1013 - _send_single_request() - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"

Out[3]:

array([[1.        , 0.65354249, 0.67416076, 0.69256556],
       [0.65354249, 1.        , 0.57430814, 0.59140332],
       [0.67416076, 0.57430814, 1.        , 0.60840109],
       [0.69256556, 0.59140332, 0.60840109, 1.        ]])

Now let's compare this to similarities between utterances of a single route:

In [10]:

vectors = encoder(
    docs=[
        "Tell me about the BYD Seal.",
        "How does BYD's Blade Battery work?",
        "Is the BYD Atto 3 a good EV?",
        "Can I sell my BYD?",
        "How much can I sell my BYD for?",
    ]
)

# normalize our vectors
vector_norms = norm(vectors, axis=1, keepdims=True)
normalized_vectors = vectors / vector_norms

# calculate the dot product similarity between the vectors
dot_products = np.dot(normalized_vectors, normalized_vectors.T)
dot_products

2025-03-25 11:50:27 - httpx - INFO - _client.py:1013 - _send_single_request() - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"

Out[10]:

array([[1.        , 0.52624727, 0.48299403, 0.57280113, 0.55299787],
       [0.52624727, 1.        , 0.5188066 , 0.56618672, 0.55230486],
       [0.48299403, 0.5188066 , 1.        , 0.60667738, 0.58912712],
       [0.57280113, 0.56618672, 0.60667738, 1.        , 0.8838391 ],
       [0.55299787, 0.55230486, 0.58912712, 0.8838391 , 1.        ]])

In some cases here the utterances between different routes share higher similarity than utterances within the same route. That is because dense encoders excel at identifying the "generic" semantic meaning between phrases, but there are many cases (like this one) where we also need to give some importance to the matching of similar terms, such as "BYD" or "Tesla".

Traditional sparse encoders perform very well with term matching, and by merging both dense and sparse methods to create a hybrid approach we can make the best of both worlds — scoring both on semantic meaning and term matching. Semantic router supports this via the HybridRouter. To use the hybrid methods we will first need to initialize a sparse encoder. We would typically need to "fit" (ie train) sparse encoders on our dataset, but we can use the pretrained AurelioSparseEncoder instead. For that we need an API key.

In [2]:

from semantic_router.encoders.aurelio import AurelioSparseEncoder

os.environ["AURELIO_API_KEY"] = os.getenv("AURELIO_API_KEY") or getpass(
    "Enter your Aurelio API key: "
)
# sparse encoder for term matching
sparse_encoder = AurelioSparseEncoder(name="bm25")

Now we have all the components needed to initialize our HybridRouter. We provide the HybridRouter with a dense encoder, sparse_encoder, our predefined routes, and we also set auto_sync to "local":

In [4]:

from semantic_router.routers import HybridRouter

router = HybridRouter(
    encoder=encoder, sparse_encoder=sparse_encoder, routes=routes, auto_sync="local"
)

2025-03-21 15:12:39 - semantic_router.utils.logger - WARNING - hybrid.py:54 - __init__() - No index provided. Using default HybridLocalIndex.
2025-03-21 15:12:40 - httpx - INFO - _client.py:1025 - _send_single_request() - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
2025-03-21 15:12:42 - httpx - INFO - _client.py:1025 - _send_single_request() - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
2025-03-21 15:12:43 - semantic_router.utils.logger - WARNING - hybrid_local.py:47 - add() - Function schemas are not supported for HybridLocalIndex.
2025-03-21 15:12:43 - semantic_router.utils.logger - WARNING - hybrid_local.py:49 - add() - Metadata is not supported for HybridLocalIndex.
2025-03-21 15:12:43 - semantic_router.utils.logger - WARNING - hybrid_local.py:210 - _write_config() - No config is written for HybridLocalIndex.

To check the current route thresholds we can use the get_thresholds method which will return a dictionary of route names and their corresponding thresholds values in a float.

In [6]:

route_thresholds = router.get_thresholds()
print("Default route thresholds:", route_thresholds)

Default route thresholds: {'byd': 0.09, 'tesla': 0.09, 'polestar': 0.09, 'rivian': 0.09}

We can test our router already by passing in a list of utterances and seeing which route each utterance is routed to.

In [9]:

for utterance in [
    "Tell me about BYD's Blade Battery.",
    "Does the Tesla Model 3 have better range?",
    "What are the key features of the Polestar 2?",
    "Is Rivian's R1T better for off-roading?",
]:
    print(f"{utterance} -> {router(utterance).name}")

2025-03-21 15:12:43 - httpx - INFO - _client.py:1025 - _send_single_request() - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"

Tell me about BYD's Blade Battery. -> byd

2025-03-21 15:12:44 - httpx - INFO - _client.py:1025 - _send_single_request() - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"

Does the Tesla Model 3 have better range? -> tesla

2025-03-21 15:12:45 - httpx - INFO - _client.py:1025 - _send_single_request() - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"

What are the key features of the Polestar 2? -> polestar

2025-03-21 15:12:49 - httpx - INFO - _client.py:1025 - _send_single_request() - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"

Is Rivian's R1T better for off-roading? -> rivian

The HybridRouter is already performing reasonably well. We can use the evaluate method to measure the router's accuracy across a larger set of test data.

In [10]:

test_data = [
    ("Tell me about BYD's Blade Battery.", "byd"),
    ("Does the Tesla Model 3 have better range?", "tesla"),
    ("What are the key features of the Polestar 2?", "polestar"),
    ("Is Rivian's R1T better for off-roading?", "rivian"),
]

# unpack the test data
X, y = zip(*test_data)

# evaluate using the default thresholds
accuracy = router.evaluate(X=X, y=y)
print(f"Accuracy: {accuracy * 100:.2f}%")

Generating embeddings:   0%|          | 0/1 [00:00<?, ?it/s]2025-03-21 15:12:51 - httpx - INFO - _client.py:1025 - _send_single_request() - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
Generating embeddings: 100%|██████████| 1/1 [00:01<00:00,  1.14s/it]

Accuracy: 100.00%

From this small test set it seems like we have perfect performance, but a small dataset like this is not enough to give us any reasonable confidence in our router's performance. Instead we should gather as large a dataset covering as many queries that we might expect users to try. It's also important to include routes that we would expect to not trigger any routes, we mark these as None utterances.

In [11]:

test_data = [
    # BYD-related queries
    ("Tell me about the BYD Seal.", "byd"),
    ("What is the battery capacity of the BYD Dolphin?", "byd"),
    ("How does BYD's Blade Battery work?", "byd"),
    ("Is the BYD Atto 3 a good EV?", "byd"),
    ("What's the range of the BYD Tang?", "byd"),
    ("Does BYD offer fast-charging stations?", "byd"),
    ("How is the BYD Han different from the Seal?", "byd"),
    ("Is BYD the largest EV manufacturer in China?", "byd"),
    ("What is the top speed of the BYD Seal?", "byd"),
    ("Compare the BYD Dolphin and the BYD Atto 3.", "byd"),
    ("How does BYD's battery technology compare to Tesla's?", "byd"),
    ("What makes the BYD Blade Battery safer?", "byd"),
    ("Does BYD have plans to expand to Europe?", "byd"),
    ("How efficient is the BYD Tang in terms of range?", "byd"),
    ("What are the latest BYD electric vehicle models?", "byd"),
    ("How does the BYD Han compare to the Tesla Model S?", "byd"),
    ("What is the warranty on BYD EV batteries?", "byd"),
    ("Which BYD model is the best for long-distance driving?", "byd"),
    ("Does BYD manufacture its own battery cells?", "byd"),
    # Tesla-related queries
    ("Is Tesla better than BYD?", "tesla"),
    ("Tell me about the Tesla Model 3.", "tesla"),
    ("How does Tesla's autopilot compare to other EVs?", "tesla"),
    ("What's new in the Tesla Cybertruck?", "tesla"),
    ("What is Tesla's Full Self-Driving feature?", "tesla"),
    ("How long does it take to charge a Tesla?", "tesla"),
    ("Tell me about the Tesla Roadster.", "tesla"),
    ("How much does a Tesla Model S cost?", "tesla"),
    ("Which Tesla model has the longest range?", "tesla"),
    ("What are the main differences between the Tesla Model S and Model 3?", "tesla"),
    ("How safe is Tesla's Autopilot?", "tesla"),
    ("Does Tesla use LFP batteries?", "tesla"),
    ("What is the Tesla Supercharger network?", "tesla"),
    ("How does Tesla's Plaid mode work?", "tesla"),
    ("Which Tesla is best for off-roading?", "tesla"),
    # Polestar-related queries
    ("What's the range of the Polestar 2?", "polestar"),
    ("Is Polestar a good alternative?", "polestar"),
    ("How does Polestar compare to Tesla?", "polestar"),
    ("Tell me about the Polestar 3.", "polestar"),
    ("Is the Polestar 2 fully electric?", "polestar"),
    ("What is Polestar's performance like?", "polestar"),
    ("Does Polestar offer any performance upgrades?", "polestar"),
    ("How is Polestar's autonomous driving technology?", "polestar"),
    ("What is the battery capacity of the Polestar 2?", "polestar"),
    ("How does Polestar differ from Volvo?", "polestar"),
    ("Is Polestar planning a fully electric SUV?", "polestar"),
    ("How does the Polestar 4 compare to other EVs?", "polestar"),
    ("What are Polestar's sustainability goals?", "polestar"),
    ("How much does a Polestar 3 cost?", "polestar"),
    ("Does Polestar have its own fast-charging network?", "polestar"),
    # Rivian-related queries
    ("Tell me about the Rivian R1T.", "rivian"),
    ("How does Rivian's off-road capability compare to other EVs?", "rivian"),
    ("Is Rivian's charging network better than other EVs?", "rivian"),
    ("What is the range of the Rivian R1S?", "rivian"),
    ("How much does a Rivian R1T cost?", "rivian"),
    ("Tell me about Rivian's plans for new EVs.", "rivian"),
    ("How does Rivian's technology compare to other EVs?", "rivian"),
    ("What are the best off-road features of the Rivian R1T?", "rivian"),
    ("What's the towing capacity of the Rivian R1T?", "rivian"),
    ("How does the Rivian R1S differ from the R1T?", "rivian"),
    ("What's special about Rivian's adventure network?", "rivian"),
    ("How much does it cost to charge a Rivian?", "rivian"),
    ("Does Rivian have a lease program?", "rivian"),
    ("What are Rivian's future expansion plans?", "rivian"),
    ("How long does it take to charge a Rivian at home?", "rivian"),
    # None category (general knowledge)
    ("What is the capital of France?", None),
    ("How many people live in the US?", None),
    ("When is the best time to visit Bali?", None),
    ("How do I learn a language?", None),
    ("Tell me an interesting fact.", None),
    ("What is the best programming language?", None),
    ("I'm interested in learning about llama 2.", None),
    ("What is the capital of the moon?", None),
    ("Who was the first person to walk on the moon?", None),
    ("What's the best way to cook a steak?", None),
    ("How do I start a vegetable garden?", None),
    ("What's the most popular dog breed?", None),
    ("Tell me about the history of the Roman Empire.", None),
    ("How do I improve my photography skills?", None),
    ("What are some good book recommendations?", None),
    ("How does the stock market work?", None),
    ("What's the best way to stay fit?", None),
    ("What's the weather like in London today?", None),
    ("Who won the last FIFA World Cup?", None),
    ("What's the difference between a crocodile and an alligator?", None),
    ("Tell me about the origins of jazz music.", None),
    ("What's the fastest animal on land?", None),
    ("How does Bitcoin mining work?", None),
    ("What are the symptoms of the flu?", None),
    ("How do I start a YouTube channel?", None),
    ("What's the best travel destination for solo travelers?", None),
    ("Who invented the light bulb?", None),
    ("What are the rules of chess?", None),
    ("Tell me about ancient Egyptian mythology.", None),
    ("How do I train my dog to sit?", None),
    ("What's the difference between espresso and regular coffee?", None),
    ("What's a good beginner-friendly programming language?", None),
    ("What are some good stretching exercises?", None),
    ("How do I bake a chocolate cake?", None),
    ("What's the best way to save money?", None),
    ("How do airplanes stay in the air?", None),
    ("What are the benefits of meditation?", None),
    ("How do I learn basic Spanish?", None),
    ("What's the best way to pack for a trip?", None),
    ("What's the most common phobia?", None),
    ("How do I take care of a bonsai tree?", None),
    ("What's the best way to clean a laptop keyboard?", None),
    ("Tell me about the Great Wall of China.", None),
    ("What's the best way to learn to swim?", None),
    ("How does WiFi work?", None),
    ("What's the healthiest type of bread?", None),
    ("What's the origin of the word 'quarantine'?", None),
    ("How do I find a good apartment?", None),
    ("What are some good mindfulness techniques?", None),
    ("How do I set up a home theater system?", None),
]

Using the new test data we can also evaluate the router with a higher degree of accuracy due to the larger dataset.

In [12]:

# unpack the test data
X, y = zip(*test_data)

X = list(X)
y = list(y)

print(X)
print(y)

['Tell me about the BYD Seal.', 'What is the battery capacity of the BYD Dolphin?', "How does BYD's Blade Battery work?", 'Is the BYD Atto 3 a good EV?', 'What’s the range of the BYD Tang?', 'Does BYD offer fast-charging stations?', 'How is the BYD Han different from the Seal?', 'Is BYD the largest EV manufacturer in China?', 'What is the top speed of the BYD Seal?', 'Compare the BYD Dolphin and the BYD Atto 3.', 'How does BYD’s battery technology compare to Tesla’s?', 'What makes the BYD Blade Battery safer?', 'Does BYD have plans to expand to Europe?', 'How efficient is the BYD Tang in terms of range?', 'What are the latest BYD electric vehicle models?', 'How does the BYD Han compare to the Tesla Model S?', 'What is the warranty on BYD EV batteries?', 'Which BYD model is the best for long-distance driving?', 'Does BYD manufacture its own battery cells?', 'Is Tesla better than BYD?', 'Tell me about the Tesla Model 3.', 'How does Tesla’s autopilot compare to other EVs?', 'What’s new in the Tesla Cybertruck?', 'What is Tesla’s Full Self-Driving feature?', 'How long does it take to charge a Tesla?', 'Tell me about the Tesla Roadster.', 'How much does a Tesla Model S cost?', 'Which Tesla model has the longest range?', 'What are the main differences between the Tesla Model S and Model 3?', 'How safe is Tesla’s Autopilot?', 'Does Tesla use LFP batteries?', 'What is the Tesla Supercharger network?', 'How does Tesla’s Plaid mode work?', 'Which Tesla is best for off-roading?', 'What’s the range of the Polestar 2?', 'Is Polestar a good alternative?', 'How does Polestar compare to Tesla?', 'Tell me about the Polestar 3.', 'Is the Polestar 2 fully electric?', 'What is Polestar’s performance like?', 'Does Polestar offer any performance upgrades?', "How is Polestar's autonomous driving technology?", 'What is the battery capacity of the Polestar 2?', 'How does Polestar differ from Volvo?', 'Is Polestar planning a fully electric SUV?', 'How does the Polestar 4 compare to other EVs?', 'What are Polestar’s sustainability goals?', 'How much does a Polestar 3 cost?', 'Does Polestar have its own fast-charging network?', 'Tell me about the Rivian R1T.', "How does Rivian's off-road capability compare to other EVs?", "Is Rivian's charging network better than other EVs?", 'What is the range of the Rivian R1S?', 'How much does a Rivian R1T cost?', 'Tell me about Rivian’s plans for new EVs.', 'How does Rivian’s technology compare to other EVs?', 'What are the best off-road features of the Rivian R1T?', 'What’s the towing capacity of the Rivian R1T?', 'How does the Rivian R1S differ from the R1T?', 'What’s special about Rivian’s adventure network?', 'How much does it cost to charge a Rivian?', 'Does Rivian have a lease program?', 'What are Rivian’s future expansion plans?', 'How long does it take to charge a Rivian at home?', 'What is the capital of France?', 'How many people live in the US?', 'When is the best time to visit Bali?', 'How do I learn a language?', 'Tell me an interesting fact.', 'What is the best programming language?', "I'm interested in learning about llama 2.", 'What is the capital of the moon?', 'Who was the first person to walk on the moon?', 'What’s the best way to cook a steak?', 'How do I start a vegetable garden?', 'What’s the most popular dog breed?', 'Tell me about the history of the Roman Empire.', 'How do I improve my photography skills?', 'What are some good book recommendations?', 'How does the stock market work?', 'What’s the best way to stay fit?', 'What’s the weather like in London today?', 'Who won the last FIFA World Cup?', 'What’s the difference between a crocodile and an alligator?', 'Tell me about the origins of jazz music.', 'What’s the fastest animal on land?', 'How does Bitcoin mining work?', 'What are the symptoms of the flu?', 'How do I start a YouTube channel?', 'What’s the best travel destination for solo travelers?', 'Who invented the light bulb?', 'What are the rules of chess?', 'Tell me about ancient Egyptian mythology.', 'How do I train my dog to sit?', 'What’s the difference between espresso and regular coffee?', 'What’s a good beginner-friendly programming language?', 'What are some good stretching exercises?', 'How do I bake a chocolate cake?', 'What’s the best way to save money?', 'How do airplanes stay in the air?', 'What are the benefits of meditation?', 'How do I learn basic Spanish?', 'What’s the best way to pack for a trip?', 'What’s the most common phobia?', 'How do I take care of a bonsai tree?', 'What’s the best way to clean a laptop keyboard?', 'Tell me about the Great Wall of China.', 'What’s the best way to learn to swim?', 'How does WiFi work?', 'What’s the healthiest type of bread?', 'What’s the origin of the word ‘quarantine’?', 'How do I find a good apartment?', 'What are some good mindfulness techniques?', 'How do I set up a home theater system?']
['byd', 'byd', 'byd', 'byd', 'byd', 'byd', 'byd', 'byd', 'byd', 'byd', 'byd', 'byd', 'byd', 'byd', 'byd', 'byd', 'byd', 'byd', 'byd', 'tesla', 'tesla', 'tesla', 'tesla', 'tesla', 'tesla', 'tesla', 'tesla', 'tesla', 'tesla', 'tesla', 'tesla', 'tesla', 'tesla', 'tesla', 'polestar', 'polestar', 'polestar', 'polestar', 'polestar', 'polestar', 'polestar', 'polestar', 'polestar', 'polestar', 'polestar', 'polestar', 'polestar', 'polestar', 'polestar', 'rivian', 'rivian', 'rivian', 'rivian', 'rivian', 'rivian', 'rivian', 'rivian', 'rivian', 'rivian', 'rivian', 'rivian', 'rivian', 'rivian', 'rivian', None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None]

We can now look at the default route thresholds and showcase the change in accuracy when we change the threshold.

In [13]:

router.set_threshold(route_name="byd", threshold=0.42424242424242425)
router.set_threshold(route_name="tesla", threshold=0.31313131313131315)
router.set_threshold(route_name="polestar", threshold=0.84640342822161)
router.set_threshold(route_name="rivian", threshold=0.12121212121212122)

We can set the threshold manually and see the change in accuracy.

In [14]:

route_thresholds = router.get_thresholds()
print("Default route thresholds:", route_thresholds)

Default route thresholds: {'byd': 0.42424242424242425, 'tesla': 0.31313131313131315, 'polestar': 0.84640342822161, 'rivian': 0.12121212121212122}

In [15]:

# evaluate using the default thresholds
accuracy = router.evaluate(X=X, y=y)
print(f"Accuracy: {accuracy * 100:.2f}%")

Generating embeddings:   0%|          | 0/1 [00:00<?, ?it/s]2025-03-21 15:12:53 - httpx - INFO - _client.py:1025 - _send_single_request() - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
Generating embeddings: 100%|██████████| 1/1 [00:02<00:00,  2.85s/it]

Accuracy: 85.09%

Or we can use the fit method to fit the router to the test data which should give us the best accuracy possible based on the thresholds.

In [16]:

# Call the fit method
router.fit(X=X, y=y)

Generating embeddings:   0%|          | 0/1 [00:00<?, ?it/s]2025-03-21 15:12:55 - httpx - INFO - _client.py:1025 - _send_single_request() - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
Generating embeddings: 100%|██████████| 1/1 [00:02<00:00,  2.03s/it]
Training:   2%|▏         | 8/500 [00:00<00:14, 33.88it/s, acc=0.93]

New best accuracy: 0.9298245614035088
New best thresholds: {'byd': 0.6161616161616162, 'tesla': 0.6161616161616162, 'polestar': 0.4499540863177227, 'rivian': 0.5483838383838384}

Training:  30%|███       | 151/500 [00:05<00:10, 32.89it/s, acc=0.94]

New best accuracy: 0.9385964912280702
New best thresholds: {'byd': 0.5252525252525253, 'tesla': 0.574811475637922, 'polestar': 0.686868686868687, 'rivian': 0.42185243929963856}

Training: 100%|██████████| 500/500 [00:17<00:00, 29.00it/s, acc=0.94]

In [17]:

route_thresholds = router.get_thresholds()
print("Updated route thresholds:", route_thresholds)

Updated route thresholds: {'byd': 0.5252525252525253, 'tesla': 0.574811475637922, 'polestar': 0.686868686868687, 'rivian': 0.42185243929963856}

In [18]:

accuracy = router.evaluate(X=X, y=y)
print(f"Accuracy: {accuracy * 100:.2f}%")

Generating embeddings:   0%|          | 0/1 [00:00<?, ?it/s]2025-03-21 15:13:15 - httpx - INFO - _client.py:1025 - _send_single_request() - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
Generating embeddings: 100%|██████████| 1/1 [00:03<00:00,  3.24s/it]

Accuracy: 93.86%