#!/usr/bin/env python
# coding: utf-8

# [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/aurelio-labs/semantic-router/blob/main/docs/encoders/aurelio-bm25.ipynb) [![Open nbviewer](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/nbviewer-shield.svg)](https://nbviewer.org/github/aurelio-labs/semantic-router/blob/main/docs/encoders/aurelio-bm25.ipynb)

# # Using PineconeIndex for Hybrid Routes

# Hybrid indexes combine both sparse and dense encodings to produce more accurate results. The dense encoder allows us to search based on semantic meaning, while the sparse encoder allows us to search based on text matches. Merging both dense and sparse into a single hybrid retrieval step allows us to step up our performance beyond what dense-only or sparse-only could achieve.

# ## Getting Started

# We start by installing semantic-router. Support for the new `AurelioSparseEncoder` parameter was added in `semantic-router==0.1.0`.

# In[8]:


get_ipython().system('pip install -qU "semantic-router[pinecone]==0.1.0"')


# We start by defining a dictionary mapping routes to example phrases that should trigger those routes.

# In[1]:


from semantic_router import Route

politics = Route(
    name="politics",
    utterances=[
        "isn't politics the best thing ever",
        "why don't you tell me about your political opinions",
        "don't you just love the president",
        "don't you just hate the president",
        "they're going to destroy this country!",
        "they will save the country!",
    ],
)


# Let's define another for good measure:

# In[2]:


chitchat = Route(
    name="chitchat",
    utterances=[
        "how's the weather today?",
        "how are things going?",
        "lovely weather today",
        "the weather is horrendous",
        "let's go to the chippy",
    ],
)

routes = [politics, chitchat]


# Now we initialize our embedding models. We are going to use a hybrid index which requires both a dense and sparse encoder. For the sparse encoder we will use the pretrained `bm25` model from the Aurelio Platform and OpenAI's `text-embedding-3-small` for the dense encoder.
# 
# To get an API key for the Aurelio Platform, we head to the [Aurelio Platform](https://platform.aurelio.ai/settings/api-keys).

# In[3]:


import os
from getpass import getpass
from semantic_router.encoders.aurelio import AurelioSparseEncoder

os.environ["AURELIO_API_KEY"] = os.getenv("AURELIO_API_KEY") or getpass(
    "Enter Aurelio API Key: "
)

sparse_encoder = AurelioSparseEncoder(name="bm25")


# Sparse encoders return dictionaries containing the the indices and values of the non-zero elements in the sparse matrix.

# In[4]:


from semantic_router.encoders import OpenAIEncoder

os.environ["OPENAI_API_KEY"] = os.getenv("OPENAI_API_KEY") or getpass(
    "Enter OpenAI API Key: "
)

encoder = OpenAIEncoder(name="text-embedding-3-small", score_threshold=0.3)


# We now have both our sparse and dense encoders. When using both sparse and dense encoders we need to initialize an index that supports hybrid, such as the `HybridLocalIndex` or `PineconeIndex`.

# In[5]:


from semantic_router.index import PineconeIndex

os.environ["PINECONE_API_KEY"] = os.getenv("PINECONE_API_KEY") or getpass(
    "Enter Pinecone API Key: "
)

index = PineconeIndex(
    index_name="hybrid-test",
    dimensions=1536,
    metric="dotproduct",
)


# Now we define the `HybridRouter`. When called, the router will consume text (a query) and output the category (`Route`) it belongs to — to initialize a `HybridRouter` we need an `encoder`, `sparse_encoder` our `routes`, and the hybrid `index` we just define.

# In[6]:


from semantic_router.routers import HybridRouter

router = HybridRouter(
    encoder=encoder,
    sparse_encoder=sparse_encoder,
    routes=routes,
    index=index,
)


# Let's see if our local and remote instances are synchronized...

# In[7]:


router.is_synced()


# It seems like our `router` is not synchronized, meaning there are differences between the utterances in our local `HybridRouter` and the remote `PineconeIndex`. We can view the differences by calling `get_utterance_diff()`:

# In[8]:


router.get_utterance_diff()


# From this, we can see that every utterance is preceeded by a `-` meaning it is unique to the local `HybridRouter`. So it seems our `PineconeIndex` is missing all utterances. We can confirm this further by calling `router.index.get_utterances()` to see all utterances in the remote `PineconeIndex`:

# In[9]:


router.index.get_utterances()


# As expected, we have no utterances in the remote `PineconeIndex`. The reason for this is that when initializing our `HybridRouter` we did not specify an `auto_sync` parameter, so `auto_sync` defaulted to `None`. When `auto_sync=None` no synchronization is performed during initialization. Let's try again with `auto_sync="local"`, meaning take what we have locally and overwrite the remote `PineconeIndex` with these local values.

# In[10]:


router = HybridRouter(
    encoder=encoder,
    sparse_encoder=sparse_encoder,
    routes=routes,
    index=index,
    auto_sync="local",
)


# Now let's check our sync state:

# In[11]:


router.is_synced()


# In[12]:


router.get_utterance_diff()


# ... NEED TO FINISH HERE

# In[13]:


router("it's raining cats and dogs today")


# In[14]:


router("I'm interested in learning about llama 2")


# In this case, we return `None` because no matches were identified. We always recommend optimizing your `RouteLayer` for optimal performance, you can see how in [this notebook](https://github.com/aurelio-labs/semantic-router/blob/main/docs/06-threshold-optimization.ipynb).

# ---