pykoi
provides simple UI to launch a chatbot UI based on your LLMs, including your own finetuned LLM, a pretrained LLM from huggingface, or OpenAI/Anthropic/Bedrock APIs. This demo shows how to create and launch an LLM chatbot UI and database (with Database) from Open-source LLMs from Huggingface. Let's get started!
To run this jupyter notebook, you need a pykoi
environment with the huggingface
option. You can follow the installation guide to set up the environment.
You may also need pip install ipykernel
to run the kernel environment.
If you are a normal user of pykoi
, you can skip this step. However, if you modify the pykoi code and want to test your changes, you can uncomment the code below.
# %reload_ext autoreload
# %autoreload 2
# import sys
# sys.path.append(".")
# sys.path.append("..")
# sys.path.append("../..")
## pip install ipykernel
from pykoi import Application
from pykoi.chat import ModelFactory
from pykoi.chat import QuestionAnswerDatabase
from pykoi.component import Chatbot, Dashboard
To a launch a pykoi App, you only need to customize 3 components: a model, a database and a chatbot. Let's try it out!
qa_database = QuestionAnswerDatabase()
model = ModelFactory.create_model(
model_source="huggingface",
pretrained_model_name_or_path="tiiuae/falcon-7b",
trust_remote_code=True, ## TODO: set as default
load_in_8bit=True
)
/opt/conda/envs/pykoi/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm
[HuggingfaceModel] loading model...
WARNING: You are currently loading Falcon using legacy code contained in the model repository. Falcon has now been fully ported into the Hugging Face transformers library. For the most up-to-date and high-performance version of the Falcon model code, please update to the latest version of transformers and then load the model without the trust_remote_code=True argument. Loading checkpoint shards: 100%|██████████| 2/2 [00:36<00:00, 18.18s/it]
[HuggingfaceModel] loading tokenizer...
chatbot = Chatbot(model=model, feedback="vote")
Now we can launch the chatbot using the model and database. Once you run the below line, you will see a Tunnel link which ends with ngrok-free.app
. Click this link and you can see the below interface:
You may need to set a ngrok token (one time) by uncomment the below line, add your personal ngrok token and run this ngrok config
line.
# !ngrok config add-authtoken xxx
nest_asyncio
¶Add nest_asyncio
to avoid error such as asyncio.run() cannot be called from a running event loop
. Since we're running another interface inside a Jupyter notebook where an asyncio event loop is already running, we'll encounter the error. (since The uvicorn.run() function uses asyncio.run(), which isn't compatible with a running event loop.)
# !pip install -q nest_asyncio
import nest_asyncio
nest_asyncio.apply()
app = Application(debug=False, share=True)
app.add_component(chatbot)
app.run()
INFO: Started server process [7578] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:5000 (Press CTRL+C to quit)
Public URL: https://a63d9b47dea54a.lhr.life
Once you collect enough data via the chatbot app above, you can visualize your data via the dashboard.
qa_dashboard = Dashboard(database=qa_database)
app = Application(debug=False, share=True)
app.add_component(chatbot)
app.add_component(qa_dashboard)
app.run()
You will only see one localhost link after you run the app.run()
below. If you are on EC2 or a remote server, you can tunnel it back to your laptop via the following options:
Now we can launch the dashboard. Click this above link and you can see the below interface: