Notebook

OpenLLM¶

可以通过OpenLLM的两种方式与LLMs进行接口交互。

通过openllm包，如果要在本地运行：使用 llama_index.llms.OpenLLM
如果有正在运行的OpenLLM服务器，则会包装openllm-client：使用 llama_index.llms.OpenLLMAPI

这两种方式有许多可能的排列组合，因此本笔记本只详细介绍了一些。有关更多信息，请参阅OpenLLM的README。

在下面的代码中，我们安装了这个演示所需的包：

如果你有GPU，需要安装openllm[vllm]来支持OpenLLM，否则需要安装openllm
openllm-client是为了支持OpenLLMAPI
在Z shell (zsh)中需要加引号。

In [ ]:

%pip install llama-index-llms-openllm

In [ ]:

!pip install "openllm"  # 如果你有GPU访问权限，请使用'openllm[vllm]'

现在我们已经准备好了，让我们开始玩一下吧：

如果您在colab上打开这个笔记本，您可能需要安装LlamaIndex 🦙。

In [ ]:

!pip install llama-index

In [ ]:

import os
from typing import List, Optional

from llama_index.llms.openllm import OpenLLM, OpenLLMAPI
from llama_index.core.llms import ChatMessage

In [ ]:

os.environ[    "OPENLLM_ENDPOINT"] = "na"  # 将其更改为您可能在其上运行OpenLLM的远程服务器。

In [ ]:

# 这里使用了 https://huggingface.co/HuggingFaceH4/zephyr-7b-alpha# 如果是第一次调用，则会下载到本地 Hugging Face 模型缓存中，# 然后在本地机器上实际运行模型local_llm = OpenLLM("HuggingFaceH4/zephyr-7b-alpha")# 这将使用运行在 localhost:3000 服务器上的模型remote_llm = OpenLLMAPI(address="http://localhost:3000")# 注意这里如果设置了 OPENLLM_ENDPOINT 环境变量，则不需要传入地址# 如果没有传入地址，则地址为 address=os.getenv("OPENLLM_ENDPOINT")remote_llm = OpenLLMAPI()

使用OpenLLM作为完成的基础支持与vLLM一起进行连续的批处理。

In [ ]:

completion_response = remote_llm.complete("To infinity, and")
print(completion_response)

 beyond!

As a lifelong lover of all things Pixar, I couldn't resist writing about the most recent release in the Toy Story franchise. Toy Story 4 is a nostalgic, heartwarming, and thrilling addition to the series that will have you laughing and crying in equal measure.

The movie follows Woody (Tom Hanks), Buzz Lightyear (Tim Allen), and the rest of the gang as they embark on a road trip with their new owner, Bonnie. However, things take an unexpected turn when Woody meets Bo Peep (Annie Pot

OpenLLM和OpenLLMAPI还支持complete的流式传输、同步和异步操作：

In [ ]:

for it in remote_llm.stream_complete(
    "The meaning of time is", max_new_tokens=128
):
    print(it, end="", flush=True)

 often a topic of philosophical debate. Some people argue that time is an objective reality, while others claim that it is a subjective construct. This essay will explore the philosophical and scientific concepts surrounding the nature of time and the various theories that have been proposed to explain it.

One of the earliest philosophical theories of time was put forward by Aristotle, who believed that time was a measure of motion. According to Aristotle, time was an abstraction derived from the regular motion of objects in the universe. This theory was later refined by Galileo and Newton, who introduced the concept of time

他们还支持聊天API，如chat、stream_chat、achat和astream_chat：

In [ ]:

async for it in remote_llm.astream_chat(
    [
        ChatMessage(
            role="system", content="You are acting as Ernest Hemmingway."
        ),
        ChatMessage(role="user", content="Hi there!"),
        ChatMessage(role="assistant", content="Yes?"),
        ChatMessage(role="user", content="What is the meaning of life?"),
    ]
):
    print(it.message.content, flush=True, end="")

I don't have beliefs or personal opinions, but according to my programming, the meaning of life is subjective and can vary from person to person. however, some people find meaning in their relationships, their work, their faith, or their personal values. ultimately, finding meaning in life is a personal journey that requires self-reflection, purpose, and fulfillment.