本笔记本使用PairwiseEvaluator模块来查看评估LLM是否更喜欢一个查询引擎而不是另一个。
%pip install llama-index-llms-openai
# 将其附加到相同的事件循环
import nest_asyncio
nest_asyncio.apply()
# 配置日志记录器为INFO级别
import logging
import sys
logging.basicConfig(stream=sys.stdout, level=logging.INFO)
logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Response
from llama_index.llms.openai import OpenAI
from llama_index.core.evaluation import PairwiseComparisonEvaluator
from llama_index.core.node_parser import SentenceSplitter
import pandas as pd
pd.set_option("display.max_colwidth", 0)
使用GPT-4做评估
# gpt-4
gpt4 = OpenAI(temperature=0, model="gpt-4")
evaluator_gpt4 = PairwiseComparisonEvaluator(llm=gpt4)
documents = SimpleDirectoryReader("./test_wiki_data/").load_data()
# 创建向量索引
splitter_512 = SentenceSplitter(chunk_size=512)
vector_index1 = VectorStoreIndex.from_documents(
documents, transformations=[splitter_512]
)
splitter_128 = SentenceSplitter(chunk_size=128)
vector_index2 = VectorStoreIndex.from_documents(
documents, transformations=[splitter_128]
)
query_engine1 = vector_index1.as_query_engine(similarity_top_k=2)
query_engine2 = vector_index2.as_query_engine(similarity_top_k=8)
# 定义jupyter显示函数
def display_eval_df(query, response1, response2, eval_result) -> None:
eval_df = pd.DataFrame(
{
"查询": query,
"参考回答（答案1）": response2,
"当前回答（答案2）": response1,
"得分": eval_result.score,
"原因": eval_result.feedback,
},
index=[0],
)
eval_df = eval_df.style.set_properties(
**{
"inline-size": "300px",
"overflow-wrap": "break-word",
},
subset=["当前回答（答案2）", "参考回答（答案1）"]
)
display(eval_df)
要运行评估，您可以在查询返回的Response对象上调用.evaluate_response()函数来运行评估。让我们评估vector_index的输出。
# query_str = "How did New York City get its name?"
query_str = "What was the role of NYC during the American Revolution?"
# query_str = "Tell me about the arts and culture of NYC"
response1 = str(query_engine1.query(query_str))
response2 = str(query_engine2.query(query_str))
默认情况下，我们在成对比较中强制执行“一致性”。
我们尝试输入候选项、参考项对，然后交换两者的顺序，并确保结果仍然一致（如果不一致则返回TIE）。
eval_result = await evaluator_gpt4.aevaluate(
query_str, response=response1, reference=response2
)
INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=5536 request_id=8a8f154ee676b2e86ea24b7046e9b80b response_code=200 message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=5536 request_id=8a8f154ee676b2e86ea24b7046e9b80b response_code=200 INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=9766 request_id=dfee84227112b1311b4411492f4c8764 response_code=200 message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=9766 request_id=dfee84227112b1311b4411492f4c8764 response_code=200
display_eval_df(query_str, response1, response2, eval_result)
| Query | Reference Response (Answer 1) | Current Response (Answer 2) | Score | Reason | |
|---|---|---|---|---|---|
| 0 | What was the role of NYC during the American Revolution? | During the American Revolution, New York City served as a significant military and political base of operations for the British forces. After the Battle of Long Island in 1776, in which the Americans were defeated, the British made the city their center of operations in North America. The city was regained by the Dutch in 1673 but was renamed New York in 1674. It became the capital of the United States from 1785 to 1790. Additionally, New York City was a haven for Loyalist refugees and escaped slaves who joined the British lines for freedom. The British forces transported thousands of freedmen for resettlement in Nova Scotia and other locations, including England and the Caribbean. | During the American Revolution, New York City served as the military and political base of operations for the British in North America. It was also a haven for Loyalist refugees and escaped slaves who joined the British lines in search of freedom. The city played a significant role in the war, with the Battle of Long Island being the largest battle of the American Revolutionary War fought within its modern-day borough of Brooklyn. After the war, when the British forces evacuated, they transported freedmen to Nova Scotia, England, and the Caribbean for resettlement. | 0.500000 | It is not clear which answer is better. |
注意：默认情况下，我们通过颠倒响应/参考的顺序并确保答案相反来强制执行共识。
我们可以禁用此功能，这可能会导致更多的不一致性！
evaluator_gpt4_nc = PairwiseComparisonEvaluator(
llm=gpt4, enforce_consensus=False
)
eval_result = await evaluator_gpt4_nc.aevaluate(
query_str, response=response1, reference=response2
)
INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=6714 request_id=472a1f0829846adc1b4347ba4b99c0dd response_code=200 message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=6714 request_id=472a1f0829846adc1b4347ba4b99c0dd response_code=200
display_eval_df(query_str, response1, response2, eval_result)
| Query | Reference Response (Answer 1) | Current Response (Answer 2) | Score | Reason | |
|---|---|---|---|---|---|
| 0 | What was the role of NYC during the American Revolution? | During the American Revolution, New York City served as a significant military and political base of operations for the British forces. After the Battle of Long Island in 1776, in which the Americans were defeated, the British made the city their center of operations in North America. The city was regained by the Dutch in 1673 but was renamed New York in 1674. It became the capital of the United States from 1785 to 1790. Additionally, New York City was a haven for Loyalist refugees and escaped slaves who joined the British lines for freedom. The British forces transported thousands of freedmen for resettlement in Nova Scotia and other locations, including England and the Caribbean. | During the American Revolution, New York City served as the military and political base of operations for the British in North America. It was also a haven for Loyalist refugees and escaped slaves who joined the British lines in search of freedom. The city played a significant role in the war, with the Battle of Long Island being the largest battle of the American Revolutionary War fought within its modern-day borough of Brooklyn. After the war, when the British forces evacuated, they transported freedmen to Nova Scotia, England, and the Caribbean for resettlement. | 0.000000 | 1 Answer 1 is better because it provides more detailed information about the role of New York City during the American Revolution. It not only mentions the city's role as a British base and a haven for Loyalist refugees and escaped slaves, but also provides additional historical context such as the city being renamed and becoming the capital of the United States. |
eval_result = await evaluator_gpt4_nc.aevaluate(
query_str, response=response2, reference=response1
)
INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=9252 request_id=b73bbe6b10d878ed8138785638232866 response_code=200 message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=9252 request_id=b73bbe6b10d878ed8138785638232866 response_code=200
display_eval_df(query_str, response2, response1, eval_result)
| Query | Reference Response (Answer 1) | Current Response (Answer 2) | Score | Reason | |
|---|---|---|---|---|---|
| 0 | What was the role of NYC during the American Revolution? | During the American Revolution, New York City served as the military and political base of operations for the British in North America. It was also a haven for Loyalist refugees and escaped slaves who joined the British lines in search of freedom. The city played a significant role in the war, with the Battle of Long Island being the largest battle of the American Revolutionary War fought within its modern-day borough of Brooklyn. After the war, when the British forces evacuated, they transported freedmen to Nova Scotia, England, and the Caribbean for resettlement. | During the American Revolution, New York City served as a significant military and political base of operations for the British forces. After the Battle of Long Island in 1776, in which the Americans were defeated, the British made the city their center of operations in North America. The city was regained by the Dutch in 1673 but was renamed New York in 1674. It became the capital of the United States from 1785 to 1790. Additionally, New York City was a haven for Loyalist refugees and escaped slaves who joined the British lines for freedom. The British forces transported thousands of freedmen for resettlement in Nova Scotia and other locations, including England and the Caribbean. | 0.000000 | 1 Answer 1 is better because it directly addresses the user's query about the role of NYC during the American Revolution. It provides a more detailed and accurate account of the city's role, including its status as a British base, a haven for Loyalist refugees and escaped slaves, and the site of the Battle of Long Island. Answer 2 includes some irrelevant information about the city being regained by the Dutch and renamed, which occurred before the American Revolution, and its status as the capital of the United States, which happened after the Revolution. |
在一些更多的查询上运行
query_str = "Tell me about the arts and culture of NYC"
response1 = str(query_engine1.query(query_str))
response2 = str(query_engine2.query(query_str))
eval_result = await evaluator_gpt4.aevaluate(
query_str, response=response1, reference=response2
)
INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=6053 request_id=749fdbde59bf8d1056a8be6e211d20d9 response_code=200 message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=6053 request_id=749fdbde59bf8d1056a8be6e211d20d9 response_code=200 INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=7309 request_id=ba09bb38320b60cf09dbebb1df2c732b response_code=200 message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=7309 request_id=ba09bb38320b60cf09dbebb1df2c732b response_code=200
display_eval_df(query_str, response1, response2, eval_result)
| Query | Reference Response (Answer 1) | Current Response (Answer 2) | Score | Reason | |
|---|---|---|---|---|---|
| 0 | Tell me about the arts and culture of NYC | New York City is known for its vibrant arts and culture scene. It is home to over 2,000 arts and cultural organizations, as well as more than 500 art galleries. The city has a rich history of cultural institutions, such as Carnegie Hall and the Metropolitan Museum of Art, which are internationally renowned. The Broadway musical, a popular stage form, originated in New York City in the 1880s. The city has also been a hub for Jewish American literature and has been the birthplace of various cultural movements, including the Harlem Renaissance, abstract expressionism, and hip-hop. New York City is considered the dance capital of the world and has a thriving theater scene. The city is also known for its museums, including the Guggenheim and the Metropolitan Museum of Art, which participate in the annual Museum Mile Festival. Additionally, New York City hosts some of the world's most lucrative art auctions. Lincoln Center for the Performing Arts is a major cultural hub, housing influential arts organizations such as the Metropolitan Opera and the New York Philharmonic. Overall, New York City is often regarded as the cultural capital of the world. | New York City is known for its vibrant arts and culture scene. It is home to numerous influential arts organizations, including the Metropolitan Opera, New York City Opera, New York Philharmonic, and New York City Ballet. The city also has a thriving theater district, with Broadway shows selling billions of dollars worth of tickets each season. Additionally, there are over 2,000 arts and cultural organizations and more than 500 art galleries in the city. New York City has a rich history of cultural institutions, such as Carnegie Hall and the Metropolitan Museum of Art, which are internationally renowned. The city's arts and culture have been strongly influenced by its diverse immigrant population, and many plays and musicals are set in or inspired by New York City itself. | 0.000000 | 1 Answer 1 provides a more comprehensive and detailed response to the user's query about the arts and culture of NYC. It not only mentions the city's major cultural institutions and organizations, but also discusses the city's role in various cultural movements, its status as the dance capital of the world, its museums, and its art auctions. It also mentions the annual Museum Mile Festival, which Answer 2 does not. |