Chapter 99
使用 Transformers Agents 构建具有工具调用超能力的智能体 🦸
Notebooktest225 cells
使用 Transformers Agents 构建具有工具调用超能力的智能体 🦸
作者: Aymeric Roucher
这个 notebook 展示了如何使用 [Transformers Agents](https://huggingface.co/docs/transformers/en/agents) 来构建出色的智能体！
什么是智能体？智能体是由大型语言模型（LLM）驱动的系统，它们使得 LLM（通过精心设计的提示和输出解析）能够使用特定的工具来解决问题。
这些工具基本上是 LLM 自身无法很好执行的功能：例如，对于像 Llama-3-70B 这样的文本生成 LLM，这可能是一个图像生成工具、网络搜索工具、计算器...
什么是 Transformers Agents ？它是我们 transformers 库的一个扩展，提供了构建自己的智能体的构建块！在文档 中了解更多信息。
让我们看看如何使用它，以及它能解决哪些用例。
我们从源代码安装 transformers agents ，你可以使用 pip install transformers[agents] 轻松安装。
In [1]python · cell 2
python
!pip install smolagentsOutput
Requirement already satisfied: smolagents in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (1.0.0) Requirement already satisfied: torch in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from smolagents) (2.3.0) Requirement already satisfied: torchaudio in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from smolagents) (2.3.0) Requirement already satisfied: torchvision in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from smolagents) (0.18.0) Requirement already satisfied: transformers>=4.0.0 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from smolagents) (4.47.1) Requirement already satisfied: requests>=2.32.3 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from smolagents) (2.32.3) Requirement already satisfied: rich>=13.9.4 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from smolagents) (13.9.4) Requirement already satisfied: pandas>=2.2.3 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from smolagents) (2.2.3) Requirement already satisfied: jinja2>=3.1.4 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from smolagents) (3.1.4) Requirement already satisfied: pillow>=11.0.0 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from smolagents) (11.1.0) Requirement already satisfied: markdownify>=0.14.1 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from smolagents) (0.14.1) Requirement already satisfied: gradio>=5.8.0 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from smolagents) (5.9.1) Requirement already satisfied: duckduckgo-search>=6.3.7 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from smolagents) (7.2.0) Requirement already satisfied: python-dotenv>=1.0.1 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from smolagents) (1.0.1) Requirement already satisfied: e2b-code-interpreter>=1.0.3 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from smolagents) (1.0.3) Requirement already satisfied: litellm>=1.55.10 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from smolagents) (1.57.0) Requirement already satisfied: click>=8.1.7 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from duckduckgo-search>=6.3.7->smolagents) (8.1.7) Requirement already satisfied: primp>=0.9.3 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from duckduckgo-search>=6.3.7->smolagents) (0.9.3) Requirement already satisfied: lxml>=5.3.0 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from duckduckgo-search>=6.3.7->smolagents) (5.3.0) Requirement already satisfied: attrs>=21.3.0 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from e2b-code-interpreter>=1.0.3->smolagents) (23.2.0) Requirement already satisfied: e2b<2.0.0,>=1.0.4 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from e2b-code-interpreter>=1.0.3->smolagents) (1.0.5) Requirement already satisfied: httpx<1.0.0,>=0.20.0 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from e2b-code-interpreter>=1.0.3->smolagents) (0.27.2) Requirement already satisfied: aiofiles<24.0,>=22.0 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from gradio>=5.8.0->smolagents) (23.2.1) Requirement already satisfied: anyio<5.0,>=3.0 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from gradio>=5.8.0->smolagents) (3.7.1) Requirement already satisfied: fastapi<1.0,>=0.115.2 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from gradio>=5.8.0->smolagents) (0.115.6) Requirement already satisfied: ffmpy in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from gradio>=5.8.0->smolagents) (0.4.0) Requirement already satisfied: gradio-client==1.5.2 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from gradio>=5.8.0->smolagents) (1.5.2) Requirement already satisfied: huggingface-hub>=0.25.1 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from gradio>=5.8.0->smolagents) (0.27.1) Requirement already satisfied: markupsafe~=2.0 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from gradio>=5.8.0->smolagents) (2.1.5) Requirement already satisfied: numpy<3.0,>=1.0 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from gradio>=5.8.0->smolagents) (2.1.3) Requirement already satisfied: orjson~=3.0 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from gradio>=5.8.0->smolagents) (3.10.11) Requirement already satisfied: packaging in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from gradio>=5.8.0->smolagents) (24.2) Requirement already satisfied: pydantic>=2.0 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from gradio>=5.8.0->smolagents) (2.9.2) Requirement already satisfied: pydub in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from gradio>=5.8.0->smolagents) (0.25.1) Requirement already satisfied: python-multipart>=0.0.18 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from gradio>=5.8.0->smolagents) (0.0.20) Requirement already satisfied: pyyaml<7.0,>=5.0 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from gradio>=5.8.0->smolagents) (6.0.1) Requirement already satisfied: ruff>=0.2.2 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from gradio>=5.8.0->smolagents) (0.3.4) Requirement already satisfied: safehttpx<0.2.0,>=0.1.6 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from gradio>=5.8.0->smolagents) (0.1.6) Requirement already satisfied: semantic-version~=2.0 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from gradio>=5.8.0->smolagents) (2.10.0) Requirement already satisfied: starlette<1.0,>=0.40.0 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from gradio>=5.8.0->smolagents) (0.41.3) Requirement already satisfied: tomlkit<0.14.0,>=0.12.0 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from gradio>=5.8.0->smolagents) (0.12.0) Requirement already satisfied: typer<1.0,>=0.12 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from gradio>=5.8.0->smolagents) (0.12.5) Requirement already satisfied: typing-extensions~=4.0 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from gradio>=5.8.0->smolagents) (4.12.2) Requirement already satisfied: uvicorn>=0.14.0 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from gradio>=5.8.0->smolagents) (0.30.6) Requirement already satisfied: fsspec in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from gradio-client==1.5.2->gradio>=5.8.0->smolagents) (2024.3.1) Requirement already satisfied: websockets<15.0,>=10.0 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from gradio-client==1.5.2->gradio>=5.8.0->smolagents) (12.0) Requirement already satisfied: aiohttp in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from litellm>=1.55.10->smolagents) (3.9.3) Requirement already satisfied: importlib-metadata>=6.8.0 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from litellm>=1.55.10->smolagents) (8.5.0) Requirement already satisfied: jsonschema<5.0.0,>=4.22.0 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from litellm>=1.55.10->smolagents) (4.22.0) Requirement already satisfied: openai>=1.55.3 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from litellm>=1.55.10->smolagents) (1.59.3) Requirement already satisfied: tiktoken>=0.7.0 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from litellm>=1.55.10->smolagents) (0.8.0) Requirement already satisfied: tokenizers in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from litellm>=1.55.10->smolagents) (0.21.0) Requirement already satisfied: beautifulsoup4<5,>=4.9 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from markdownify>=0.14.1->smolagents) (4.12.3) Requirement already satisfied: six<2,>=1.15 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from markdownify>=0.14.1->smolagents) (1.16.0) Requirement already satisfied: python-dateutil>=2.8.2 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from pandas>=2.2.3->smolagents) (2.9.0.post0) Requirement already satisfied: pytz>=2020.1 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from pandas>=2.2.3->smolagents) (2024.1) Requirement already satisfied: tzdata>=2022.7 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from pandas>=2.2.3->smolagents) (2024.1) Requirement already satisfied: charset-normalizer<4,>=2 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from requests>=2.32.3->smolagents) (3.3.2) Requirement already satisfied: idna<4,>=2.5 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from requests>=2.32.3->smolagents) (3.6) Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from requests>=2.32.3->smolagents) (2.0.7) Requirement already satisfied: certifi>=2017.4.17 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from requests>=2.32.3->smolagents) (2023.11.17) Requirement already satisfied: markdown-it-py>=2.2.0 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from rich>=13.9.4->smolagents) (3.0.0) Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from rich>=13.9.4->smolagents) (2.18.0) Requirement already satisfied: filelock in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from transformers>=4.0.0->smolagents) (3.13.1) Requirement already satisfied: regex!=2019.12.17 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from transformers>=4.0.0->smolagents) (2024.5.10) Requirement already satisfied: safetensors>=0.4.1 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from transformers>=4.0.0->smolagents) (0.4.3) Requirement already satisfied: tqdm>=4.27 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from transformers>=4.0.0->smolagents) (4.66.1) Requirement already satisfied: sympy in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from torch->smolagents) (1.12) Requirement already satisfied: networkx in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from torch->smolagents) (3.3) Requirement already satisfied: sniffio>=1.1 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from anyio<5.0,>=3.0->gradio>=5.8.0->smolagents) (1.3.0) Requirement already satisfied: soupsieve>1.2 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from beautifulsoup4<5,>=4.9->markdownify>=0.14.1->smolagents) (2.5) Requirement already satisfied: httpcore<2.0.0,>=1.0.5 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from e2b<2.0.0,>=1.0.4->e2b-code-interpreter>=1.0.3->smolagents) (1.0.7) Requirement already satisfied: protobuf<6.0.0,>=3.20.0 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from e2b<2.0.0,>=1.0.4->e2b-code-interpreter>=1.0.3->smolagents) (5.29.0) Requirement already satisfied: h11<0.15,>=0.13 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from httpcore<2.0.0,>=1.0.5->e2b<2.0.0,>=1.0.4->e2b-code-interpreter>=1.0.3->smolagents) (0.14.0) Requirement already satisfied: zipp>=3.20 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from importlib-metadata>=6.8.0->litellm>=1.55.10->smolagents) (3.21.0) Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from jsonschema<5.0.0,>=4.22.0->litellm>=1.55.10->smolagents) (2023.12.1) Requirement already satisfied: referencing>=0.28.4 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from jsonschema<5.0.0,>=4.22.0->litellm>=1.55.10->smolagents) (0.35.1) Requirement already satisfied: rpds-py>=0.7.1 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from jsonschema<5.0.0,>=4.22.0->litellm>=1.55.10->smolagents) (0.18.1) Requirement already satisfied: mdurl~=0.1 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from markdown-it-py>=2.2.0->rich>=13.9.4->smolagents) (0.1.2) Requirement already satisfied: distro<2,>=1.7.0 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from openai>=1.55.3->litellm>=1.55.10->smolagents) (1.8.0) Requirement already satisfied: jiter<1,>=0.4.0 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from openai>=1.55.3->litellm>=1.55.10->smolagents) (0.7.1) Requirement already satisfied: annotated-types>=0.6.0 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from pydantic>=2.0->gradio>=5.8.0->smolagents) (0.6.0) Requirement already satisfied: pydantic-core==2.23.4 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from pydantic>=2.0->gradio>=5.8.0->smolagents) (2.23.4) Requirement already satisfied: shellingham>=1.3.0 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from typer<1.0,>=0.12->gradio>=5.8.0->smolagents) (1.5.4) Requirement already satisfied: aiosignal>=1.1.2 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from aiohttp->litellm>=1.55.10->smolagents) (1.3.1) Requirement already satisfied: frozenlist>=1.1.1 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from aiohttp->litellm>=1.55.10->smolagents) (1.4.1) Requirement already satisfied: multidict<7.0,>=4.5 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from aiohttp->litellm>=1.55.10->smolagents) (6.0.5) Requirement already satisfied: yarl<2.0,>=1.0 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from aiohttp->litellm>=1.55.10->smolagents) (1.9.4) Requirement already satisfied: mpmath>=0.19 in /Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages (from sympy->torch->smolagents) (1.3.0)
In [2]python · cell 3
python
!pip install datasets huggingface_hub langchain sentence-transformers faiss-cpu serpapi google-search-results openai -q1. 🏞️ 多模态 + 🌐 网络浏览助手
对于这个用例，我们想要展示一个能够浏览网络并能够生成图像的智能体。
为了构建它，我们只需要准备两个工具：图像生成和网络搜索。
- 对于图像生成，我们从 Hub 加载一个工具，该工具使用 HF 推理 API（无服务器）使用 Stable Diffusion 生成图像。
- 对于网络搜索，我们加载一个 LangChain 工具。
In [3]python · cell 5
python
from transformers import Tool, load_tool, CodeAgent, InferenceClientModel
# Import tool from Hub
image_generation_tool = load_tool("m-ric/text-to-image", trust_remote_code=True)
# Import tool from LangChain
from langchain.agents import load_tools
search_tool = Tool.from_langchain(load_tools(["serpapi"])[0])
model = InferenceClientModel("meta-llama/Llama-3.1-70B-Instruct")
# Initialize the agent with both tools
agent = CodeAgent(
tools=[image_generation_tool, search_tool], model=model
)
# Run it!
result = agent.run(
"Generate me a photo of the car that James bond drove in the latest movie.",
)
resultOutput
/Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm
[0;31m---------------------------------------------------------------------------[0m [0;31mImportError[0m Traceback (most recent call last) Cell [0;32mIn[3], line 1[0m [0;32m----> 1[0m [38;5;28;01mfrom[39;00m [38;5;21;01mtransformers[39;00m [38;5;28;01mimport[39;00m Tool, load_tool, CodeAgent, InferenceClientModel [1;32m 3[0m [38;5;66;03m# Import tool from Hub[39;00m [1;32m 4[0m image_generation_tool [38;5;241m=[39m load_tool([38;5;124m"[39m[38;5;124mm-ric/text-to-image[39m[38;5;124m"[39m, trust_remote_code[38;5;241m=[39m[38;5;28;01mTrue[39;00m) [0;31mImportError[0m: cannot import name 'InferenceClientModel' from 'transformers' (/Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages/transformers/__init__.py)
2. 📚💬 带有迭代查询优化和来源选择的 RAG
快速定义：检索增强生成（RAG）是 “使用大型语言模型（LLM）来回答用户查询，但基于从知识库检索到的信息来构建答案”。
这种方法相比使用普通或微调的 LLM 有许多优势：列举一些，它允许将答案建立在真实事实的基础上并减少虚构，它允许为 LLM 提供特定领域的知识，并且它允许对知识库中的信息访问进行细粒度控制。
-
现在假设我们想要执行 RAG，但增加了动态生成某些参数的约束。例如，根据用户查询，我们可能想要将搜索限制在知识库的特定子集，或者我们可能想要调整检索到的文档数量。难点在于：如何根据用户查询动态调整这些参数？
-
RAG 的一个常见失败案例是基于用户查询的检索没有返回任何相关的支持文档。有没有一种方法，在之前的结果不相关时，通过修改查询重新调用检索器来进行迭代？
🔧 好吧，我们可以以简单的方式解决上述问题：我们将让我们的智能体控制检索器的参数！
➡️ 让我们展示如何做到这一点。我们首先加载一个我们想要执行 RAG 的知识库：这个数据集是许多 huggingface 包的文档页面汇总，以 markdown 格式存储。
In [1]python · cell 8
python
import datasets
knowledge_base = datasets.load_dataset("m-ric/huggingface_doc", split="train")Output
/Users/aymeric/.pyenv/versions/3.12.0/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm
现在我们通过处理数据集并将其存储到向量数据库中来准备知识库，以便检索器使用。我们将使用 LangChain，因为它具有用于向量数据库的优秀工具：
In [ ]python · cell 10
python
from langchain.docstore.document import Document
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.vectorstores import FAISS
from langchain_community.embeddings import HuggingFaceEmbeddings
source_docs = [
Document(page_content=doc["text"], metadata={"source": doc["source"].split("/")[1]})
for doc in knowledge_base
]
docs_processed = RecursiveCharacterTextSplitter(chunk_size=500).split_documents(
source_docs
)[:1000]
embedding_model = HuggingFaceEmbeddings(model_name="thenlper/gte-small")
vectordb = FAISS.from_documents(documents=docs_processed, embedding=embedding_model)现在我们已经准备好了数据库，让我们构建一个基于它回答用户查询的 RAG 系统！
我们希望我们的系统根据查询只从最相关的信息来源中选择。
我们的文档页面来自以下来源：
In [3]python · cell 12
python
all_sources = list(set([doc.metadata["source"] for doc in docs_processed]))
print(all_sources)Output
['evaluate', 'course', 'deep-rl-class', 'peft', 'hf-endpoints-documentation', 'blog', 'gradio', 'datasets', 'datasets-server', 'transformers', 'optimum', 'hub-docs', 'pytorch-image-models', 'diffusers']
In [1]python · cell 13
python
import json
from smolagents import Tool
from langchain_core.vectorstores import VectorStore
class RetrieverTool(Tool):
name = "retriever"
description = "Retrieves some documents from the knowledge base that have the closest embeddings to the input query."
inputs = {
"query": {
"type": "text",
"description": "The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.",
},
"source": {"type": "text", "description": ""},
"number_of_documents": {
"type": "text",
"description": "the number of documents to retrieve. Stay under 10 to avoid drowning in docs",
},
}
output_type = "text"
def __init__(self, vectordb: VectorStore, all_sources: str, **kwargs):
super().__init__(**kwargs)
self.vectordb = vectordb
self.inputs["source"][
"description"
] = f"The source of the documents to search, as a str representation of a list. Possible values in the list are: {all_sources}. If this argument is not provided, all sources will be searched."
def forward(self, query: str, source: str = None, number_of_documents=7) -> str:
assert isinstance(query, str), "Your search query must be a string"
number_of_documents = int(number_of_documents)
if source:
if isinstance(source, str) and "[" not in str(
source
): # if the source is not representing a list
source = [source]
source = json.loads(str(source).replace("'", '"'))
docs = self.vectordb.similarity_search(
query,
filter=({"source": source} if source else None),
k=number_of_documents,
)
if len(docs) == 0:
return "No documents found with this filtering. Try removing the source filter."
return "Retrieved documents:\n\n" + "\n===Document===\n".join(
[doc.page_content for doc in docs]
)可选：将你的检索器工具分享到 Hub
要将你的工具分享到 Hub，首先将检索器工具定义单元格中的代码复制粘贴到一个名为例如 retriever.py 的新文件中。
当工具从单独的文件加载后，你可以使用以下代码将其推送到 Hub（确保使用具有写入访问权限的 token 登录）
In [ ]python · cell 15
python
share_to_hub = False
if share_to_hub:
from huggingface_hub import login
from retriever import RetrieverTool
login("your_token")
tool = RetrieverTool(vectordb, all_sources)
tool.push_to_hub(repo_id="m-ric/retriever-tool")运行智能体!
In [38]python · cell 17
python
from smolagents import HfModel, ToolCallingAgent, load_tool
model = HfModel("meta-llama/Meta-Llama-3-70B-Instruct")
retriever_tool = load_tool(
"m-ric/retriever-tool", vectordb=vectordb, all_sources=all_sources
)
agent = ToolCallingAgent(tools=[retriever_tool], model=model, verbose=0)
agent_output = agent.run("Please show me a LORA finetuning script")
print("Final output:")
print(agent_output)Output
A new version of the following files was downloaded from https://huggingface.co/spaces/m-ric/retriever-tool:
- retriever.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
[33;1m======== New task ========[0m
[37;1mPlease show me a LORA finetuning script[0m
[33;1mCalling tool: 'retriever' with arguments: {'number_of_documents': '5', 'query': 'LORA finetuning script', 'source': "['transformers', 'blog']"}[0m
[33;1mCalling tool: 'retriever' with arguments: {'number_of_documents': '5', 'query': 'LORA finetuning script'}[0m
[33;1mCalling tool: 'retriever' with arguments: {'number_of_documents': '5', 'query': 'train_text_to_image_lora.py'}[0m
[33;1mCalling tool: 'final_answer' with arguments: https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_lora.py[0m
Final output: https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_lora.py
发生了什么？首先，智能体启动了检索器，并考虑了特定的来源（['transformers', 'blog']）。
但是这次检索没有产生足够的结果 ⇒ 没关系！智能体可以迭代之前的结果，因此它只是用不那么严格的搜索参数重新运行了它的检索。
因此，研究成功了！
请注意，使用调用检索器作为工具并可以动态修改查询和其他检索参数的 LLM 智能体是 RAG 的更一般的表述，这也涵盖了像迭代查询优化这样的许多 RAG 改进技术。
3. 💻 调试 Python 代码
In [41]python · cell 19
python
from smolagents import CodeAgent
agent = CodeAgent(tools=[])
code = """
list=[0, 1, 2]
for i in range(4):
print(list(i))
"""
final_answer = agent.run(
"I have some code that creates a bug: please debug it and return the final code",
code=code,
)Output
[33;1m======== New task ========[0m
[37;1mI have some code that creates a bug: please debug it and return the final code
You have been provided with these initial arguments: {'code': '\nlist=[0, 1, 2]\n\nfor i in range(4):\n print(list(i))\n'}.[0m
[33;1m==== Agent is executing the code below:[0m
[0m[38;5;109mlist[39m[38;5;109;01m=[39;00m[38;5;7m[[39m[38;5;139m0[39m[38;5;7m,[39m[38;5;7m [39m[38;5;139m1[39m[38;5;7m,[39m[38;5;7m [39m[38;5;139m2[39m[38;5;7m][39m
[38;5;109mprint[39m[38;5;7m([39m[38;5;109mlist[39m[38;5;7m)[39m
[38;5;109;01mfor[39;00m[38;5;7m [39m[38;5;7mi[39m[38;5;7m [39m[38;5;109;01min[39;00m[38;5;7m [39m[38;5;109mrange[39m[38;5;7m([39m[38;5;139m4[39m[38;5;7m)[39m[38;5;7m:[39m
[38;5;7m [39m[38;5;109mprint[39m[38;5;7m([39m[38;5;109mlist[39m[38;5;7m[[39m[38;5;7mi[39m[38;5;7m][39m[38;5;7m)[39m[0m
[33;1m====[0m
[31;20mFailed while trying to execute the code below:
[0mlist=[0, 1, 2]
print(list)
for i in range(4):
print(list[i])[0m
This failed due to the following error:
list index out of range[0m
Traceback (most recent call last):
File "/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py", line 823, in step
result = self.python_evaluator(code_action, available_tools, state=self.state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/python_interpreter.py", line 511, in evaluate_python_code
line_result = evaluate_ast(node, state, tools)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/python_interpreter.py", line 404, in evaluate_ast
return evaluate_for(expression, state, tools)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/python_interpreter.py", line 313, in evaluate_for
line_result = evaluate_ast(node, state, tools)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/python_interpreter.py", line 401, in evaluate_ast
return evaluate_ast(expression.value, state, tools)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/python_interpreter.py", line 365, in evaluate_ast
return evaluate_call(expression, state, tools)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/python_interpreter.py", line 215, in evaluate_call
args = [evaluate_ast(arg, state, tools) for arg in call.args]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/python_interpreter.py", line 423, in evaluate_ast
return evaluate_subscript(expression, state, tools)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/python_interpreter.py", line 236, in evaluate_subscript
return value[int(index)]
~~~~~^^^^^^^^^^^^
IndexError: list index out of range
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py", line 623, in run
final_answer = self.step()
^^^^^^^^^^^
File "/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py", line 832, in step
raise AgentExecutionError(error_msg)
transformers.agents.agents.AgentExecutionError: Failed while trying to execute the code below:
[0mlist=[0, 1, 2]
print(list)
for i in range(4):
print(list[i])[0m
This failed due to the following error:
list index out of range
[33;1m==== Agent is executing the code below:[0m
[0m[38;5;109mlist[39m[38;5;109;01m=[39;00m[38;5;7m[[39m[38;5;139m0[39m[38;5;7m,[39m[38;5;7m [39m[38;5;139m1[39m[38;5;7m,[39m[38;5;7m [39m[38;5;139m2[39m[38;5;7m][39m
[38;5;109mprint[39m[38;5;7m([39m[38;5;109mlist[39m[38;5;7m)[39m
[38;5;109;01mfor[39;00m[38;5;7m [39m[38;5;7mi[39m[38;5;7m [39m[38;5;109;01min[39;00m[38;5;7m [39m[38;5;109mrange[39m[38;5;7m([39m[38;5;139m3[39m[38;5;7m)[39m[38;5;7m:[39m
[38;5;7m [39m[38;5;109mprint[39m[38;5;7m([39m[38;5;109mlist[39m[38;5;7m[[39m[38;5;7mi[39m[38;5;7m][39m[38;5;7m)[39m[0m
[33;1m====[0m
[33;1mPrint outputs:[0m
[32;20m[0, 1, 2]
0
1
2
[0m
[33;1m==== Agent is executing the code below:[0m
[0m[38;5;109mlist[39m[38;5;109;01m=[39;00m[38;5;7m[[39m[38;5;139m0[39m[38;5;7m,[39m[38;5;7m [39m[38;5;139m1[39m[38;5;7m,[39m[38;5;7m [39m[38;5;139m2[39m[38;5;7m][39m
[38;5;109mprint[39m[38;5;7m([39m[38;5;109mlist[39m[38;5;7m)[39m
[38;5;109;01mfor[39;00m[38;5;7m [39m[38;5;7mi[39m[38;5;7m [39m[38;5;109;01min[39;00m[38;5;7m [39m[38;5;109mrange[39m[38;5;7m([39m[38;5;139m3[39m[38;5;7m)[39m[38;5;7m:[39m
[38;5;7m [39m[38;5;109mprint[39m[38;5;7m([39m[38;5;109mlist[39m[38;5;7m[[39m[38;5;7mi[39m[38;5;7m][39m[38;5;7m)[39m[0m
[33;1m====[0m
[33;1mPrint outputs:[0m
[32;20m[0, 1, 2]
0
1
2
[0m
[33;1m==== Agent is executing the code below:[0m
[0m[38;5;109mlist[39m[38;5;109;01m=[39;00m[38;5;7m[[39m[38;5;139m0[39m[38;5;7m,[39m[38;5;7m [39m[38;5;139m1[39m[38;5;7m,[39m[38;5;7m [39m[38;5;139m2[39m[38;5;7m][39m
[38;5;109mprint[39m[38;5;7m([39m[38;5;109mlist[39m[38;5;7m)[39m
[38;5;109;01mfor[39;00m[38;5;7m [39m[38;5;7mi[39m[38;5;7m [39m[38;5;109;01min[39;00m[38;5;7m [39m[38;5;109mrange[39m[38;5;7m([39m[38;5;109mlen[39m[38;5;7m([39m[38;5;109mlist[39m[38;5;7m)[39m[38;5;7m)[39m[38;5;7m:[39m
[38;5;7m [39m[38;5;109mprint[39m[38;5;7m([39m[38;5;109mlist[39m[38;5;7m[[39m[38;5;7mi[39m[38;5;7m][39m[38;5;7m)[39m[0m
[33;1m====[0m
[33;1mPrint outputs:[0m
[32;20m[0, 1, 2]
0
1
2
[0m
[33;1m==== Agent is executing the code below:[0m
[0m[38;5;109mlist[39m[38;5;109;01m=[39;00m[38;5;7m[[39m[38;5;139m0[39m[38;5;7m,[39m[38;5;7m [39m[38;5;139m1[39m[38;5;7m,[39m[38;5;7m [39m[38;5;139m2[39m[38;5;7m][39m
[38;5;109mprint[39m[38;5;7m([39m[38;5;109mlist[39m[38;5;7m)[39m
[38;5;109;01mfor[39;00m[38;5;7m [39m[38;5;7mi[39m[38;5;7m [39m[38;5;109;01min[39;00m[38;5;7m [39m[38;5;109mrange[39m[38;5;7m([39m[38;5;139m3[39m[38;5;7m)[39m[38;5;7m:[39m
[38;5;7m [39m[38;5;109mprint[39m[38;5;7m([39m[38;5;109mlist[39m[38;5;7m[[39m[38;5;7mi[39m[38;5;7m][39m[38;5;7m)[39m
[38;5;7mfinal_answer[39m[38;5;7m([39m[38;5;7mcode[39m[38;5;7m)[39m[0m
[33;1m====[0m
[33;1mPrint outputs:[0m
[32;20m[0, 1, 2]
0
1
2
[0m
[33;1m>>> Final answer:[0m
[32;20m
list=[0, 1, 2]
for i in range(4):
print(list(i))
[0m
正如你所看到的，智能体尝试了给定的代码，遇到错误，分析错误，纠正代码，并在验证代码可以正常工作后返回它！
最终的代码是纠正后的代码：
In [42]python · cell 21
python
print(final_answer)Output
list=[0, 1, 2]
for i in range(4):
print(list(i))
4. 创建你自己的 LLM 引擎（OpenAI）
设置你自己的 LLM 引擎真的非常简单：
它只需要一个具有以下标准的__call__方法：
- 接受ChatML 格式 的消息列表作为输入并输出答案。
- 接受一个
stop_sequences参数，以传递生成停止的序列。 - 根据你的 LLM 接受哪种类型的消息角色，你可能还需要转换一些消息角色。
In [6]python · cell 23
python
import os
from openai import OpenAI
from smolagents.model import MessageRole, get_clean_message_list
openai_role_conversions = {
MessageRole.TOOL_RESPONSE: "user",
}
class OpenAIModel:
def __init__(self, model_name="gpt-4o-2024-05-13"):
self.model_name = model_name
self.client = OpenAI(
api_key=os.getenv("OPENAI_API_KEY"),
)
def __call__(self, messages, stop_sequences=[]):
# Get clean message list
messages = get_clean_message_list(
messages, role_conversions=openai_role_conversions
)
# Get LLM output
response = self.client.chat.completions.create(
model=self.model_name,
messages=messages,
stop=stop_sequences,
)
return response.choices[0].message.content
openai_engine = OpenAIModel()
agent = CodeAgent(model=openai_engine, tools=[])
code = """
list=[0, 1, 2]
for i in range(4):
print(list(i))
"""
final_answer = agent.run(
"I have some code that creates a bug: please debug it and return the final code",
code=code,
)Output
[33;1m======== New task ========[0m
[37;1mI have some code that creates a bug: please debug it and return the final code
You have been provided with these initial arguments: {'code': '\nlist=[0, 1, 2]\n\nfor i in range(4):\n print(list(i))\n'}.[0m
[33;1m==== Agent is executing the code below:[0m
[0m[38;5;7mmy_list[39m[38;5;7m [39m[38;5;109;01m=[39;00m[38;5;7m [39m[38;5;7m[[39m[38;5;139m0[39m[38;5;7m,[39m[38;5;7m [39m[38;5;139m1[39m[38;5;7m,[39m[38;5;7m [39m[38;5;139m2[39m[38;5;7m][39m[38;5;7m [39m[38;5;60;03m# Renamed the list to avoid using the built-in name[39;00m
[38;5;109;01mfor[39;00m[38;5;7m [39m[38;5;7mi[39m[38;5;7m [39m[38;5;109;01min[39;00m[38;5;7m [39m[38;5;109mrange[39m[38;5;7m([39m[38;5;109mlen[39m[38;5;7m([39m[38;5;7mmy_list[39m[38;5;7m)[39m[38;5;7m)[39m[38;5;7m:[39m[38;5;7m [39m[38;5;60;03m# Changed the range to be within the length of the list[39;00m
[38;5;7m [39m[38;5;109mprint[39m[38;5;7m([39m[38;5;7mmy_list[39m[38;5;7m[[39m[38;5;7mi[39m[38;5;7m][39m[38;5;7m)[39m[38;5;7m [39m[38;5;60;03m# Corrected the list access syntax[39;00m[0m
[33;1m====[0m
[33;1mPrint outputs:[0m
[32;20m0
1
2
[0m
[33;1m==== Agent is executing the code below:[0m
[0m[38;5;7mmy_list[39m[38;5;7m [39m[38;5;109;01m=[39;00m[38;5;7m [39m[38;5;7m[[39m[38;5;139m0[39m[38;5;7m,[39m[38;5;7m [39m[38;5;139m1[39m[38;5;7m,[39m[38;5;7m [39m[38;5;139m2[39m[38;5;7m][39m[38;5;7m [39m[38;5;60;03m# Renamed the list to avoid using the built-in name[39;00m
[38;5;109;01mfor[39;00m[38;5;7m [39m[38;5;7mi[39m[38;5;7m [39m[38;5;109;01min[39;00m[38;5;7m [39m[38;5;109mrange[39m[38;5;7m([39m[38;5;109mlen[39m[38;5;7m([39m[38;5;7mmy_list[39m[38;5;7m)[39m[38;5;7m)[39m[38;5;7m:[39m[38;5;7m [39m[38;5;60;03m# Changed the range to be within the length of the list[39;00m
[38;5;7m [39m[38;5;109mprint[39m[38;5;7m([39m[38;5;7mmy_list[39m[38;5;7m[[39m[38;5;7mi[39m[38;5;7m][39m[38;5;7m)[39m[38;5;7m [39m[38;5;60;03m# Corrected the list access syntax[39;00m[0m
[33;1m====[0m
[33;1mPrint outputs:[0m
[32;20m0
1
2
[0m
[33;1m==== Agent is executing the code below:[0m
[0m[38;5;7mcorrected_code[39m[38;5;7m [39m[38;5;109;01m=[39;00m[38;5;7m [39m[38;5;144m'''[39m
[38;5;144mmy_list = [0, 1, 2] # Renamed the list to avoid using the built-in name[39m
[38;5;144mfor i in range(len(my_list)): # Changed the range to be within the length of the list[39m
[38;5;144m print(my_list[i]) # Corrected the list access syntax[39m
[38;5;144m'''[39m
[38;5;7mfinal_answer[39m[38;5;7m([39m[38;5;7manswer[39m[38;5;109;01m=[39;00m[38;5;7mcorrected_code[39m[38;5;7m)[39m[0m
[33;1m====[0m
[33;1mPrint outputs:[0m
[32;20m[0m
[33;1m>>> Final answer:[0m
[32;20m
my_list = [0, 1, 2] # Renamed the list to avoid using the built-in name
for i in range(len(my_list)): # Changed the range to be within the length of the list
print(my_list[i]) # Corrected the list access syntax
[0m
In [7]python · cell 24
python
print(final_answer)Output
my_list = [0, 1, 2] # Renamed the list to avoid using the built-in name
for i in range(len(my_list)): # Changed the range to be within the length of the list
print(my_list[i]) # Corrected the list access syntax