Chapter 07
Chapter 7 Advanced Text Generation Techniques and Tools
NotebookPython 3 (ipykernel)43 cells
This notebook is for Chapter 7 of the Hands-On Large Language Models book by Jay Alammar and Maarten Grootendorst.
[OPTIONAL] - Installing Packages on
If you are viewing this notebook on Google Colab (or any other cloud vendor), you need to uncomment and run the following codeblock to install the dependencies for this chapter:
💡 NOTE: We will want to use a GPU to run the examples in this notebook. In Google Colab, go to Runtime > Change runtime type > Hardware accelerator > GPU > GPU type > T4.
In [ ]python · cell 3
python
# %%capture
# !pip install langchain>=0.1.17 openai>=1.13.3 langchain_openai>=0.1.6 transformers>=4.40.1 datasets>=2.18.0 accelerate>=0.27.2 sentence-transformers>=2.5.1 duckduckgo-search>=5.2.2 langchain_community
# !CMAKE_ARGS="-DLLAMA_CUDA=on" pip install llama-cpp-python==0.2.69Loading an LLM
In [ ]python · cell 5
python
!wget https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf/resolve/main/Phi-3-mini-4k-instruct-fp16.gguf
# If this command does not work for you, you can use the link directly to download the model
# https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf/resolve/main/Phi-3-mini-4k-instruct-fp16.ggufIn [ ]python · cell 6
python
from langchain import LlamaCpp
# Make sure the model path is correct for your system!
llm = LlamaCpp(
model_path="Phi-3-mini-4k-instruct-fp16.gguf",
n_gpu_layers=-1,
max_tokens=500,
n_ctx=2048,
seed=42,
verbose=False
)In [ ]python · cell 7
python
llm.invoke("Hi! My name is Maarten. What is 1 + 1?")Output
''
Chains
In [ ]python · cell 9
python
from langchain import PromptTemplate
# Create a prompt template with the "input_prompt" variable
template = """<s><|user|>
{input_prompt}<|end|>
<|assistant|>"""
prompt = PromptTemplate(
template=template,
input_variables=["input_prompt"]
)In [ ]python · cell 10
python
basic_chain = prompt | llmIn [ ]python · cell 11
python
# Use the chain
basic_chain.invoke(
{
"input_prompt": "Hi! My name is Maarten. What is 1 + 1?",
}
)Output
' Hello Maarten, the answer to 1 + 1 is 2.'
Multiple Chains
In [ ]python · cell 13
python
from langchain import LLMChain
# Create a chain for the title of our story
template = """<s><|user|>
Create a title for a story about {summary}. Only return the title.<|end|>
<|assistant|>"""
title_prompt = PromptTemplate(template=template, input_variables=["summary"])
title = LLMChain(llm=llm, prompt=title_prompt, output_key="title")Output
/usr/local/lib/python3.10/dist-packages/langchain_core/_api/deprecation.py:141: LangChainDeprecationWarning: The class `LLMChain` was deprecated in LangChain 0.1.17 and will be removed in 1.0. Use RunnableSequence, e.g., `prompt | llm` instead. warn_deprecated(
In [ ]python · cell 14
python
title.invoke({"summary": "a girl that lost her mother"})Output
{'summary': 'a girl that lost her mother',
'title': ' "Whispers of Love: A Journey Through Grief"'}In [ ]python · cell 15
python
# Create a chain for the character description using the summary and title
template = """<s><|user|>
Describe the main character of a story about {summary} with the title {title}. Use only two sentences.<|end|>
<|assistant|>"""
character_prompt = PromptTemplate(
template=template, input_variables=["summary", "title"]
)
character = LLMChain(llm=llm, prompt=character_prompt, output_key="character")In [ ]python · cell 16
python
# Create a chain for the story using the summary, title, and character description
template = """<s><|user|>
Create a story about {summary} with the title {title}. The main charachter is: {character}. Only return the story and it cannot be longer than one paragraph<|end|>
<|assistant|>"""
story_prompt = PromptTemplate(
template=template, input_variables=["summary", "title", "character"]
)
story = LLMChain(llm=llm, prompt=story_prompt, output_key="story")In [ ]python · cell 17
python
# Combine all three components to create the full chain
llm_chain = title | character | storyIn [ ]python · cell 18
python
llm_chain.invoke("a girl that lost her mother")Output
{'summary': 'a girl that lost her mother',
'title': ' "In Loving Memory: A Journey Through Grief"',
'character': ' The protagonist, Emily, is a resilient young girl who struggles to cope with her overwhelming grief after losing her beloved and caring mother at an early age. As she embarks on a journey of self-discovery and healing, she learns valuable life lessons from the memories and wisdom shared by those around her.',
'story': " In Loving Memory: A Journey Through Grief revolves around Emily, a resilient young girl who loses her beloved mother at an early age. Struggling to cope with overwhelming grief, she embarks on a journey of self-discovery and healing, drawing strength from the cherished memories and wisdom shared by those around her. Through this transformative process, Emily learns valuable life lessons about resilience, love, and the power of human connection, ultimately finding solace in honoring her mother's legacy while embracing a newfound sense of inner peace amidst the painful loss."}Memory
In [ ]python · cell 20
python
# Let's give the LLM our name
basic_chain.invoke({"input_prompt": "Hi! My name is Maarten. What is 1 + 1?"})Output
' Hello Maarten! The answer to 1 + 1 is 2.'
In [ ]python · cell 21
python
# Next, we ask the LLM to reproduce the name
basic_chain.invoke({"input_prompt": "What is my name?"})Output
" I'm unable to determine your name as I don't have the capability to access personal data. If you provide context or information where this question might be relevant, I could assist you further within appropriate guidelines!"
ConversationBuffer
In [ ]python · cell 23
python
# Create an updated prompt template to include a chat history
template = """<s><|user|>Current conversation:{chat_history}
{input_prompt}<|end|>
<|assistant|>"""
prompt = PromptTemplate(
template=template,
input_variables=["input_prompt", "chat_history"]
)In [ ]python · cell 24
python
from langchain.memory import ConversationBufferMemory
# Define the type of Memory we will use
memory = ConversationBufferMemory(memory_key="chat_history")
# Chain the LLM, Prompt, and Memory together
llm_chain = LLMChain(
prompt=prompt,
llm=llm,
memory=memory
)In [ ]python · cell 25
python
# Generate a conversation and ask a basic question
llm_chain.invoke({"input_prompt": "Hi! My name is Maarten. What is 1 + 1?"})Output
{'input_prompt': 'Hi! My name is Maarten. What is 1 + 1?',
'chat_history': '',
'text': " Hello Maarten! The answer to 1 + 1 is 2. Hope you're having a great day!"}In [ ]python · cell 26
python
# Does the LLM remember the name we gave it?
llm_chain.invoke({"input_prompt": "What is my name?"})Output
{'input_prompt': 'What is my name?',
'chat_history': "Human: Hi! My name is Maarten. What is 1 + 1?\nAI: Hello Maarten! The answer to 1 + 1 is 2. Hope you're having a great day!",
'text': ' Your name is Maarten.'}ConversationBufferMemoryWindow
In [ ]python · cell 28
python
from langchain.memory import ConversationBufferWindowMemory
# Retain only the last 2 conversations in memory
memory = ConversationBufferWindowMemory(k=2, memory_key="chat_history")
# Chain the LLM, Prompt, and Memory together
llm_chain = LLMChain(
prompt=prompt,
llm=llm,
memory=memory
)In [ ]python · cell 29
python
# Ask two questions and generate two conversations in its memory
llm_chain.invoke({"input_prompt":"Hi! My name is Maarten and I am 33 years old. What is 1 + 1?"})
llm_chain.invoke({"input_prompt":"What is 3 + 3?"})Output
{'input_prompt': 'What is 3 + 3?',
'chat_history': "Human: Hi! My name is Maarten and I am 33 years old. What is 1 + 1?\nAI: Hello Maarten, it's nice to meet you! The answer to 1 + 1 is 2.\n\nHowever, if you have any other questions or need further assistance, feel free to ask!",
'text': " Hello again! 3 + 3 equals 6. If there's anything else I can help you with, just let me know!"}In [ ]python · cell 30
python
# Check whether it knows the name we gave it
llm_chain.invoke({"input_prompt":"What is my name?"})Output
{'input_prompt': 'What is my name?',
'chat_history': "Human: Hi! My name is Maarten and I am 33 years old. What is 1 + 1?\nAI: Hello Maarten, it's nice to meet you! The answer to 1 + 1 is 2.\n\nHowever, if you have any other questions or need further assistance, feel free to ask!\nHuman: What is 3 + 3?\nAI: Hello again! 3 + 3 equals 6. If there's anything else I can help you with, just let me know!",
'text': ' Your name is Maarten.'}In [ ]python · cell 31
python
# Check whether it knows the age we gave it
llm_chain.invoke({"input_prompt":"What is my age?"})Output
{'input_prompt': 'What is my age?',
'chat_history': "Human: What is 3 + 3?\nAI: Hello again! 3 + 3 equals 6. If there's anything else I can help you with, just let me know!\nHuman: What is my name?\nAI: Your name is Maarten.",
'text': " I'm unable to determine your age as I don't have access to personal information. Age isn't something that can be inferred from our current conversation unless you choose to share it with me. How else may I assist you today?"}ConversationSummary
In [ ]python · cell 33
python
# Create a summary prompt template
summary_prompt_template = """<s><|user|>Summarize the conversations and update with the new lines.
Current summary:
{summary}
new lines of conversation:
{new_lines}
New summary:<|end|>
<|assistant|>"""
summary_prompt = PromptTemplate(
input_variables=["new_lines", "summary"],
template=summary_prompt_template
)In [ ]python · cell 34
python
from langchain.memory import ConversationSummaryMemory
# Define the type of memory we will use
memory = ConversationSummaryMemory(
llm=llm,
memory_key="chat_history",
prompt=summary_prompt
)
# Chain the LLM, prompt, and memory together
llm_chain = LLMChain(
prompt=prompt,
llm=llm,
memory=memory
)In [ ]python · cell 35
python
# Generate a conversation and ask for the name
llm_chain.invoke({"input_prompt": "Hi! My name is Maarten. What is 1 + 1?"})
llm_chain.invoke({"input_prompt": "What is my name?"})Output
{'input_prompt': 'What is my name?',
'chat_history': ' Summary: Human, identified as Maarten, asked the AI about the sum of 1 + 1, which was correctly answered by the AI as 2 and offered additional assistance if needed.',
'text': ' Your name in this context was referred to as "Maarten". However, since our interaction doesn\'t retain personal data beyond a single session for privacy reasons, I don\'t have access to that information. How can I assist you further today?'}In [ ]python · cell 36
python
# Check whether it has summarized everything thus far
llm_chain.invoke({"input_prompt": "What was the first question I asked?"})Output
{'input_prompt': 'What was the first question I asked?',
'chat_history': ' Summary: Human, identified as Maarten in the context of this conversation, first asked about the sum of 1 + 1 and received an answer of 2 from the AI. Later, Maarten inquired about their name but the AI clarified that personal data is not retained beyond a single session for privacy reasons. The AI offered further assistance if needed.',
'text': ' The first question you asked was "what\'s 1 + 1?"'}In [ ]python · cell 37
python
# Check what the summary is thus far
memory.load_memory_variables({})Output
{'chat_history': ' Maarten, identified in this conversation, initially asked about the sum of 1+1 which resulted in an answer from the AI being 2. Subsequently, he sought clarification on his name but the AI informed him that no personal data is retained beyond a single session due to privacy reasons. The AI then offered further assistance if required. Later, Maarten recalled and asked about the first question he inquired which was "what\'s 1+1?"'}Agents
In [ ]python · cell 39
python
import os
from langchain_openai import ChatOpenAI
# Load OpenAI's LLMs with LangChain
os.environ["OPENAI_API_KEY"] = "MY_KEY"
openai_llm = ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0)In [ ]python · cell 40
python
# Create the ReAct template
react_template = """Answer the following questions as best you can. You have access to the following tools:
{tools}
Use the following format:
Question: the input question you must answer
Thought: you should always think about what to do
Action: the action to take, should be one of [{tool_names}]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question
Begin!
Question: {input}
Thought:{agent_scratchpad}"""
prompt = PromptTemplate(
template=react_template,
input_variables=["tools", "tool_names", "input", "agent_scratchpad"]
)In [ ]python · cell 41
python
from langchain.agents import load_tools, Tool
from langchain.tools import DuckDuckGoSearchResults
# You can create the tool to pass to an agent
search = DuckDuckGoSearchResults()
search_tool = Tool(
name="duckduck",
description="A web search engine. Use this to as a search engine for general queries.",
func=search.run,
)
# Prepare tools
tools = load_tools(["llm-math"], llm=openai_llm)
tools.append(search_tool)In [ ]python · cell 42
python
from langchain.agents import AgentExecutor, create_react_agent
# Construct the ReAct agent
agent = create_react_agent(openai_llm, tools, prompt)
agent_executor = AgentExecutor(
agent=agent, tools=tools, verbose=True, handle_parsing_errors=True
)In [ ]python · cell 43
python
# What is the Price of a MacBook Pro?
agent_executor.invoke(
{
"input": "What is the current price of a MacBook Pro in USD? How much would it cost in EUR if the exchange rate is 0.85 EUR for 1 USD?"
}
)Output
[1m> Entering new AgentExecutor chain...[0m [32;1m[1;3mI need to find the current price of a MacBook Pro in USD first before converting it to EUR. Action: duckduck Action Input: "current price of MacBook Pro in USD"[0m[33;1m[1;3m[snippet: View at Best Buy. The best MacBook Pro overall The MacBook Pro 14-inch with the latest M3-series chips offers outstanding, best-in-class performance while getting fantastic battery life and ..., title: The best MacBook Pro in 2024: our picks for the top Pro models, link: https://www.techradar.com/best/best-macbook-pro], [snippet: Starts at $1,299. Upgradable to 24 GB of memory and 2 TB of storage. 67W USB-C charger included. The M2-powered MacBook Pro is available now for a starting price of $1,299 on Apple's website ..., title: MacBook Pro 13-inch (M2, 2022) review | Tom's Guide, link: https://www.tomsguide.com/reviews/macbook-pro-13-inch-m2-2022], [snippet: The late-2023 MacBook Pro update also marks the demise of the 13-inch MacBook Pro, which has been replaced by a 14-inch model with the standard M3 chip, unfortunately at a higher price than the ..., title: Best MacBook Pro Deals: March 2024 | Macworld, link: https://www.macworld.com/article/672811/best-macbook-pro-deals.html], [snippet: For the M3 Pro models, prices start at $2,249.00 for the 512GB/18GB RAM 16-inch MacBook Pro and increase to $2,649.00 for the 512GB/36GB RAM model, both of which are all-time low prices., title: Best Buy Introduces All-Time Low Prices on Apple's M3 MacBook Pro for ..., link: https://www.macrumors.com/2024/04/01/best-buy-m3-macbook-pro/][0m[32;1m[1;3mI found the current price of a MacBook Pro in USD, now I need to convert it to EUR using the exchange rate. Action: Calculator Action Input: $2,249.00 * 0.85[0m[36;1m[1;3mAnswer: 1911.6499999999999[0m[32;1m[1;3mI now know the final answer Final Answer: The current price of a MacBook Pro in USD is $2,249.00. It would cost approximately 1911.65 EUR with an exchange rate of 0.85 EUR for 1 USD.[0m [1m> Finished chain.[0m
{'input': 'What is the current price of a MacBook Pro in USD? How much would it cost in EUR if the exchange rate is 0.85 EUR for 1 USD?',
'output': 'The current price of a MacBook Pro in USD is $2,249.00. It would cost approximately 1911.65 EUR with an exchange rate of 0.85 EUR for 1 USD.'}