Trace Your Haystack Application

Published Aug 19, 2024

Haystack is an open-source framework for building LLM applications, retrieval-augmented generative pipelines and search systems that work intelligently over large document collections.

Haystack makes it very easy to get off and running quickly with search and RAG LLM apps. Combining Haystack with Phoenix gives you the tools to not only build these apps quickly, but improve and augment them to be truly production-ready.

Combining Phoenix and Haystack gives you:

Effortless Tracing: With just a single line of code, Phoenix enables tracing, giving you deep insights into your application’s behavior.
Enhanced Debugging: Phoenix’s detailed tracing allows you to pinpoint issues quickly, ensuring smoother and faster debugging processes.
Comprehensive Evaluations: Run evaluations to continuously improve your AI application’s performance, ensuring it meets the highest standards.

Step-by-Step Guide: Setting Up a Basic RAG Application with Haystack and Phoenix

Install the Initial Libraries

First, install the Phoenix and Haystack libraries:

pip install openinference-instrumentation-haystack haystack-ai arize-phoenix opentelemetry-sdk opentelemetry-exporter-otlp

Next, we need to set our necessary API keys:

import os
import getpass
os.environ["OPENAI_API_KEY"] = getpass.getpass("OpenAI API Key:")

Launch Phoenix & Connect it to your Application

Launch a local Phoenix instance on your machine by running the following command:

python -m phoenix.server.main serve

Let’s connect our local environment to our Phoenix instance to receive traces:

from opentelemetry import trace as trace_api
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk import trace as trace_sdk
from opentelemetry.sdk.trace.export import SimpleSpanProcessor


tracer_provider = trace_sdk.TracerProvider()
span_exporter = OTLPSpanExporter("http://localhost:6006/v1/traces")
span_processor = SimpleSpanProcessor(span_exporter)
tracer_provider.add_span_processor(span_processor)
trace_api.set_tracer_provider(tracer_provider)

Here we are defining a new OpenTelemetry provider, and pointing it towards our Phoenix instance as a processor.

If you prefer to connect to a cloud instance of Phoenix hosted for you, follow these steps.

Now we can add our Haystack auto-instrumentor to generate all the telemetry we’ll need from our application:

from openinference.instrumentation.haystack import HaystackInstrumentor
HaystackInstrumentor().instrument(skip_dep_check=true, tracer_provider=tracer_provider)

Now, any Haystack code we run will appear as traces and spans in our Phoenix instance.

Set Up Your Haystack App

Initialize your Haystack environment by setting up a document store, retriever, and reader. Here’s a basic example:

import os
from haystack import Pipeline, Document
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
from haystack.components.generators import OpenAIGenerator
from haystack.components.builders.answer_builder import AnswerBuilder
from haystack.components.builders.prompt_builder import PromptBuilder


# Write documents to InMemoryDocumentStore
document_store = InMemoryDocumentStore()
document_store.write_documents([
   Document(content="My name is Jean and I live in Paris."),
   Document(content="My name is Mark and I live in Berlin."),
   Document(content="My name is Giorgio and I live in Rome.")
])


# Build a RAG pipeline
prompt_template = """
Given these documents, answer the question.
Documents:
{% for doc in documents %}
   {{ doc.content }}
{% endfor %}
Question: {{question}}
Answer:
"""


retriever = InMemoryBM25Retriever(document_store=document_store)
prompt_builder = PromptBuilder(template=prompt_template)
llm = OpenAIGenerator(model="gpt-3.5-turbo")


rag_pipeline = Pipeline()
rag_pipeline.add_component("retriever", retriever)
rag_pipeline.add_component("prompt_builder", prompt_builder)
rag_pipeline.add_component("llm", llm)
rag_pipeline.connect("retriever", "prompt_builder.documents")
rag_pipeline.connect("prompt_builder", "llm")

Let’s break down this code. First we need to define a document store. We are using the simplest option of a basic in memory document store, however Haystack provides multiple options for more complex options.

Next we’re creating three components in our Haystack pipeline: a Retriever, a Prompt Builder, and an LLM. The Retriever is connected to our in-memory document store, and will be used to grab relevant documents from that store when queries come in. Our Prompt Builder serves as the template for what our eventual LLM request will look like. Finally, our LLM handles the actual response generation.

Call Our Haystack Pipeline

Now we are ready to call our Haystack pipeline:

# Ask a question
question = "Who lives in Paris?"
results = rag_pipeline.run(
   {
       "retriever": {"query": question},
       "prompt_builder": {"question": question},
   }
)


print(results["llm"]
["replies"])

We need to pass our question to both stages of our pipeline. First to our Retriever to grab the relevant documents from our docs store, and second, to our prompt builder to generate our prompt. Haystack handles passing the retrieved documents into our Prompt Builder.