How To Set Up CrewAI Observability

Why Observability is Important with CrewAI

In the world of autonomous AI agents, the ability to monitor and evaluate their performance is the key to unlocking their full potential. Observability allows you to see exactly what your agents are doing, how they are performing, and where they can be improved. Just as visibility into KPIs is critical for understanding and optimizing employee performance, observability in CrewAI gives you a clear view into the activities and successes of your AI agents.

With this visibility, you can identify where your agents excel, where they might need adjustments, and whether the larger goals of your projects are being met. This level of insight is essential for continuous improvement and for ensuring that your AI agents are aligned with your strategic objectives.

Introduction To CrewAI

Imagine you could create and design your own personal employees. You can use them to whatever end: to help you with your day-to-day, to automate parts of your life, to even run your business. You can give them a job title, a backstory, and goals. You can assign one of them tasks, or better yet, have a team of diversified workers working together to solve hard tasks, each one specialized and focused on their individual units of work to accomplish a larger goal.

CrewAI makes this a real possibility with their open source agent framework. Built upon LangChain, CrewAI abstracts much of the cognitive work using their framework for orchestrating role-playing, autonomous AI agents.

By fostering collaborative intelligence, CrewAI empowers agents to work together seamlessly, and tackle complex tasks. Imagine tree-of-thought, but you get to design the characters to help you traverse that path autonomously. Their unique take on autonomous agents is intuitive to set up and worth testing and trying out.

Core Concepts of CrewAI

Agents

Think of an agent as a member of a team, with specific skills and a particular job to do. Agents can have different roles like ‘Researcher’, ‘Writer’, or ‘Customer Support’, each contributing to the overall goal of the crew. This is probably the most intuitive concept to understand and notably fun to build.

There are many attributes to choose from but here is where you can customize your individual worker and give them

Each agent can be customized with attributes like:

• Role (e.g., ‘Researcher’, ‘Writer’, ‘Customer Support’)
• Goal
• Backstory
• Tools they can use
• Verbose mode for detailed logging
• Delegation capabilities

from custom_agent import CustomAgent

agent1 = CustomAgent(
    role="agent role",
    goal="who is {input}?",
    backstory="agent backstory",
    verbose=True,
)

Tasks

Crew AI tasks are specific assignments completed by agents. They include:

• Description of the work to be done
• The agent responsible for execution
• Required tools
• Expected output

Tasks can be collaborative, involving multiple agents working together.

from crewai import Task

task = Task(
    description='Find and summarize the latest and most relevant news on AI',
    agent=sales_agent,
    expected_output='A bullet list summary of the top 5 most important AI news',
)

Tools

CrewAI tools empower agents with capabilities ranging from web searching and data analysis to collaboration and delegating tasks among coworkers.

There are many CrewAI tools that empower agents with a wide range of capabilities, including:

• Web searching
• Data analysis
• Collaboration features
• Task delegation among coworkers

Developers can also craft custom tools tailored for their agent’s needs or utilize pre-built options:

from crewai_tools import BaseTool

class MyCustomTool(BaseTool):
    name: str = "Name of my tool"
    description: str = "Clear description for what this tool is useful for, your agent will need this information to use it."

    def _run(self, argument: str) -> str:
        # Implementation goes here
        return "Result from custom tool"

Processes

Processes orchestrate the execution of tasks by agents, akin to project management in human teams. These processes ensure tasks are distributed and executed efficiently, in alignment with a predefined strategy.

• Sequential: Executes tasks sequentially, ensuring tasks are completed in an orderly progression.
• Hierarchical: Organizes tasks in a managerial hierarchy, where tasks are delegated and executed based on a structured chain of command. A manager language model (manager_llm) or a custom manager agent (manager_agent) must be specified in the crew to enable the hierarchical process, facilitating the creation and management of tasks by the manager.
• Consensual Process (Planned): Aiming for collaborative decision-making among agents on task execution, this process type introduces a democratic approach to task management within CrewAI. It is planned for future development and is not currently implemented in the codebase.

from crewai import Crew
from crewai.process import Process
from langchain_openai import ChatOpenAI

# Example: Creating a crew with a sequential process
crew = Crew(
    agents=my_agents,
    tasks=my_tasks,
    process=Process.sequential
)

Crews

A crew in CrewAI represents a collaborative group of agents working together to achieve a set of tasks. Each crew defines the strategy for task execution, agent collaboration, and the overall workflow.

from crewai import Crew

# Assemble the crew with a sequential process
my_crew = Crew(
    agents=[researcher, writer],
    tasks=[research_task, write_article_task],
    process=Process.sequential,
    full_output=True,
    verbose=True,
)

Pipeline

A pipeline in crewAI represents a structured workflow that allows for the sequential or parallel execution of multiple crews. It provides a way to organize complex processes involving multiple stages, where the output of one stage can serve as input for subsequent stages.

from crewai import Pipeline

# Assemble the pipeline
my_pipeline = Pipeline(
    stages=[research_crew, analysis_crew, writing_crew]
)

Planning

The planning feature in CrewAI allows you to add planning capability to your crew. When enabled, before each Crew iteration, all Crew information is sent to an AgentPlanner that will plan the tasks step by step, and this plan will be added to each task description.

from crewai import Crew, Agent, Task, Process

# Assemble your crew with planning capabilities
my_crew = Crew(
    agents=self.agents,
    tasks=self.tasks,
    process=Process.sequential,
    planning=True,
)

Setting Up Observability for CrewAI with Arize-Phoenix: Code Walkthrough

1. Setting Up the Environment

The first snippet installs all the necessary Python packages required for the project, including CrewAI, Phoenix, and the OpenTelemetry SDK. These tools enable the creation, monitoring, and observability of AI agents, ensuring that your setup is ready to handle the tasks ahead.

!pip install -q arize-phoenix opentelemetry-sdk opentelemetry-exporter-otlp crewai crewai_tools openinference-instrumentation-crewai

2. Launching the Phoenix App

In the second snippet, the Phoenix app is launched using the phoenix package. This app provides a user interface for monitoring and visualizing the performance and activities of your CrewAI agents, giving you real-time insights into their operations.

import phoenix as px
session = px.launch_app()

If you prefer to host a version of Phoenix yourself, see these instructions.

3. Configuring OpenTelemetry Tracing

This snippet configures the OpenTelemetry tracing setup. It establishes a tracer provider and a span processor, which sends tracing data to a local server. This setup is essential for capturing and exporting detailed logs and traces of your AI agents’ actions, allowing you to monitor their performance and troubleshoot issues.

from opentelemetry import trace as trace_api
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk import trace as trace_sdk
from opentelemetry.sdk.trace.export import SimpleSpanProcessor

tracer_provider = trace_sdk.TracerProvider()
span_exporter = OTLPSpanExporter("http://localhost:6006/v1/traces")
span_processor = SimpleSpanProcessor(span_exporter)
tracer_provider.add_span_processor(span_processor)
trace_api.set_tracer_provider(tracer_provider)

4. Instrumenting CrewAI with OpenTelemetry

Here, the CrewAI framework is instrumented with OpenTelemetry using the CrewAIInstrumentor. This step ensures that all actions performed by CrewAI agents are automatically traced and logged, providing visibility into the inner workings of your AI agents without requiring manual instrumentation for each agent or task.

from openinference.instrumentation.crewai import CrewAIInstrumentor

CrewAIInstrumentor().instrument(skip_dep_check=True)

5. Setting API Keys for External Tools

The fifth snippet prompts the user to enter their API keys for OpenAI and Serper if they are not already set in the environment. These keys are necessary for accessing external tools and services that CrewAI agents may need, such as language models or web search capabilities.

import os
import getpass

# Prompt the user for their API keys if they haven't been set
openai_key = os.getenv("OPENAI_API_KEY", "OPENAI_API_KEY")
serper_key = os.getenv("SERPER_API_KEY", "SERPER_API_KEY")

if openai_key == "OPENAI_API_KEY":
    openai_key = getpass.getpass("Please enter your OPENAI_API_KEY: ")

if serper_key == "SERPER_API_KEY":
    serper_key = getpass.getpass("Please enter your SERPER_API_KEY: ")

# Set the environment variables with the provided keys
os.environ["OPENAI_API_KEY"] = openai_key
os.environ["SERPER_API_KEY"] = serper_key

6. Defining Agents, Tasks, and Processes

In this final snippet, two AI agents—’Senior Research Analyst’ and ‘Tech Content Strategist’—are defined, each with specific roles, goals, and tasks. The tasks involve conducting AI research and writing a blog post. These agents are then assembled into a crew that operates in a sequential process, collaborating to complete the tasks. The crew is finally activated to perform its work, demonstrating how CrewAI can automate complex workflows.

from crewai import Agent, Task, Crew, Process
from crewai_tools import SerperDevTool


search_tool = SerperDevTool()

# Define your agents with roles and goals
researcher = Agent(
  role='Senior Research Analyst',
  goal='Uncover cutting-edge developments in AI and data science',
  backstory="""You work at a leading tech think tank.
  Your expertise lies in identifying emerging trends.
  You have a knack for dissecting complex data and presenting actionable insights.""",
  verbose=True,
  allow_delegation=False,
  # You can pass an optional llm attribute specifying what model you wanna use.
  # llm=ChatOpenAI(model_name="gpt-3.5", temperature=0.7),
  tools=[search_tool]
)
writer = Agent(
  role='Tech Content Strategist',
  goal='Craft compelling content on tech advancements',
  backstory="""You are a renowned Content Strategist, known for your insightful and engaging articles.
  You transform complex concepts into compelling narratives.""",
  verbose=True,
  allow_delegation=True
)

# Create tasks for your agents
task1 = Task(
  description="""Conduct a comprehensive analysis of the latest advancements in AI in 2024.
  Identify key trends, breakthrough technologies, and potential industry impacts.""",
  expected_output="Full analysis report in bullet points",
  agent=researcher
)

task2 = Task(
  description="""Using the insights provided, develop an engaging blog
  post that highlights the most significant AI advancements.
  Your post should be informative yet accessible, catering to a tech-savvy audience.
  Make it sound cool, avoid complex words so it doesn't sound like AI.""",
  expected_output="Full blog post of at least 4 paragraphs",
  agent=writer
)

# Instantiate your crew with a sequential process
crew = Crew(
  agents=[researcher, writer],
  tasks=[task1, task2],
  verbose=1,
  process = Process.sequential
)

# Get your crew to work!
result = crew.kickoff()

print("######################")
print(result)

📓📓 Run the code with this fully packaged Colab. 📓📓

Takeaways

• Enhanced Visibility for AI Agents: Observability is critical in maximizing the performance of CrewAI agents. By using tools like Arize Phoenix, you gain deep insights into how your agents operate, allowing you to fine-tune their actions, optimize processes, and ensure that your projects are on track.
• Streamlined Task Management: CrewAI offers a robust framework for orchestrating autonomous agents to handle complex tasks. Understanding the core concepts such as agents, tasks, tools, processes, and pipelines can help you effectively manage and deploy AI-powered solutions tailored to your specific needs.
• Collaborative Intelligence: CrewAI’s ability to foster collaborative intelligence through well-defined processes and customizable agents allows for a highly adaptable and efficient team of AI workers. This opens up new possibilities for automating and enhancing business operations.
• Scalable and Customizable: Whether you need simple task automation or complex multi-stage processes, CrewAI provides the flexibility to scale and customize your AI agent teams, ensuring they meet your strategic objectives.
• Getting Started: With the provided quick start examples, integrating CrewAI with observability tools like Arize Phoenix becomes straightforward, enabling you to immediately begin leveraging the power of autonomous agents in your projects.

Additional Resources

• 📚 DOCS: get started with open source CrewAI observability
• ⭐ SHARE THE LOVE: if you like this guide, please consider giving Arize Phoenix a star on GitHub