Agents & Multi-Agent

Giskard OSS: Evaluation, Red Teaming and Test Generation for AI Agents

giskard-ai/giskard-oss

An open-source Python library for evaluating and testing LLM-based and agentic systems, including multi-turn evaluations, LLM-as-judge checks and automated vulnerability scanning.

Open official repository ↗ Project homepage ↗
★ 5.7KStars
⑂ 513Forks
85Open issues
PythonLanguage
Apache-2.0License
Q@project.QualityScoreEditorial score

Overview

Giskard v3 provides a modular, async-first architecture for wrapping and testing LLMs, black-box agents and multi-step pipelines. Its beta components cover scenario-based evaluations and agent vulnerability scanning. A dedicated v3 RAG evaluation package is planned, while some vulnerability-scanning and RAG capabilities still rely on the legacy v2 implementation.

Key features

  • Scenario API for evaluating synchronous or asynchronous targets
  • Built-in string matching, comparison, regular-expression and semantic-similarity checks
  • LLM-as-judge checks including Groundedness, Conformity and LLMJudge
  • Multi-turn agent evaluation and scenario suites
  • Automated red teaming and vulnerability scanning
  • Prompt-injection payload dataset
  • Extensible custom ScenarioGenerator support
  • Modular packages with focused dependency sets
  • Optional aggregated telemetry with documented environment-variable opt-out

Requirements, installation and quick start

Install the main package with `pip install giskard`. Add vulnerability and quality scanning with `pip install "giskard[scan]"`, or install the scanner directly with `pip install giskard-scan`. Install an LLM provider SDK with an extra such as `pip install "giskard[openai]"`. Checks can also be installed directly using `pip install giskard-checks`. Legacy v2 is available with `pip install "giskard[llm]>2,<3"`, but it is no longer actively maintained.

Usage

For a quick evaluation, define a target callable, create a `Scenario`, add an interaction and a check such as `Groundedness`, then await `scenario.run()` and print the report. For a vulnerability scan, define an async agent callable and await `vulnerability_scan(target=my_agent, description="A customer support chatbot for an e-commerce platform.", languages=["en"])`. Groundedness and scan generation require a configured LLM provider extra and API key.

Model compatibility and use cases

The v3 target interface can wrap an LLM, black-box agent or multi-step pipeline through a sync or async callable. The stated default model for Groundedness is `openai/gpt-4o-mini`. OpenAI and Anthropic provider extras are mentioned, but a complete supported-model or provider list is not stated in the repository metadata. MCP support is not stated in the repository metadata.

License and risk notes

The repository metadata identifies the license as Apache-2.0. License reference: https://api.github.com/licenses/apache-2.0

Release and maintenance

The supplied README describes v3 as a fresh rewrite focused on dynamic, multi-turn testing, lighter dependencies, a more capable AI vulnerability scanner and improved RAG evaluation. It also states that v2 remains available but is no longer actively maintained. Version numbers and release dates are not stated in the supplied repository record.

Firecrawl

firecrawl/firecrawl

★ 161.1KTypeScript

LangChain

langchain-ai/langchain

★ 143.6KPython

RAGFlow

infiniflow/ragflow

★ 86.7KGo