Giskard OSS – LLM Agent Evaluation and...

Giskard OSS: Evaluation, Red Teaming and Test Generation for AI Agents

giskard-ai/giskard-oss

An open-source Python library for evaluating and testing LLM-based and agentic systems, including multi-turn evaluations, LLM-as-judge checks and automated vulnerability scanning.

概要

Giskard v3 provides a modular, async-first architecture for wrapping and testing LLMs, black-box agents and multi-step pipelines. Its beta components cover scenario-based evaluations and agent vulnerability scanning. A dedicated v3 RAG evaluation package is planned, while some vulnerability-scanning and RAG capabilities still rely on the legacy v2 implementation.

主な機能

Scenario API for evaluating synchronous or asynchronous targets
Built-in string matching, comparison, regular-expression and semantic-similarity checks
LLM-as-judge checks including Groundedness, Conformity and LLMJudge
Multi-turn agent evaluation and scenario suites
Automated red teaming and vulnerability scanning
Prompt-injection payload dataset
Extensible custom ScenarioGenerator support
Modular packages with focused dependency sets
Optional aggregated telemetry with documented environment-variable opt-out

要件、インストール、クイックスタート

Install the main package with `pip install giskard`. Add vulnerability and quality scanning with `pip install "giskard[scan]"`, or install the scanner directly with `pip install giskard-scan`. Install an LLM provider SDK with an extra such as `pip install "giskard[openai]"`. Checks can also be installed directly using `pip install giskard-checks`. Legacy v2 is available with `pip install "giskard[llm]>2,<3"`, but it is no longer actively maintained.

使用方法

For a quick evaluation, define a target callable, create a `Scenario`, add an interaction and a check such as `Groundedness`, then await `scenario.run()` and print the report. For a vulnerability scan, define an async agent callable and await `vulnerability_scan(target=my_agent, description="A customer support chatbot for an e-commerce platform.", languages=["en"])`. Groundedness and scan generation require a configured LLM provider extra and API key.

モデルの互換性とユースケース

The v3 target interface can wrap an LLM, black-box agent or multi-step pipeline through a sync or async callable. The stated default model for Groundedness is `openai/gpt-4o-mini`. OpenAI and Anthropic provider extras are mentioned, but a complete supported-model or provider list is not stated in the repository metadata. MCP support is not stated in the repository metadata.

ライセンスとリスクに関する注意事項

The repository metadata identifies the license as Apache-2.0. License reference: https://api.github.com/licenses/apache-2.0

リリースとメンテナンス

The supplied README describes v3 as a fresh rewrite focused on dynamic, multi-turn testing, lighter dependencies, a more capable AI vulnerability scanner and improved RAG evaluation. It also states that v2 remains available but is no longer actively maintained. Version numbers and release dates are not stated in the supplied repository record.

Giskard OSS: Evaluation, Red Teaming and Test Generation for AI Agents

概要

主な機能

要件、インストール、クイックスタート

使用方法

モデルの互換性とユースケース

ライセンスとリスクに関する注意事項

リリースとメンテナンス

skills — Reusable agent skills and workflows

Hermes Agent

Firecrawl

LangChain

RAGFlow

OpenHands Agent Canvas

Unsloth: Local Model Training, Inference, and Agent Integration

AnythingLLM by Mintplex-Labs

Giskard OSS: Evaluation, Red Teaming and Test Generation for AI Agents

概要

主な機能

要件、インストール、クイックスタート

使用方法

モデルの互換性とユースケース

ライセンスとリスクに関する注意事項

リリースとメンテナンス

関連オープンソースプロジェクト