Agentes e multiagente

Giskard OSS: Evaluation, Red Teaming and Test Generation for AI Agents

giskard-ai/giskard-oss

An open-source Python library for evaluating and testing LLM-based and agentic systems, including multi-turn evaluations, LLM-as-judge checks and automated vulnerability scanning.

Abrir repositório oficial ↗ Página inicial do projeto ↗
★ 5,7KEstrelas
⑂ 513Forks
85Problemas em aberto
PythonIdioma
Apache-2.0Licença
Q@project.QualityScorePontuação editorial

Visão geral

Giskard v3 provides a modular, async-first architecture for wrapping and testing LLMs, black-box agents and multi-step pipelines. Its beta components cover scenario-based evaluations and agent vulnerability scanning. A dedicated v3 RAG evaluation package is planned, while some vulnerability-scanning and RAG capabilities still rely on the legacy v2 implementation.

Principais recursos

  • Scenario API for evaluating synchronous or asynchronous targets
  • Built-in string matching, comparison, regular-expression and semantic-similarity checks
  • LLM-as-judge checks including Groundedness, Conformity and LLMJudge
  • Multi-turn agent evaluation and scenario suites
  • Automated red teaming and vulnerability scanning
  • Prompt-injection payload dataset
  • Extensible custom ScenarioGenerator support
  • Modular packages with focused dependency sets
  • Optional aggregated telemetry with documented environment-variable opt-out

Requisitos, instalação e início rápido

Install the main package with `pip install giskard`. Add vulnerability and quality scanning with `pip install "giskard[scan]"`, or install the scanner directly with `pip install giskard-scan`. Install an LLM provider SDK with an extra such as `pip install "giskard[openai]"`. Checks can also be installed directly using `pip install giskard-checks`. Legacy v2 is available with `pip install "giskard[llm]>2,<3"`, but it is no longer actively maintained.

Uso

For a quick evaluation, define a target callable, create a `Scenario`, add an interaction and a check such as `Groundedness`, then await `scenario.run()` and print the report. For a vulnerability scan, define an async agent callable and await `vulnerability_scan(target=my_agent, description="A customer support chatbot for an e-commerce platform.", languages=["en"])`. Groundedness and scan generation require a configured LLM provider extra and API key.

Compatibilidade do modelo e casos de uso

The v3 target interface can wrap an LLM, black-box agent or multi-step pipeline through a sync or async callable. The stated default model for Groundedness is `openai/gpt-4o-mini`. OpenAI and Anthropic provider extras are mentioned, but a complete supported-model or provider list is not stated in the repository metadata. MCP support is not stated in the repository metadata.

Observações sobre licença e riscos

The repository metadata identifies the license as Apache-2.0. License reference: https://api.github.com/licenses/apache-2.0

Lançamento e manutenção

The supplied README describes v3 as a fresh rewrite focused on dynamic, multi-turn testing, lighter dependencies, a more capable AI vulnerability scanner and improved RAG evaluation. It also states that v2 remains available but is no longer actively maintained. Version numbers and release dates are not stated in the supplied repository record.

Firecrawl

firecrawl/firecrawl

★ 161,1KTypeScript

LangChain

langchain-ai/langchain

★ 143,6KPython

RAGFlow

infiniflow/ragflow

★ 86,7KGo