giskard

The Python Risk Identification Tool for generative AI (PyRIT) is an open source framework built to empower security professionals and engineers to proactively identify risks in generative AI systems.

A curated list of useful resources that cover Offensive AI.

A list of backdoor learning resources

a prompt injection scanner for custom LLM applications

A security scanner for your LLM agentic workflows

RuLES: a benchmark for evaluating rule-following in language models

Toolkits to create a human-in-the-loop approval layer to monitor and guide AI agents workflow in real-time.

A curated list of academic events on AI Security & Privacy

[CCS'24] SafeGen: Mitigating Unsafe Content Generation in Text-to-Image Models

Whistleblower is a offensive security tool for testing against system prompt leakage and capability discovery of an AI application exposed through API. Built for AI engineers, security researchers and folks who want to know what's going on inside the LLM-based app they use daily