Popular GitHub Repositories for Aisafety

Materials for the course Principles of AI: LLMs at UPenn (Stat 9911, Spring 2025). LLM architectures, training paradigms (pre- and post-training, alignment), test-time computation, reasoning, safety and robustness (jailbreaking, oversight, uncertainty), representations, interpretability (circuits), etc.

The official GitHub repo for the paper ""Nuclear Deployed!": Analyzing Catastrophic Risks in Decision-making of Autonomous LLM Agents"

The go-to API for detecting and preventing prompt injection attacks.

Finetuning of Mistral Nemo 13B on the WildJailbreak dataset to produce a red-teaming model