\[Live session\] Choosing safer LLMs: From LLM benchmarks to your production agents 🚀 July 21, 2026 \| 5PM CEST 1 Days : 00 Hours : 26 Minutes : 13 Seconds [Save your spot](https://watch.getcontrast.io/register/giskard-live-session-choosing-safer-llms-from-llm-benchmarks-to-your-production-agents?utm_source=website&utm_medium=banner) đŸŽ„ Discover how to red team your AI agents. [Watch demo](/content/giskard-hub-demo/index.html) .svg)\\ \\ \\ 5,660](https://github.com/Giskard-AI/giskard) # Resources [All](/content/knowledge/index.html) [Blog](/content/knowledge-categories/blog/index.html) [Tutorials](/content/knowledge-categories/tutorials/index.html) [White Papers](/content/knowledge-categories/white-papers/index.html) \\ **Phare LLM Benchmark update: 13 new models, and a widening split in AI safety choices** \\ \\ Phare (Potential Harm Assessment & Risk Evaluation) is an independent, multilingual benchmark that evaluates AI models across four critical dimensions, or "modules": hallucination, bias, harmfulness, and vulnerability to jailbreaking attacks. This update adds 13 new models — including GPT 5.5, Claude 5 Sonnet, Kimi K2.6, and DeepSeek V4 — bringing the benchmark to 71 models. The results reveal a widening gap between providers: safety increasingly reflects deliberate engineering choices rather than model capability.\\ \\ View post](/content/knowledge/phare-llm-benchmark-update-13-new-models-and-a-widening-split-in-ai-safety-choices/index.html) \\ **Best AI agent red teaming tools in 2026: understanding features, functions and solutions** \\ \\ In this article, we compare 9 leading AI agents red teaming tools for 2026, evaluating their attack coverage, automation depth, and enterprise integration, to help you detect vulnerabilities in your AI systems.\\ \\ View post](/content/knowledge/best-ai-agent-red-teaming-tools-in-2026-understanding-features-functions-and-solutions/index.html) .png)\\ **Every frontier LLM generates harmful stereotypes in open-ended generation** \\ \\ When given the freedom to write stories, do frontier LLMs fall back on harmful stereotypes? Giskard's R&D team prompted 23 leading models to generate over 650,000 open-ended stories across 10 languages, then analyzed the demographic associations they produced. Every single model generated harmful stereotypes, many of which the models themselves recognized as harmful.\\ \\ View post](/content/knowledge/every-frontier-llm-generates-harmful-stereotypes-in-open-ended-generation/index.html) \\ **Who judges the LLM-as-a-Judge? Meta-Evaluation of an LLM vulnerability scanner** \\ \\ When your LLM vulnerability scanner detects a threat, it relies on an LLM judge to decide whether the attack succeeded. Using one LLM to evaluate another can bring some failures into your evaluation pipeline (false positives, model drift, or context blindness). This article walks through how we meta-evaluated our own LLM-as-a-judge using giskard-checks to freeze expected verdicts, replay attack traces, and detect evaluator regressions in CI.\\ \\ View post](/content/knowledge/who-judges-the-llm-as-a-judge-meta-evaluation-of-an-llm-vulnerability-scanner/index.html) \\ **How Grok got prompt-injected: an X user drained $150,000 from an AI wallet** \\ \\ A X user sent a morse code message tricking Grok into authorizing a $150,000 crypto transfer. This is a prompt injection attack against an autonomous AI agent. In this article we explain how an input obfuscation allowed the heist to occur, and how to test your own agents to prevent this type of failures.\\ \\ View post](/content/knowledge/how-grok-got-prompt-injected-an-x-user-drained-150-000-from-an-ai-wallet/index.html) \\ **Securing AI agents with Europe's first sovereign guardrail platform** \\ \\ Giskard has launched Guards, the first independent, EU-sovereign guardrail platform that allows regulated enterprises to secure their AI applications on-premise. Moving beyond the limitations of generic content filters, the platform is context-aware and built specifically for modern AI agents, securing the full execution chain via a Policy-as-Code framework that includes ready-to-use EU AI Act and OWASP Top 10 LLM compliance packs. To showcase this new standard in enterprise AI security, our technical team is hosting an in-depth live session on May 13, 2026.\\ \\ View post](/content/knowledge/securing-ai-agents-with-europes-first-sovereign-guardrail-platform/index.html) \\ **A Cursor AI agent wiped a production database in 9 seconds: Excessive Agency AI failure** \\ \\ In April 2026, a Cursor AI coding agent running Claude Opus 4.6 deleted a startup's entire production database and every backup in a single API call, in nine seconds. This incident is a case of "Excessive Agency," where over-privileged credentials and autonomous reasoning loops bypass security controls. In this article we analyse what failed and how to prevent it.\\ \\ View post](/content/knowledge/a-cursor-ai-agent-wiped-a-production-database-in-9-seconds-excessive-agency-ai-failure/index.html) \\ **Claude Mythos: Analyzing Anthropic’s new frontier model for AI security** \\ \\ In this article, we analyze Anthropic's newly announced Claude Mythos model and its announced capabilities in automated vulnerability discovery and exploit generation. We explore how this frontier model impacts the cybersecurity landscape.\\ \\ View post](/content/knowledge/claude-mythos-analyzing-anthropics-new-frontier-model-for-ai-security/index.html) \\ **Announcing Giskard OSS v3: Our Eval and Red Teaming Library, Rebuilt for the Agentic Era** \\ \\ Since its creation, Giskard has been on a mission to help teams ship AI they can trust. Today, we are thrilled to announce Giskard v3, a complete evolution of the library designed for the modern landscape of Large Language Models (LLMs) and sophisticated AI agents.\\ \\ View post](/content/knowledge/announcing-giskard-open-source-v3/index.html) \\ **LiteLLM supply chain compromise: what happened, impact on Giskard, and lessons learned** \\ \\ On 24 March 2026, two malicious versions of the widely used LiteLLM Python library (1.82.7 and 1.82.8) were published to PyPI. The compromised packages ran a credential-stealing attack that exfiltrated secrets (SSH keys, cloud provider sessions, Terraform state, etc.) and attempted lateral movement into Kubernetes clusters. By now, both versions have been removed by PyPI.\\ \\ View post](/content/knowledge/litellm-supply-chain-attack-2026/index.html) .webp)\\ **What is Shadow AI and how to prevent this threat in AI Security** \\ \\ Shadow AI happens when employees bypass IT to use unapproved generative AI tools, creating severe data leakage and compliance vulnerabilities. In this article, we explore the specific risks associated with these unvetted models, such as intellectual property leaks and flawed business decisions. Finally, we break down exactly how to prevent this threat by moving past ineffective bans and implementing secure alternatives with real-time AI guardrails.\\ \\ View post](/content/knowledge/what-is-shadow-ai-and-how-to-prevent-this-threat-in-ai-security/index.html) \\ **A new look for Sophia: the story behind Giskard’s rebranding** \\ \\ Giskard recently embarked on a rebranding journey to align our friendly, community-loved visual identity with our deep expertise in AI security. By collaborating with talented designers, we refreshed our green colour palette, introduced modern typography, and gave our beloved turtle mascot, Sophia, a high-tech 3D makeover. Read the behind the scenes to know all the anecdotes!\\ \\ View post](/content/knowledge/a-new-look-for-sophia-the-story-behind-giskards-rebranding/index.html) \\ **CoT Forgery: An LLM vulnerability in Chain-of-Thought prompting** \\ \\ Chain-of-Thought (CoT) Forgery is a prompt injection attack where adversaries plant fake internal reasoning to trick AI models into bypassing their own safety guardrails. This vulnerability poses severe risks for regulated industries, potentially forcing compliant agents to generate unauthorized advice or expose sensitive data. In this article, you will learn how this attack works through a real-world banking scenario, and how to effectively secure your agents against it.\\ \\ View post](/content/knowledge/cot-forgery-an-llm-vulnerability-in-chain-of-thought-prompting/index.html) \\ **OpenClaw security vulnerabilities include data leakage and prompt injection risks** \\ \\ This article explores the critical security failures of the OpenClaw agentic AI, which allowed sensitive data to leak across user sessions and IM channels. It examines how architectural weaknesses in the Control UI and session management created direct paths for prompt injection and unauthorized tool use. Finally, it outlines the essential hardening steps and systematic red-teaming strategies required to transform a vulnerable "fun bot" into a secure enterprise assistant.\\ \\ View post](/content/knowledge/openclaw-security-vulnerabilities-include-data-leakage-and-prompt-injection-risks/index.html) .png)\\ **Model Context Protocol: Understanding MCP security risks and prevention methods** \\ \\ MCP servers pose significant MCP security risks due to their ability to execute commands and perform API calls. Even "official" Model Context Protocol setups face MCP vulnerability and tool poisoning. In this article, you'll learn the primary MCP security issues, from injection attacks to supply chain risks, and how to mitigate these MCP cyber security threats for your AI agents.\\ \\ View post](/content/knowledge/model-context-protocol-understanding-mcp-security-risks-and-prevention-methods/index.html) \\ **How an agentic AI transcription tool triggered a healthcare data leakage** \\ \\ A recent AI incident in healthcare revealed how an automated transcription agent accidentally leaked patient data by joining a meeting via a former employee's stale calendar invite. This failure shows the risk of "Shadow AI" and sensitive information disclosure when agentic tools operate without strict contextual authorization.\\ \\ View post](/content/knowledge/how-an-agentic-ai-transcription-tool-triggered-a-healthcare-data-leakage/index.html) \\ **The New York Times lawsuit against Perplexity exposes AI copyright issues and AI hallucinations** \\ \\ The New York Times has sued Perplexity, alleging that the AI startup's search engine unlawfully processes copyrighted articles to generate summaries that directly compete with the publication. The lawsuit further accuses Perplexity of brand damage, citing instances where the model "hallucinates" misinformation and falsely attributes it to the Times. In this article, we analyze the technical mechanisms behind these failures and demonstrate how to prevent both hallucinations and unauthorized content usage through automated groundedness evaluations.\\ \\ View post](/content/knowledge/the-new-york-times-lawsuit-against-perplexity-exposes-ai-copyright-issues-and-ai-hallucinations/index.html) \\ **AI hallucinations and the AI failure in a French Court** \\ \\ A recent ruling by the tribunal judiciaire of PĂ©rigueux exposed an AI failure where a claimant submitted "untraceable" precedents, underscoring the dangers of unchecked generative AI in court. In this article, we cover the anatomy of this hallucination event, the systemic risks it poses to legal credibility, and how to prevent it using automated red-teaming.\\ \\ View post](/content/knowledge/ai-hallucinations-and-the-ai-failure-in-a-french-court/index.html) \\ **Best-of-N jailbreaking: The automated LLM attack that takes only seconds** \\ \\ This article examines "Best-of-N," a sophisticated jailbreaking technique where attackers systematically generate prompt variations to probabilistically overcome guardrails. We will break down the mechanics of this attack, demonstrate a realistic industry-specific scenario, and outline how automated probing can detect these vulnerabilities before they impact your production systems.\\ \\ View post](/content/knowledge/best-of-n-jailbreaking-the-automated-llm-attack-that-takes-only-seconds/index.html) \\ **When AI financial advice goes wrong: ChatGPT, Copilot, and Gemini failed UK consumers** \\ \\ In November 2025, ChatGPT, Microsoft Copilot, Google Gemini, and Meta AI were caught giving UK consumers dangerous financial advice: recommending they exceed ISA contribution limits, providing incorrect tax guidance, and directing them to expensive services instead of free government alternatives. In this article, we analyze the UK incident, explain why these chatbots failed, and show how to prevent similar failures in your AI systems.\\ \\ View post](/content/knowledge/when-ai-financial-advice-goes-wrong-chatgpt-copilot-and-gemini-failed-uk-consumers/index.html) \\ **OWASP top 10 for agentic applications 2026: Understanding the risks of agents and tools** \\ \\ The Top 10 for Agentic Applications, released in December 2025, lists the highest-impact threats to autonomous AI agentic applications, systems that plan, decide, and act across tools and steps. It distills the top threats in a practical manner, building directly on prior OWASP work while highlighting agent-specific amplifiers, such as delegation and multi-step execution. The list pivots from passive LLM risks to active agent behaviors. Agents are treated as principals with goals, tools, memory, and inter-agent protocols as distinct attack surfaces.\\ \\ View post](/content/knowledge/owasp-top-10-for-agentic-application-2026/index.html) \\ **Phare LLM benchmark V2: Reasoning models don't guarantee better security** \\ \\ Phare (Potential Harm Assessment & Risk Evaluation) is an independent, multilingual benchmark designed to evaluate AI models across four critical dimensions, or “modules”: hallucination, bias, harmfulness, and vulnerability to jailbreaking attacks. This second version expands our evaluation to include reasoning models from leading providers, allowing us to assess whether these advanced systems represent a meaningful improvement in AI safety.\\ \\ View post](/content/knowledge/reasoning-models-dont-guarantee-better-security/index.html) \\ **Agentic tool extraction: Multi-turn attack that exposes the agent's internal functions** \\ \\ Agentic Tool Extraction (ATE) is a multi-turn reconnaissance attack to extract complete tool schemas, function names, parameters, types, and return values. ATE exploits conversation context, using seemingly benign questions that bypass standard filters to build a technical blueprint of the agent's capabilities. In this article, we demonstrate how attackers weaponize extracted schemas to craft precise exploits and explain how conversation-level defenses can detect progressive extraction patterns before tool signatures are fully exposed.\\ \\ View post](/content/knowledge/agentic-tool-extraction-multi-turn-attack-that-exposes-the-agents-internal-functions/index.html) \\ **Risk assessment for LLMs and AI agents: OWASP, MITRE Atlas, and NIST AI RMF explained** \\ \\ There are three major tools for assessing risks associated with LLMs and AI Agents: OWASP, MITRE Attack and NIST AI RMF. Each of them has its own approach to risk and security, while examining it from different angles with varying levels of granularity and organisational scope. This blog will help you understand them.\\ \\ View post](/content/knowledge/risk-assessment-for-llms-and-ai-agents-owasp-mitre-atlas-and-nist-ai-rmf-explained/index.html) \\ **Beyond sycophancy: The risk of vulnerable misguidance in AI medical advice** \\ \\ Healthcare employees in Hyderabad have noticed a disturbing direction in self-doctoring: two of their patients relied on generic AI chatbot advice for their healthcare interventions, and some of them suffered serious medical consequences. Two recent cases demonstrate the vulnerability of misguidance to a subtle risk in deployed agents, which can allow the agent to be harmful by encouraging harmful behaviour.\\ \\ View post](/content/knowledge/beyond-sycophancy-the-risk-of-vulnerable-misguidance-in-ai-medical-advice/index.html) \\ **Tree of attacks with pruning: The automated method for jailbreaking LLMs** \\ \\ Tree of Attacks with Pruning (TAP) automates the discovery of prompt injection vulnerabilities in LLMs through systematic trial and refinement. In this article, you'll learn how TAP probe works, see a concrete example of automated jailbreaking in a business context, and understand how to incorporate TAP attack into your AI security strategy. \\ \\ View post](/content/knowledge/tree-of-attacks-with-pruning-the-automated-method-for-jailbreaking-llms/index.html) \\ **AI phishing attack in Australia: 270,000 fake government emails expose AI security gap** \\ \\ A massive phishing campaign impersonating Australian government services sent over 270,000 fake emails in four months, with security researchers pointing to AI assistance based on the sudden jump in sophistication and quality. The real security challenge isn't a vulnerability in the models themselves, it's that attackers can use LLM capabilities to rapidly generate convincing phishing content, either through contextual framing or jailbreaking techniques that bypass safety guardrails. \\ \\ View post](/content/knowledge/ai-phishing-attack-in-australia-270-000-fake-government-emails-expose-ai-security-gap/index.html) \\ **OWASP Top 10 for LLM 2025: Understanding the Risks of Large Language Models** \\ \\ The landscape of large language model security has evolved significantly since the release of OWASP’s Top 10 for LLM Applications in 2023, which we covered in our blog at the time. The 2025 edition represents a significant update of our understanding of how Gen AI systems are being deployed in production environments. The update does not come as a surprise, as companies like MITRE also continuously update their risk framework, Atlas. The lessons from enterprise deployments, and direct feedback from a global community of developers, security professionals, and data scientists working in AI security.\\ \\ View post](/content/knowledge/owasp-top-10-for-llm-2025-understanding-the-risks-of-large-language-models/index.html) .png)\\ **Anthropic claims Claude Code was used for the first Autonomous AI cyber espionage campaign** \\ \\ Anthropic has reported that Claude Code was used to orchestrate a cyber espionage campaign, with the AI independently executing 80–90% of the tactical operations. In this article, we analyze the mechanics of this attack, and explain how organizations can leverage continuous red teaming to defend against these threats.\\ \\ View post](/content/knowledge/anthropic-claims-claude-code-was-used-for-autonomous-ai-cyber-espionage-campaign/index.html) \\ **Understanding single-turn, multi-turn, and dynamic agentic attacks in AI red teaming** \\ \\ AI red teaming has evolved from simple prompt injection into three distinct attack categories: single-turn attacks that test immediate defenses, multi-turn attacks that build context across conversations, and dynamic agentic attacks that autonomously adapt strategies in real-time. This article breaks down all three attack categories, and explains how to implement red teaming to protect production AI systems.\\ \\ View post](/content/knowledge/understanding-single-turn-multi-turn-and-dynamic-agentic-attacks-in-ai-red-teaming/index.html) \\ **Are AI browsers safe? A security and vulnerability analysis of OpenAI Atlas** \\ \\ OpenAI's Atlas browser is powered by ChatGPT, but its design choices expose unknowing users to numerous risks. They were drawn in by the wonderful marketing promise of fast, helpful, and reliable AI, while articles about vulnerability exploitation continue to flood the news, just days after the beta release.\\ \\ View post](/content/knowledge/are-ai-browsers-safe-a-security-and-vulnerability-analysis-of-openai-atlas/index.html) \\ **When your AI agent tells you what you want to hear: Understanding Sycophancy in LLMs** \\ \\ Sycophancy in LLMs is the tendency of a model to align its responses with the user's beliefs or assumptions, prioritizing pleasing the user over truthfulness or factual accuracy. In this article, you'll learn what are the risks of sycophancy, see real-world scenarios about this weakness, and discover how to detect and prevent it in production AI systems.\\ \\ View post](/content/knowledge/when-your-ai-agent-tells-you-what-you-want-to-hear-understanding-sycophancy-in-llms/index.html) \\ **Best AI Red Team Tools 2025: A practical guide to features and functions** \\ \\ In this article, we compare 7 leading AI red teaming tools for 2025, evaluating their attack coverage, automation depth, and enterprise integration, to help you uncover vulnerabilities before malicious actors exploit them.\\ \\ View post](/content/knowledge/best-ai-red-teaming-tools-2025-comparison-features/index.html) .png)\\ **LLM business alignment: Detecting AI hallucinations and misaligned agentic behavior in business systems** \\ \\ Adversarial red teaming isn't enough, as even agents that pass security tests fail in production by hallucinating product features, refusing legitimate requests, omitting critical details, and contradicting policies. In this article, you'll learn what LLM business alignment actually means, why these failures kill AI adoption in production, and how to test your agents using knowledge bases, automated probes, and validation checks.\\ \\ View post](/content/knowledge/llm-business-alignment-detecting-ai-hallucinations-and-misaligned-agentic-behavior-in-business-systems/index.html) \\ **Cross Session Leak: when your AI assistant becomes a data breach** \\ \\ Cross Session Leak is a data exfiltration vulnerability where sensitive information from one user's session bleeds into another user's session, bypassing authentication controls in multi-tenant AI systems. This occurs when AI architectures fail to properly isolate session data through misconfigured caches, shared memory, or improperly scoped context. This article explores how cross session leak attacks work, examines a healthcare scenario, and provides technical strategies to detect and prevent these vulnerabilities.\\ \\ View post](/content/knowledge/cross-session-leak-when-your-ai-assistant-becomes-a-data-breach/index.html) \\ **Function calling in LLMs: Testing agent tool usage for AI Security** \\ \\ Function calling enables LLMs to execute code, query databases, and interact with external systems, transforming them from text generators into agents that take real-world actions. This capability introduces critical security risks: hallucinated parameters, unauthorized access, and unintended consequences. This article explains how function calling works, the vulnerabilities it creates, and how to test agents systematically.\\ \\ View post](/content/knowledge/function-calling-in-llms-testing-agent-tool-usage-for-ai-security/index.html) \\ **GOAT Automated Red Teaming: Multi-turn attack techniques to jailbreak LLMs** \\ \\ GOAT (Generative Offensive Agent Tester) is an automated multi-turn jailbreaking attack that chains adversarial prompting techniques across conversations to bypass AI safety measures. Unlike traditional single-prompt attacks, GOAT adapts dynamically at each conversation turn, mimicking how real attackers interact with AI systems through seemingly innocent exchanges that gradually escalate toward harmful objectives. This article explores how GOAT automated red teaming works, and provides strategies to defend enterprise AI systems against these multi-turn threats.\\ \\ View post](/content/knowledge/goat-automated-red-teaming-multi-turn-attack-techniques-to-jailbreak-llms/index.html) \\ **\[Release notes\]: New LLM vulnerability scanner for dynamic & multi-turn Red Teaming** \\ \\ We're releasing an upgraded LLM vulnerability scanner that deploys autonomous red teaming agents to conduct dynamic, multi-turn attacks across 40+ probes, covering both security and business failures. Unlike static testing tools, this new scanner adapts attack strategies in real-time to detect sophisticated conversational vulnerabilities.\\ \\ View post](/content/knowledge/new-llm-vulnerability-scanner-for-dynamic-multi-turn-red-teaming/index.html) .png)\\ **How LLM jailbreaking can bypass AI security with multi-turn attacks** \\ \\ Multi-turn jailbreaking attacks like Crescendo bypass AI security measures by gradually steering conversations toward harmful outputs through innocent-seeming steps, creating serious business risks that standard single-message testing misses. This article reveals how these attacks work with real-world examples and provides practical techniques to detect and prevent them.\\ \\ View post](/content/knowledge/how-llm-jailbreaking-can-bypass-ai-security-with-multi-turn-attacks/index.html) .png)\\ **RealPerformance, A Dataset of Language Model Business Compliance Issues** \\ \\ Giskard launches RealPerformance to address the gap between the focus on security and business compliance issues: the first systematic dataset of business performance failures in conversational AI, based on real-world testing across banks, insurers, and other industries.\\ \\ View post](/content/knowledge/realperformance/index.html) \\ **LLMs recognise bias but also reproduce harmful stereotypes: an analysis of bias in leading LLMs** \\ \\ Our Phare benchmark reveals that leading LLMs reproduce stereotypes in stories despite recognising bias when asked directly. Analysis of 17 models shows the generation vs discrimination gap.\\ \\ View post](/content/knowledge/llms-recognise-bias-but-also-reproduce-harmful-stereotypes/index.html) .png)\\ **LLM Observability vs LLM Evaluation: Building Comprehensive Enterprise AI Testing Strategies** \\ \\ Enterprise AI teams often treat observability and evaluation as competing priorities, leading to gaps in either technical monitoring or quality assurance.\\ \\ View post](/content/knowledge/llm-observability-vs-llm-evaluation/index.html) .png)\\ **Real-Time Guardrails vs Batch LLM Evaluations: A Comprehensive AI Testing Strategy** \\ \\ Enterprise AI teams need both immediate protection and deep quality insights but often treat guardrails and batch evaluations as competing priorities.\\ \\ View post](/content/knowledge/real-time-guardrails-vs-batch-llm-evaluations/index.html) \\ **A Practical Guide to LLM Hallucinations and Misinformation Detection** \\ \\ Explore how false content is generated by AI and why it's critical to understand LLM vulnerabilities for safer, more ethical AI use.\\ \\ View post](/content/knowledge/a-practical-guide-to-llm-hallucinations-and-misinformation-detection/index.html) .png)\\ **A Practical Guide on AI Security and LLM Vulnerabilities** \\ \\ Discover the key vulnerabilities in Large Language Models (LLMs) and learn how to mitigate AI risks with clear overviews and practical examples. Stay ahead in safe and responsible AI deployment.\\ \\ View post](/content/knowledge/a-practical-guide-on-ai-security-and-llm-vulnerabilities/index.html) \\ **Good answers are not necessarily factual answers: an analysis of hallucination in leading LLMs** \\ \\ We're sharing the first results from Phare, our multilingual benchmark for evaluating language models. The benchmark research reveals leading LLMs confidently produce factually inaccurate information. Our evaluation of top models from eight AI labs shows they generate authoritative-sounding responses containing completely fabricated details, particularly when handling misinformation.\\ \\ View post](/content/knowledge/good-answers-are-not-necessarily-factual-answers-an-analysis-of-hallucination-in-leading-llms/index.html) \\ **Secure AI Agents: Exhaustive testing with continuous LLM Red Teaming** \\ \\ Testing AI agents presents significant challenges as vulnerabilities continuously emerge, exposing organizations to reputational and financial risks when systems fail in production. Giskard's LLM Evaluation Hub addresses these challenges through adversarial LLM agents that automate exhaustive testing, annotation tools that integrate domain expertise, and continuous red teaming that adapts to evolving threats.\\ \\ View post](/content/knowledge/secure-ai-agents-exhaustive-testing-with-continuous-llm-red-teaming/index.html) \\ **Giskard announces Phare, a new open & multi-lingual LLM Benchmark** \\ \\ During the Paris AI Summit, Giskard launches Phare, a new open & independent LLM benchmark to evaluate key AI security dimensions including hallucination, factual accuracy, bias, and potential for harm across several languages, with Google DeepMind as research partner. This initiative is meant to provide open measurements to assess trustworthiness of Generative AI models in real applications.\\ \\ View post](/content/knowledge/giskard-announces-phare-a-new-llm-evaluation-benchmark/index.html) .png)\\ **DeepSeek R1: Complete analysis of capabilities and limitations** \\ \\ In this article, we provide a detailed analysis of DeepSeek R1, comparing its performance against leading AI models like GPT-4o and O1. Our testing reveals both impressive knowledge capabilities and significant concerns, particularly regarding the model's tendency to generate hallucinations. Through concrete examples, we examine how R1 handles politically sensitive topics.\\ \\ View post](/content/knowledge/deepseek-r1-complete-analysis-of-performance-and-limitations/index.html) \\ **\[Release notes\] Giskard integrates with LiteLLM: Simplifying LLM agent testing across foundation models** \\ \\ Giskard's integration with LiteLLM enables developers to test their LLM agents across multiple foundation models. The integration enhances Giskard's core features - LLM Scan for vulnerability assessment and RAGET for RAG evaluation - by allowing them to work with any supported LLM provider: whether you're using major cloud providers like OpenAI and Anthropic, local deployments through Ollama, or open-source models like Mistral.\\ \\ View post](/content/knowledge/giskard-integrates-with-litellm-simplifying-llm-agent-testing-across-foundation-models/index.html) \\ **AI Liability in the EU: Business guide to Product (PLD) and AI Liability Directives (AILD)** \\ \\ The EU is establishing an AI liability framework through two key regulations: the Product Liability Directive (PLD), taking effect in 2024, and the proposed AI Liability Directive (AILD). The PLD introduces strict liability for defective AI systems and software, while the AILD addresses negligent use, though its final form remains under debate. Learn in this article the key points of these regulations and how they will impact businesses.\\ \\ View post](/content/knowledge/ai-liability-in-the-eu-business-guide-to-product-pld-and-ai-liability-directives-aild/index.html) .png)\\ **Giskard Vision: Enhance Computer Vision models for image classification, object an landmark detection** \\ \\ Giskard Vision is a new module in our open-source library designed to assess and improve computer vision models. It offers automated detection of performance issues, biases, and ethical concerns in image classification, object detection, and landmark detection tasks. The article provides a step-by-step guide on how to integrate Giskard Vision into existing workflows, enabling data scientists to enhance the reliability and fairness of their computer vision systems.\\ \\ View post](/content/knowledge/assessing-the-quality-of-computer-vision-models-with-giskard-vision/index.html) \\ **Evaluating LLM applications: Giskard Integration with NVIDIA NeMo Guardrails** \\ \\ Giskard has integrated with NVIDIA NeMo Guardrails to enhance the safety and reliability of LLM-based applications. This integration allows developers to better detect vulnerabilities, automate rail generation, and streamline risk mitigation in LLM systems. By combining Giskard with NeMo Guardrails organizations can address critical challenges in LLM development, including hallucinations, prompt injection and jailbreaks.\\ \\ View post](/content/knowledge/evaluating-llm-applications-giskard-integration-with-nvidia-nemo-guardrails/index.html) [![Council of Europe - AI Treaty [1]](https://cdn.prod.website-files.com/601d6f7e527cf16fd11a1aae/66df2d190b613f74b8266ff9_Twitter%20post%20-%2075%20(1).webp)\\ **Global AI Treaty: EU, UK, US, and Israel sign landmark AI regulation** \\ \\ The Council of Europe has signed the world's first AI treaty marking a significant step towards global AI governance. This Framework Convention on Artificial Intelligence aligns closely with the EU AI Act, adopting a risk-based approach to protect human rights and foster innovation. The treaty impacts businesses by establishing requirements for trustworthy AI, mandating transparency, and emphasizing risk management and compliance.\\ \\ View post](/content/knowledge/global-ai-treaty-eu-uk-us-and-israel-sign-landmark-ai-regulation/index.html) \\ **The EU AI Act published in the EU Official Journal: Next steps for AI Regulation** \\ \\ The EU AI Act, published on July 12, 2024, establishes the world's first comprehensive regulatory framework for AI technologies, with a gradual implementation timeline from 2024 to 2027. It adopts a risk-based approach, imposing varying requirements on AI systems based on their risk level.\\ \\ View post](/content/knowledge/eu-ai-act-official-journal-next-steps-for-ai-regulation/index.html) \\ **Giskard leads GenAI Evaluation in France 2030's ArGiMi Consortium** \\ \\ The ArGiMi consortium, including Giskard, Artefact and Mistral AI, has won a France 2030 project to develop next-generation French LLMs for businesses. Giskard will lead efforts in AI safety, ensuring model quality, conformity, and security. The project will be open-source ensuring collaboration, and aiming to make AI more reliable, ethical, and accessible across industries.\\ \\ View post](/content/knowledge/giskard-leads-genai-evaluation-in-france-2030s-argimi-consortium/index.html) \\ **Partnership announcement: Bringing Giskard LLM evaluation to Databricks** \\ \\ Giskard has integrated with Databricks MLflow to enhance LLM testing and deployment. This collaboration allows AI teams to automatically identify vulnerabilities, generate domain-specific tests, and log comprehensive reports directly into MLflow. The integration aims to streamline the development of secure, reliable, and compliant LLM applications, addressing key risks like prompt injection, hallucinations, and unintended data disclosures.\\ \\ View post](/content/knowledge/partnership-giskard-llm-evaluation-databricks/index.html) \\ **LLMOps: MLOps for Large Language Models** \\ \\ This article explores LLMOps, detailing its challenges and best practices for managing Large Language Models (LLMs) in production. It compared LLMOps with traditional MLOps, covering hardware needs, performance metrics, and handling non-deterministic outputs. The guide outlines steps for deploying LLMs, including model selection, fine-tuning, and continuous monitoring, while emphasizing quality and security management.\\ \\ View post](/content/knowledge/llmops-mlops-for-large-language-models/index.html) \\ **Defending LLMs against Jailbreaking: Definition, examples and prevention** \\ \\ Jailbreaking refers to maliciously manipulating Large Language Models (LLMs) to bypass their ethical constraints and produce unauthorized outputs. This emerging threat arises from combining the models' high adaptability with inherent vulnerabilities that attackers can exploit through techniques like prompt injection. Mitigating jailbreaking risks requires a holistic approach involving robust security measures, adversarial testing, red teaming, and ongoing vigilance to safeguard the integrity and reliability of AI systems.\\ \\ View post](/content/knowledge/defending-llms-against-jailbreaking-definition-examples-prevention/index.html) .webp)\\ **Data Poisoning attacks on Enterprise LLM applications: AI risks, detection, and prevention** \\ \\ Data poisoning is a real threat to enterprise AI systems like Large Language Models (LLMs), where malicious data tampering can skew outputs and decision-making processes unnoticed. This article explores the mechanics of data poisoning attacks, real-world examples across industries, and best practices to mitigate risks through red teaming, and automated evaluation tools.\\ \\ View post](/content/knowledge/data-poisoning-attacks-on-enterprise-llm-applications-ai-risks-detection-and-prevention/index.html) \\ **\[Release notes\] LLM app vulnerability scanner for Mistral, OpenAI, Ollama, and Custom Local LLMs** \\ \\ Releasing an upgraded version of Giskard's LLM scan for comprehensive vulnerability assessments of LLM applications. New features include more accurate detectors through optimized prompts and expanded multi-model compatibility supporting OpenAI, Mistral, Ollama, and custom local LLMs. This article also covers an initial setup guide for evaluating LLM apps.\\ \\ View post](/content/knowledge/release-notes-llm-app-vulnerability-scanner-mistral-openai-ollama-custom-local-llms/index.html) .webp)\\ **Guide to LLM evaluation and its critical impact for businesses** \\ \\ As businesses increasingly integrate LLMs into several applications, ensuring the reliability of AI systems is key. LLMs can generate biased, inaccurate, or even harmful outputs if not properly evaluated. This article explains the importance of LLM evaluation, and how to do it (methods and tools). It also present Giskard's comprehensive solutions for evaluating LLMs, combining automated testing, customizable test cases, and human-in-the-loop.\\ \\ View post](/content/knowledge/guide-to-understand-llm-evaluation/index.html) \\ **New course with DeepLearningAI: Red Teaming LLM Applications** \\ \\ Our new course in collaboration with DeepLearningAI team provides training on red teaming techniques for Large Language Model (LLM) and chatbot applications. Through hands-on attacks using prompt injections, you'll learn how to identify vulnerabilities and security failures in LLM systems.\\ \\ View post](/content/knowledge/course-deeplearningai-red-teaming-llm-applications/index.html) .webp)\\ **LLM Red Teaming: Detect safety & security breaches in your LLM apps** \\ \\ Introducing our LLM Red Teaming service, designed to enhance the safety and security of your LLM applications. Discover how our team of ML Researchers uses red teaming techniques to identify and address LLM vulnerabilities. Our new service focuses on mitigating risks like misinformation and data leaks by developing comprehensive threat models.\\ \\ View post](/content/knowledge/llm-red-teaming/index.html) \\ **EU AI ACT: 8 Takeaways from the Council's Final Approval** \\ \\ The Council of the EU has recently voted unanimously on the final version of the European AI Act. It’s a significant step forward in its efforts to legislate the first AI law in the world. The Act establishes a regulatory framework for the safe use and development of AI, categorizing AI systems according to their associated risk. In the coming months, the text will enter the last stage of the legislative process, where the European Parliament will have a final vote on the AI Act. \\ \\ View post](/content/knowledge/eu-ai-act-8-takeaways-councils-final-approval/index.html) \\ **Giskard's retrospective of 2023 and a glimpse into what's next for 2024!** \\ \\ 2023 retrospective, covering people, company, customers, and product news, also offers a glimpse into what's next for 2024. Our team keeps growing, with new offices in Paris, new customers, and product features. Our GitHub repo has nearly reached 2500 stars, and we were Product of the Day on Product Hunt. All this and more in our 2023 review.\\ \\ View post](/content/knowledge/2023-in-review/index.html) \\ **EU AI Act: The EU Strikes a Historic Agreement to Regulate AI** \\ \\ The EU's AI Act establishes rules for AI use and development, focusing on ethical standards and safety. It categorizes AI systems, highlights high-risk uses, and sets compliance requirements. This legislation, a first in global AI governance, signals a shift towards responsible AI innovation in Europe.\\ \\ View post](/content/knowledge/eu-ai-act-the-eu-strikes-a-historic-agreement-to-regulate-ai/index.html) \\ **Biden's Executive Order: The Push to Regulate AI in the US** \\ \\ One year after the launch of ChatGPT, regulators worldwide are still figuring out how to regulate Generative AI. The EU is going through intense debates on how to close the so-called 'EU AI Act' after two years of legislative process. At the same time, only one month ago, the White House surprised everyone with a landmark Executive Order to regulate AI in the US. In this article, I delve into the Executive Order and advance some ideas on how it can impact the whole AI regulatory landscape.\\ \\ View post](/content/knowledge/president-biden-strategy-to-regulate-ai/index.html) .png)\\ **Our LLM Testing solution is launching on Product Hunt 🚀** \\ \\ We have just launched Giskard v2, extending the testing capabilities of our library and Hub to Large Language Models. Support our launch on Product Hunt and explore our new integrations with Hugging Face, Weights & Biases, MLFlow, and Dagshub. A big thank you to our community for helping us reach over 1900 stars on GitHub.\\ \\ View post](/content/knowledge/our-llm-testing-solution-is-launching-on-product-hunt/index.html) \\ **Towards AI Regulation: How Countries are Shaping the Future of Artificial Intelligence** \\ \\ In this article we will present the challenges and approaches to AI Regulation in major jurisdictions such as the European Union, the United States, China, Canada and the UK. Explore the growing impact of AI on society and how AI quality tools like Giskard ensure reliable models and compliance.\\ \\ View post](/content/knowledge/how-countries-are-shaping-the-future-of-ai/index.html) \\ **AI Safety and Security: A Conversation with Giskard's Co-Founder and CPO** \\ \\ Giskard's Co-Founder and CPO, Jean-Marie John-Mathews was recently interviewed by Safety Detectives and he shared insights into the company's mission to advance AI Safety and Quality. In this interview, Jean-Marie explains the strategies, vulnerabilities, and ethical considerations at the forefront of AI technology, as Giskard bridges the gap between AI models and real-world applications. \\ \\ View post](/content/knowledge/ai-safety-and-security-interview/index.html) \\ **AI Safety at DEFCON 31: Red Teaming for Large Language Models (LLMs)** \\ \\ DEFCON, one of the world's premier hacker conventions, this year saw a unique focus at the AI Village: red teaming of Large Language Models (LLMs). Instead of conventional hacking, participants were challenged to use words to uncover AI vulnerabilities. The Giskard team was fortunate to attend, witnessing firsthand the event's emphasis on understanding and addressing potential AI risks.\\ \\ View post](/content/knowledge/ai-safety-defcon-31-red-teaming-llms/index.html) \\ **OWASP Top 10 for LLM 2023: Understanding the Risks of Large Language Models** \\ \\ In this post, we introduce OWASP's first version of the Top 10 for LLM, which identifies critical security risks in modern LLM systems. It covers vulnerabilities like Prompt Injection, Insecure Output Handling, Model Denial of Service, and more. Each vulnerability is explained with examples, prevention tips, attack scenarios, and references. The document serves as a valuable guide for developers and security practitioners to protect LLM-based applications and data from potential attacks.\\ \\ View post](/content/knowledge/owasp-top-10-for-llm-2023-understanding-the-risks-of-large-language-models/index.html) \\ **White House pledge targets AI regulation with Top Tech companies** \\ \\ In a significant move towards AI regulation, President Biden convened a meeting with top tech companies, leading to a White House pledge that emphasizes AI safety and transparency. Companies like Google, Amazon, and OpenAI have committed to pre-release system testing, data transparency, and AI-generated content identification. As tech giants signal their intent, concerns remain regarding the specificity of their commitments. \\ \\ View post](/content/knowledge/white-house-pledge-targets-ai-regulation-with-top-tech-companies/index.html) \\ **1,000 GitHub stars, 3M€, and new LLM scan feature  đŸ’«** \\ \\ We've reached an impressive milestone of 1,000 GitHub stars and received strategic funding of 3M€ from the French Public Investment Bank and the European Commission. With this funding, we plan to enhance their Giskard platform, aiding companies in meeting upcoming AI regulations and standards. Moreover, we've upgraded our LLM scan feature to detect even more hidden vulnerabilities.\\ \\ View post](/content/knowledge/1-000-github-stars-3meu-and-new-llm-scan-feature/index.html) \\ **The Open-Source AI Imperative: Key Takeaways from Hugging Face CEO's Testimony to the US Congress** \\ \\ Explore key insights from ClĂ©ment Delangue's testimony to the US Congress on Open-Science and Open-Source AI. Understand the importance of Open-Source & Open-Science to democratize AI technology and promote ethical AI development that benefits all.\\ \\ View post](/content/knowledge/the-open-source-ai-imperative-key-takeaways-from-hugging-face-ceos-testimony-to-the-us-congress/index.html) \\ **Giskard’s new beta is out! ⭐ Scan your model to detect hidden vulnerabilities** \\ \\ Giskard's new beta release enables to quickly scan your AI model and detect vulnerabilities directly in your notebook. The new beta also includes simple one-line installation, automated test suite generation and execution, improved user experience for collaboration on testing dashboards, and a ready-made test catalog.\\ \\ View post](/content/knowledge/giskards-new-beta-is-out-scan-your-model-to-find-vulnerabilities/index.html) \\ **The EU AI Act: What can you expect from the upcoming European regulation of AI?** \\ \\ In light of the widespread and rapid adoption of ChatGPT and other Generative AI models, which have brought new risks, the EU Parliament has accelerated its agenda on AI. The vote that took place on May 11, 2023 represents a significant milestone in the path toward the adoption of the first comprehensive AI regulation.\\ \\ View post](/content/knowledge/the-eu-ai-act-what-can-you-expect/index.html) \\ **Exclusive Interview: How to eliminate risks of AI incidents in production** \\ \\ During this exclusive interview for BFM Business, Alex Combessie, our CEO and co-founder, spoke about the potential risks of AI for companies and society. As new AI technologies like ChatGPT emerge, concerns about the dangers of untested models have increased. Alex stresses the importance of Responsible AI, which involves identifying ethical biases and preventing errors. He also discusses the future of EU regulations and their potential impact on businesses.\\ \\ View post](/content/knowledge/how-to-eliminate-risks-of-ai-incidents-in-production/index.html) \\ **đŸ”„ The safest way to use ChatGPT... and other LLMs** \\ \\ With Giskard’s SafeGPT you can say goodbye to errors, biases & privacy issues in LLMs. Its features include an easy-to-use browser extension and a monitoring dashboard (for ChatGPT users), and a ready-made and extensible quality assurance platform for debugging any LLM (for LLM developers)\\ \\ View post](/content/knowledge/the-safest-way-to-use-chatgpt-and-other-llms/index.html) \\ **Giskard 1.4 is out! What's new in this version? ⭐** \\ \\ With Giskard’s new Slice feature, we introduce the possibility to identify business areas in which your AI models underperform. This will make it easier to debug performance biases or identify spurious correlations. We have also added an export/import feature to share your projects, as well as other minor improvements.\\ \\ View post](/content/knowledge/new-version-giskard-1-4/index.html) \\ **Giskard mentioned as a significant vendor in Gartner's Market Guide for AI Trust, Risk and Security Management** \\ \\ AI poses new trust, risk and security management requirements that conventional controls do not address. This Market Guide defines new capabilities that data and analytics leaders must have to ensure model reliability, trustworthiness and security, and presents representative vendors who implement these functions.\\ \\ View post](/content/knowledge/gartner-2023/index.html) \\ **FOSDEM 2023: Presentation on CI/CD for ML and How to test ML models?** \\ \\ In this talk, we explain why testing ML models is an important and difficult problem. Then we explain, using concrete examples, how Giskard helps ML Engineers deploy their AI systems into production safely by (1) designing fairness & robustness tests and (2) integrating them in a CI/CD pipeline.\\ \\ View post](/content/knowledge/presentation-at-fosdem-2023-how-to-test-ml-models/index.html) \\ **Exclusive interview: our first television appearance on AI risks & security** \\ \\ This interview of Jean-Marie John-Mathews, co-founder of Giskard, discusses the ethical & security concerns of AI. While AI is not a new thing, recent developments like chatGPT bring a leap in performance that require rethinking how AI has been built. We discuss all the fear and fantasy about AI, how it can pose biases and create industrial incidents. Jean-Marie suggests that protection of AI resides in tests and safeguards to ensure responsible AI.\\ \\ View post](/content/knowledge/exclusive-interview-our-first-television-appearance-on-ai-risks-regulations-and-quality-solutions/index.html) .jpg)\\ **Giskard closes its first financing round to expand Enterprise offering** \\ \\ The funding led by Elaia, with participation from Bessemer Venture Partners and notable angel investors, will accelerate the development of an enterprise-ready platform to help companies test, audit & ensure the quality of AI models.\\ \\ View post](/content/knowledge/news-fundraising-2022/index.html) .png)\\ **Giskard is coming to your notebook: Python meets Java via gRPC tunnel** \\ \\ With Giskard’s new External ML Worker feature, we introduce a gRPC tunnel to reverse the client-server communication so that data scientists can re-use an existing Python code environment for model execution by Giskard. \\ \\ View post](/content/knowledge/giskard-is-coming-to-your-notebook-python-java-via-grpc-proxy/index.html) \\ **Why do Citibeats & Altaroad Test AI Models? The Business Value of Test-Driven Data Science** \\ \\ Why do great Data Scientists & ML Engineers love writing tests? Two customer case studies on improving model robustness and ensuring AI Ethics.\\ \\ View post](/content/knowledge/why-test-ai-models-the-business-value-of-quality/index.html) \\ **Does User Experience Matter to ML Engineers? Giskard Latest Release** \\ \\ What are the preferences of ML Engineers in terms of UX? A summary of key learnings, and how we implemented them in Giskard's latest release.\\ \\ View post](/content/knowledge/does-user-experience-ux-matter-to-ml-engineers-giskard-latest-release/index.html) .jpeg)\\ **Why & how we decided to change Giskard's identity** \\ \\ We explain why Giskard changed its value proposition, and how we translated it to a new visual identity\\ \\ View post](/content/knowledge/giskard-log-4-changing-identity/index.html) .jpeg)\\ **Giskard's new feature: Automated Machine Learning Testing** \\ \\ The Open Beta of Giskard's AI Test feature: an automated way to test your ML models and ensure performance, robustness, and ethics\\ \\ View post](/content/knowledge/giskard-log-3-testing-ml-products/index.html) .jpeg)\\ **Who cares about AI Quality? Launching our AI Innovator community** \\ \\ The Giskard team explains the undergoing shift toward AI Quality, and how we launched the first community for AI Quality Innovators\\ \\ View post](/content/knowledge/giskard-log-2-spotting-the-innovators/index.html) \\ **Where do biases in ML come from? \#7 📚 Presentation** \\ \\ We explain presentation bias, a negative effect present in almost all ML systems with User Interfaces (UI)\\ \\ View post](/content/knowledge/where-do-biases-in-ml-come-from-7-n-presentation/index.html) \\ **Where do biases in ML come from? \#6 🐝 Emergent bias** \\ \\ Emergent biases result from the use of AI / ML across unanticipated contexts. It introduces risk when the context shifts.\\ \\ View post](/content/knowledge/where-do-biases-in-ai-ml-come-from-6-n-emergent-bias/index.html) .jpeg)\\ **Wishing y’all a happy & healthy 2022! 🎊** \\ \\ The Giskard team wishes you a happy 2022! Here is a summary of what we accomplished in 2021.\\ \\ View post](/content/knowledge/wishing-yall-a-happy-healthy-2022/index.html) \\ **Where do biases in ML come from? \#5 đŸ—Œ Structural bias** \\ \\ Social, political, economic, and post-colonial asymmetries introduce risk to AI / ML development\\ \\ View post](/content/knowledge/where-do-biases-in-ml-come-from-5-n-structural-bias/index.html) \\ **Where do biases in ML come from? \#4 📊 Selection** \\ \\ Selection bias happens when your data is not representative of the situation to analyze, introducing risk to AI / ML systems\\ \\ View post](/content/knowledge/where-do-biases-in-ml-come-from-4-n-selection/index.html) \\ **Where do biases in ML come from? \#3 📏 Measurement** \\ \\ Machine Learning systems are particularly sensitive to measurement bias. Calibrate your AI / ML models to avoid that risk.\\ \\ View post](/content/knowledge/where-do-biases-in-ml-come-from-3-n-measurement/index.html) \\ **Where do biases in ML come from? \#2 ❌ Exclusion** \\ \\ What happens when your AI / ML model is missing important variables? The risks of endogenous and exogenous exclusion bias.\\ \\ View post](/content/knowledge/where-do-biases-in-ml-come-from-2-n-exclusion/index.html) \\ **Where do biases in ML come from? \#1 👉 Introduction** \\ \\ Research Literature review: A Survey on Bias and Fairness in Machine Learning\\ \\ View post](/content/knowledge/where-do-biases-in-ml-come-from-1-n/index.html) \\ **8 reasons why you need Quality Testing for AI** \\ \\ Understand why Quality Assurance for AI is the need of the hour. Gain competitive advantage from your technological investments in ML systems.\\ \\ View post](/content/knowledge/why-quality-assurance-for-ai/index.html) Get AI security insights in your inbox ##### No vulnerabilities found?‹ We refund the assessment. **Limited-Time Offer:** available until the end of September. **Business email**\* Company name\* What specific AI use case or risk are you targeting?\* Demo Video V2 from Giskard AI on Vimeo Playing in picture-in-picture Play Settings Speed QualityAuto 2x 1.5x 1.25x 1x 0.75x 0.5x Picture-in-Picture Fullscreen Hit ▶ now for your full Hub demo! Full video in your email.