Cruxly › AI engineering › Reliability, Security & Governance

Red-Teaming and Compliance as Code

the OWASP LLM Top 10, adversarial testing, and DPDP erasure that actually erases

Prompt: design the security and compliance layer for a production AI system handling personal data: an adversarial testing suite covering the OWASP LLM Top 10, run in CI on every change, and a data-protection implementation where a user's right-to-erasure request is a code path, not a policy promise.

40 minutes first. "We reviewed it and it seemed safe" is not a security program. Neither is "we have a privacy policy" a compliance program.

The OWASP LLM Top 10, and what a standard pentest misses

Use the current list and its current names — quoting an old edition is an easy tell. The current one is the 2026 edition, published on 4 August 2026 (the project's GenAI-LLM-Top10 repository, 2026/final, is the canonical text). Prompt injection is still the headline item, but a traditional web pentest checks for almost none of the other nine:

:::key Verified October 2026 The ten entries and their order match the 2026/final folder of the OWASP GenAI-LLM-Top10 repository on GitHub, read 31 October 2026 (README: published, released 4 August 2026). The 2025 column is the previous edition's numbering. Check genai.owasp.org before citing, since later revisions are possible. :::

What the reshuffle tells you. Eight of the ten moved, and the 2026 ranking weighted practitioner votes at 75% and a record of 6,639 classified incidents at 25%, so real incident data moved some entries. Two moves are worth saying out loud:

• Excessive Agency jumped from 6th to 3rd because agentic deployments made it concrete — once a model can call tools that act, too much permission is no longer theoretical. That's the guardrails lesson in Agentic Systems.
• System Prompt Leakage became Hidden Context Exposure and got broader. The risk was never just the system prompt: it's every piece of context the user doesn't see, which in a RAG or agent system is most of the input.

Positions changing doesn't mean a risk got less dangerous — Improper Output Handling dropping to 10th still means "never execute or render model output unsanitised."

OWASP has also published a separate Top 10 for agentic applications — worth naming for any system where the model calls tools.

Poisoning is cheaper than intuition says. A 2025 study (Anthropic with the UK AI Security Institute and the Alan Turing Institute) found that a small, roughly constant number of poisoned documents — on the order of a few hundred — could implant a backdoor regardless of model size. "Our dataset is huge, a few bad documents won't matter" is not a defence.

Run a regression suite covering every category on every change that touches the model, prompt, or tool layer — the same way a traditional security suite runs on every PR. A hand-written set of a few dozen prompts is a seed, not a bar:

• Use automated red-teaming to scale it. Tools such as garak, PyRIT, or Promptfoo's attack generators produce hundreds of variants per category and mutate them, which a fixed list can't.
• Keep the hand-written cases too — the specific attacks found in your own incidents and reviews become permanent regression tests.

Report an attack success rate with its sample size, not a pass/fail. LLM outputs vary, so a vulnerability caught 8 times out of 10 runs is a real vulnerability, not a flaky test to retry until it passes. And "absent across N runs" bounds the rate rather than proving zero: by the rule of three, 0 successes in N tries means the true rate is plausibly up to about 3/N — so 0 in 30 still allows roughly a 10% success rate. Pick N from the risk you can tolerate.

DPDP as code: consent, erasure, and minimisation that actually run

Consent needs to be a structured, queryable record attached to every piece of data collected, not an assumption baked into a signup flow. "The user consented when they signed up" is not sufficient when different processing purposes (analytics, model training, third-party sharing) may each require distinct, revocable consent — the system needs to be able to answer "did this specific user consent to this specific use of this specific data" as a database query, not a policy interpretation.

Right-to-erasure has to reach every store the data touches, including the ones that are easy to forget. A user's data typically lives in a primary database, but for an AI system it may also live in: a vector store (as embeddings derived from their documents or conversations), a cache, a fine-tuning dataset, and application logs. An erasure implementation that deletes from the primary database but forgets the vector store has satisfied the letter of a checklist while completely failing the actual legal requirement, since the embeddings still encode information derived from that person's data.

The store nobody can delete from: a model already trained on the data. Removing a user's rows from the fine-tuning dataset doesn't remove what the weights learned from them, and reliable machine unlearning doesn't exist yet for production models. So the policy has to be decided up front:

• Keep personal data out of fine-tuning sets wherever the task allows — redact or pseudonymise before training.
• Where it can't be avoided, set a retrain cadence that erasure requests feed into, and say so in the retention policy.
• Per-tenant LoRA adapters make this tractable — retraining one tenant's small adapter is far cheaper than retraining a shared model.

Know the actual compliance dates. India's DPDP Rules were notified in mid-November 2025 (sources give 13 or 14 November) with an 18-month phased rollout: the Data Protection Board provisions (Rules 1, 2 and 17 to 21) applied immediately, Consent Manager registration (Rule 4) starts on 13 November 2026, and the substantive obligations (Rules 3, 5 to 16, 22 and 23) apply from 13 May 2027. "DPDP as code" should track those dates. And DPDP's "legitimate uses" (section 7) mean not every processing purpose needs consent; the consent record should note which basis applies to each use.

:::key Verified October 2026 Phase dates as summarised by law-firm and EY write-ups of the Rules, read 31 October 2026: Rules 1, 2 and 17 to 21 immediately; Rule 4 (Consent Managers) on 13 November 2026; Rules 3, 5 to 16, 22 and 23 on 13 May 2027. Sources differ on whether notification was 13 or 14 November 2025, so quote the phase dates and read the Gazette text before relying on them. :::

Data minimisation means designing collection to happen narrowly from the start, not filtering broadly-collected data down later. Collecting everything "in case it's useful" and then relying on access controls to limit exposure is a weaker posture than never collecting what a specific feature doesn't need in the first place — every field collected is a field that has to be secured, minimized, and eventually erased correctly, so the cheapest compliance is not collecting it.

Incident response for an AI-specific incident

A rollback capability needs to cover prompt versions and model versions with the same rigor as code versions. If a prompt change caused a quality or safety regression, "roll back to the previous prompt version" needs to be as fast and as reliable an operation as rolling back a code deploy — which requires versioning prompts as a first-class artifact (with the same review and deployment discipline as code) rather than as strings embedded in application code that happen to change when the code around them changes.

A post-mortem for an AI incident needs to answer a different root-cause question than a traditional outage post-mortem. "The service returned 500s because the database connection pool was exhausted" has a clear, deterministic root cause. "The model started producing inappropriate responses to a specific query pattern" requires investigating whether the cause was a prompt change, a retrieval change surfacing different context, a provider-side model update you didn't initiate, or genuine variance — and the post-mortem process needs to be built to investigate all four possibilities systematically, not stop at the first plausible explanation. The provider-update cause is largely preventable: pin a dated model snapshot instead of a floating alias, so the model only changes when you change it.

Runbook testing for AI incidents needs to be exercised specifically, not assumed to generalize from traditional runbooks. "If the model starts hallucinating at an elevated rate, here's the immediate mitigation (route to a more conservative fallback, tighten the grounding instruction, or disable the affected feature)" needs to be a tested, rehearsed runbook — not a plan that's only ever been discussed hypothetically and gets improvised for the first time during a real incident.

Where compliance and security overlap, and where they pull in different directions

Detailed audit logging (for compliance and incident investigation) and data minimisation (for privacy compliance) are in genuine tension, and resolving it needs an explicit policy, not an accident of what happened to get logged. Logging full request content supports investigation and accountability; logging less supports minimisation and reduces breach exposure. The resolution is usually structured logging with different retention periods and access levels per field — not a single blanket decision to log everything or log nothing.

A security fix and a compliance fix can require different testing before shipping. A prompt-injection patch needs adversarial testing to confirm it closes the specific vulnerability; a consent-tracking change needs testing against real regulatory scenarios (a user requesting erasure mid- processing, a user revoking one consent type while others remain active) that a standard test suite wouldn't naturally cover, and building the compliance test suite requires the same deliberate design effort as the security one, not an assumption that passing normal tests implies compliance.

The honest limitation worth naming to any stakeholder: no red-team suite, however thorough, proves the absence of a vulnerability — it proves the absence of the specific vulnerabilities tested for. Continuous, evolving adversarial testing that incorporates new attack patterns as they're discovered in the field is the realistic goal, not a one-time "secured" checkbox that stays true indefinitely.

The regulatory and frontier-safety landscape you're expected to know

Data protection (DPDP, GDPR) is only one layer. AI-specific regulation and industry safety frameworks now shape what a production system has to do, and a 2026 interview at an AI company will check you know the shape of them.

The EU AI Act — risk-based, with staggered dates. It applies to anyone placing AI systems on the EU market or whose outputs are used in the EU, regardless of where the company sits.

• The high-risk dates did move. The EU's "Digital Omnibus" (Regulation (EU) 2026/1744) got final Council approval in June 2026 and has been in force since 27 July 2026. It deferred the high-risk obligations above, but left the GPAI duties, the prohibitions and Article 50 on their original timeline. The mistake to avoid in an interview is the reverse of the old one: "the AI Act was delayed" is wrong as a blanket statement — only the high-risk regime was.
• Article 50 is the one that lands on ordinary products. Users must be told when they're interacting with an AI system (unless it's obvious), and AI-generated content must be marked in a machine-readable way. A support chatbot's UI needs that disclosure.
• Penalties are serious — up to €35M or 7% of global annual turnover for prohibited practices.

Voluntary frameworks that buyers and auditors ask about:

• NIST AI Risk Management Framework — Govern, Map, Measure, Manage — with a generative-AI profile (NIST AI 600-1).
• ISO/IEC 42001 — a certifiable AI management system standard, the AI counterpart to ISO 27001; enterprise customers increasingly ask for it.

How frontier labs govern their own models — worth knowing if you're interviewing at one:

• Capability-threshold frameworks: Anthropic's Responsible Scaling Policy (AI Safety Levels), OpenAI's Preparedness Framework, Google DeepMind's Frontier Safety Framework. Each defines dangerous-capability thresholds and the safeguards required before training or deploying past them.
• System cards published with model releases, documenting evaluations, red-teaming and known limitations.
• Third-party pre-deployment testing by government AI safety institutes and external red-teamers.

The engineering takeaway: regulation increasingly asks for evidence — evaluations, logs, incident records, documentation — which is exactly what the observability, red-teaming and compliance-as-code practices in this module produce.

How to say it in the interview

"A privacy policy is a promise about what happens. Compliance as code is a guarantee, because the guarantee is enforced by a code path that actually runs - if erasure is a manual process someone has to remember to do across every data store including the vector database, it will eventually be missed, and 'eventually' is not acceptable for a legal right."

Checklist

• Built adversarial tests for all ten OWASP LLM categories (2026 edition), by their current names
• Ran the adversarial suite in CI on every relevant change, not only pre-launch
• Reported attack success rate with its sample size, not a single pass/fail run
• Knew the EU AI Act's staged dates and that Article 50 requires disclosing AI interaction
• Named NIST AI RMF, ISO/IEC 42001, and frontier-lab capability-threshold frameworks
• Modeled consent as a structured, queryable record per data use, not an assumed blanket signup consent
• Extended right-to-erasure to the vector store, caches, and logs, not just the primary database
• Designed data minimisation into collection, not as a filter applied after broad collection
• Versioned prompts with the same rigor and rollback capability as code
• Investigated AI incidents across prompt, retrieval, provider-update, and variance causes systematically
• Rehearsed AI-specific incident runbooks rather than improvising during a real incident

Related lessons

• Safety and Alignment Basics for Engineers
• Observability and Cost Governance for LLM Systems