Guardrails, Not Gates: Governing Generative AI Without Killing Velocity

Guardrails for generative systems are usually discussed as content filtering. That is the smaller half. The larger half is architectural: what the model is permitted to do, what its output is trusted to mean, and which actions require a human. Systems that get the architecture right survive a jailbreak with an embarrassing transcript. Systems that get it wrong survive it with an incident.

Prompt injection is a structural property, not a bug to be patched

A language model receives one undifferentiated token stream. Your instructions and the user's data occupy the same channel, with no mechanism that marks one as privileged. This is not an implementation defect — it is how the architecture works, and it is why prompt injection has no clean solution at the model layer.

Indirect injection is the serious variant. The attacker never speaks to your system at all: they place instructions in content your system will later retrieve — a web page, a PDF, a support ticket, a calendar invite — and your own retrieval step delivers the payload into the context with full apparent legitimacy.

The controls that do work are outside the model. Treat every model output as untrusted input to whatever consumes it. Constrain what the model can reach, so a successful injection has nothing valuable to reach for. And require human confirmation for actions that cannot be undone.

Least privilege applied to tools

An agent with broad tool access is an attacker's remote execution primitive once injection succeeds. The exposure is not what the agent normally does — it is the full union of what every tool can do, invoked in any order the attacker can induce.

• Scope each tool to the minimum it needs. A ticket-reading tool reads the requesting user's tickets, not the ticket system.
• Carry the end user's identity through to the tool call. An agent acting with service-account privileges is a privilege escalation waiting for a prompt.
• Separate read from write. Retrieval and drafting need no write access at all; the write step is where confirmation belongs.
• Make irreversible actions — sending, paying, deleting, publishing, granting — require explicit human approval showing the exact action, not a summary of it.
• Rate-limit and budget-cap per session, so a loop induced by injection is bounded in blast radius and cost.

The design test is direct: assume the model has been fully compromised and is executing the attacker's instructions. What can it do? If the answer includes anything you would not accept, the guardrail belongs in the permission model, not the prompt.

Output is untrusted, and the vulnerabilities are the classic ones

Model output flowing unvalidated into a downstream system reintroduces every injection class the industry already knows. Generated SQL executed directly is SQL injection with extra steps. Generated markup rendered as HTML is cross-site scripting. Generated shell commands are command injection. Generated URLs fetched server-side are server-side request forgery.

Every existing control applies unchanged: parameterised queries, contextual output encoding, allow-lists, schema validation on structured output, and no path from generated text to an interpreter. None of this is new work — it is applying known practice to a source of input that people have not yet learned to distrust.

class Transfer(BaseModel):
    account_id: str = Field(pattern=r'^ACC[0-9]{10}$')
    amount: Decimal = Field(gt=0, le=10_000)
    currency: Literal['USD', 'EUR', 'GBP']

def handle(raw: str, user: User) -> Result:
    try:
        req = Transfer.model_validate_json(raw)      # shape
    except ValidationError as e:
        return Result.reject('malformed', e)

    if not user.owns(req.account_id):                # authority
        audit.flag('cross-account attempt', user, req)
        return Result.reject('unauthorised')

    return pending_confirmation(req)                 # human approves the action

Layered filtering, and the arithmetic that makes it worth doing

Independent controls multiply. Three layers each catching 90% of attempts leave one in a thousand, provided their failures are genuinely uncorrelated.

Independence is bought by using different mechanisms rather than different models. A deterministic policy check, a classifier, and a permission boundary fail for different reasons. Three variations on the same classifier fail together.

Grounding verification for anything factual

Where a system answers from retrieved documents, verifying that each claim is supported by the retrieved text is both a quality control and a safety control. The practical implementation decomposes the answer into atomic claims and checks each for entailment against the cited passage — an inexpensive natural language inference task.

Unsupported claims are then flagged, removed, or the answer regenerated with an explicit instruction to answer only from the provided context. The measurable outcome is that the system's refusals become trustworthy: a system that declines when the corpus lacks the answer is more useful than one that always answers, because its answers can be relied upon without checking.

Red-team continuously, and keep the transcripts

Jailbreak techniques evolve faster than release cycles. A guardrail suite validated at launch decays, and the decay is invisible without ongoing adversarial testing. Maintain an evaluation set of attack prompts, extend it with every new technique reported publicly and every real attempt observed in your own logs, and run it on every deployment as a gate rather than a report.

Log every input, retrieval, tool call and output with enough fidelity to reconstruct a session. When an incident occurs — and over a long enough horizon one will — the question will be exactly what the model was given and exactly what it did. Systems that cannot answer that question end up disabled while the investigation runs, which is usually a more expensive outcome than the incident itself.