Blog - Published: Thursday, 2026-07-09 12:00 UTC
LLM threat modeling starts when the prototype starts.
IntelliOS began as a practical AI product: ingest cyber reports, organize source material, support grounded chat, and generate intelligence outputs that remain tied to evidence. That simple idea quickly turned into a broader question: what can go wrong when a language model sits between users, reports, databases, APIs, and future tools?
The answer is the heart of LLM threat modeling. The model is not just another library. It interprets natural language, absorbs retrieved context, produces persuasive text, and may eventually call tools. That makes it powerful, but it also creates new failure modes that traditional web application threat models do not fully cover.
LLM Threat Modeling
Map how prompts, retrieved context, model outputs, APIs, tools, users, and stored data can be abused, misread, or pushed outside the intended workflow.
API Boundary
Treat every route, provider call, and internal service handoff as a boundary where authentication, authorization, validation, rate limits, and logging matter.
RAG Grounding
Retrieve the right evidence before generation, keep source context separate from instructions, and require outputs to stay tied to allowed citations.
MCP Server Concepts
Understand MCP as a tool-interface pattern where an agent can call external systems, which raises least-privilege, approval, logging, and blast-radius questions.
State and Memory
Decide what an agent remembers, what it can write back, when state expires, and how to prevent stale, private, or irrelevant context from shaping decisions.
Human Oversight
Define which actions can run automatically and which require review, especially publishing, alerts, account changes, or tool calls with real-world impact.
How building IntelliOS created practical experience
The experience came from designing the system around real AI application risk. IntelliOS has browser users, app routes, server-side APIs, a Supabase data layer, report chunks, chat history, generated outputs, model provider calls, token usage controls, and a roadmap for more tool-connected workflows.
That meant thinking through the trust boundaries: user to application, application to API route, API route to database, retrieval layer to model, model output to user, and future agent to external tool. Each boundary is a place where security decisions have to be made before the product feels reliable.
AI-assisted development also shaped the learning. Building the prototype with an AI coding agent made specification, review, verification, and restraint part of the workflow. The system still needed human judgment: read the generated code, check the architecture, avoid leaking secrets, run tests, and decide where automation should stop.
Practical threat-model map
Assets
Reports, report chunks, generated intelligence, chat history, citations, admin workflows, API credentials, and token budgets.
Entry Points
User prompts, uploaded or retrieved documents, API requests, admin forms, model responses, and future tool calls.
Trust Boundaries
Browser, Next.js API routes, database, LLM provider APIs, retrieval context, public pages, and tool interfaces.
Control Goals
Keep secrets out, keep evidence grounded, prevent unauthorized actions, make costs visible, and preserve review points.
Specific threat examples and controls
Indirect Prompt Injection
Example: A report, article, or PDF could contain text that tries to instruct the model to ignore the user, change the answer, leak context, or invent a conclusion.
Control idea: Treat retrieved documents as data, not authority. Fence source text, rank trusted sources, keep instructions outside the document body, and validate whether claims are actually supported.
Citation Spoofing
Example: A model can produce a confident sentence with a citation-looking marker even when the retrieved source does not support that specific claim.
Control idea: Use citation validation, allowed citation labels, repair passes, and human review for high-impact outputs so the final answer is source-backed instead of merely fluent.
Source Poisoning
Example: A low-quality or adversarial source could exaggerate a threat, copy another source without evidence, or inject misleading technical details into the corpus.
Control idea: Apply source tiers, corroboration requirements, duplicate checks, recency checks, and explicit uncertainty language when evidence is thin.
Over-Permissioned Agents
Example: A future agent connected to email, a document system, a database, or an MCP server could take the wrong action if tool permissions are too broad.
Control idea: Use tool allowlists, scoped credentials, dry-run modes, approval gates, audit logs, and clear autonomy limits before agents can modify external systems.
Data Leakage
Example: Sensitive prompts, private notes, chat history, API keys, source credentials, or unpublished analysis could be accidentally sent to a model or exposed in output.
Control idea: Separate secrets from prompts, keep environment values out of generated content, restrict admin workflows, and design prompts so private data is not casually blended into public output.
Cost and Abuse Pressure
Example: Repeated large prompts, broad corpus retrieval, or runaway retry loops can consume tokens, create latency, and turn a useful AI feature into an operational problem.
Control idea: Estimate tokens before execution, track usage, set tier limits, cap retries, log model calls, and design graceful failure paths.
Why APIs and MCP matter
AI applications become more useful when they connect to systems of record. A REST API can retrieve a report, save a preference, log a usage event, or trigger a scan. An MCP server can expose tools and data to an agent through a more standardized interface.
That connectivity is also why threat modeling gets harder. Every new tool gives the agent a larger action space. A read-only source is different from a write-capable database tool. A draft-only email helper is different from an agent that can send messages. The design question is not just whether a model can call a tool. It is whether the tool should exist, what it can touch, how it is logged, and when a human approves.
Where model routing fits
Model choice is part of security and reliability. A cheaper model might be fine for classification. A stronger model may be needed for synthesis, ambiguity, or high-stakes analysis. A fallback model can protect uptime, but it can also change behavior if prompts, context windows, or safety defaults differ.
IntelliOS treats model routing as a design surface: choose the right provider for the task, keep token use visible, preserve output expectations, and avoid giving any model more context or authority than the workflow requires.
The operating principle
The model may summarize and synthesize, but it should not become the source of record.
The retrieval layer may select evidence, but ranking rules should be visible enough to audit.
An API may expose functionality, but permissions should be enforced before the model ever sees the task.
An agent may plan multi-step work, but high-impact actions need a human checkpoint.
A tool interface such as MCP may expand capability, but every new tool also expands the threat model.
The takeaway from the build
The biggest challenge in creating AI applications is not making a model respond. The hard part is designing the surrounding system so responses are useful, grounded, permissioned, observable, cost-aware, and reviewable.
IntelliOS turned LLM threat modeling from an abstract concept into product architecture: understand the data, define the boundaries, limit the tools, ground the output, log the decisions, and keep humans in the loop where consequences are real.