Secure LLM inference: enforce access control on confidential data

As organizations deploy large language models (LLMs) for code assistance, document analysis, financial forecasting, and customer intelligence, productivity is soaring. However, they also face a critical challenge: preventing models from leaking confidential information to unauthorized users.

Enterprises interact with LLMs in two key phases. First, during fine-tuning, which is the training process where a general-purpose LLM is specialized on an organization's proprietary data, such as internal documents, codebases, and financial records, so that the model learns domain-specific knowledge. Second, during inferencing, which is the runtime phase where users send prompts to the model and receive responses. It is at inference time, when users actively query the model, that access control matters most: the model has already absorbed confidential knowledge from fine-tuning, and now it must decide what to reveal and to whom.

When you fine-tune an LLM on confidential data, that knowledge embeds into model weights — the numerical parameters that encode everything the model learned. Once fine-tuned, the model doesn't just reference confidential data; it internalizes it. Today, LLMs don't distinguish between requesters. Every authenticated user gets identical treatment.

Today's strategies for handling confidential data in AI solutions are inadequate:

• AI lockdown, where enterprises ban all confidential data from LLMs, which is difficult to enforce and it is prone to violation in other deployments.
• Private silos with binary access, where enterprises deploy isolated LLM instances per department so that users either have full access to an LLM or no access to it, which has teams incurring high infrastructure and operational costs for managing multiple systems.
• Post-hoc filtering, where enterprises deploy guardrails that scan outputs for sensitive content, which can produce false negatives that could create compliance exposure and produce false positives that could destroy user experience.

Enterprises are forced to choose between AI productivity and data security. They shouldn't have to.

The problem isn't just about simple and plain access control, it is about context-aware intelligence, which is the ability of an AI system to tailor its responses based on who is asking, what they're authorized to know, and why they're asking, rather than treating every query identically. A prompt to the LLM about next quarter's product roadmap should yield radically different answers depending on the context of the requester, as in whether they are a partner, a contractor, or an executive:

• A partner should get a response about planned release dates and scope information that is limited at the tagline level.
• A contractor should ideally get a need-to-know response such as the code freeze timeline of the feature they are contributing to.
• An executive should get the detailed roadmap across all features and teams, along with a risk assessment and recommended mitigation steps.

Retrieval-based access control doesn't suffice when document-level permissions are too coarse and complex reasoning is required. What's needed is adapter-based access control.

Adapter-based control attaches lightweight, domain-specific model extensions (called adapters) to a single shared base model instead of deploying separate LLM instances per security domain. Access policies determine which adapters a given user can activate. This delivers fine-grained, per-query authorization without the cost of dedicated model silos. It becomes the architectural foundation of a secure inferencing framework, one where access control is enforced at the point of inference, not bolted on after the fact.

A secure inferencing framework: secure-inference

The llm-d project is an open-source Kubernetes-native distributed LLM inference framework for building production-grade infrastructure for enterprise AI deployments. It provides intelligent scheduling, cache-aware routing, disaggregated serving, and scalable multi-node inferencing by supporting a large number of adapters.

The secure-inference project integrates with llm-d and addresses critical security and access control requirements for organizations running AI at scale. It uses LoRA adapters (low-rank adaptation adapters) for each domain with policy-enforced routing.

How adapter-based secure inference works

The secure-inference architecture has two layers or phases:

• Training phase: Partition sensitive data by security domain. Create multiple, small LoRA adapters for each domain, such as finance, HR, product, or engineering. The sensitive data is stored in an access-controlled registry. Each adapter loads quickly and contains only its security domain's knowledge.

• Inference phase: User requests are authenticated using JSON Web Token (JWT) or single sign-on (SSO) and then authorized (at the gateway) based on user attributes (role, department, clearance level) against policy engine (Open Policy Agent). The appropriate adapter is selected based on a semantic analysis of the user's query and the request is sent to the inference pool with the correct adapter loaded. A response is generated from the appropriate data. Users without authorization receive responses from base model only (general knowledge), never from confidential adapters.

A key feature of this architecture is that authorization happens at the gateway before the request reaches the model. Unauthorized users cannot access restricted adapters under any circumstances.

Example: Per-role inference routing

Here is an example flow for a software engineer:

1. User prompt: "Show me database optimization code for the customer service platform"
2. Gateway authenticates using JWT (role: engineer, dept: platform, clearance: internal)
3. Policy Check: Does user have access to Engineering LoRA? (YES)
4. Semantic Matching: Best adapter is Engineering LoRA (similarity: 0.91)
5. Route to inference pool with Engineering LoRA loaded
6. Response sent: Detailed code snippets from proprietary codebase with architecture context.

A contractor with the same query would be denied access to Engineering LoRA and receive only general programming advice from the base model.

Ready to get started?

Start with the quickstart guide from the secure-inference GitHub Repository.

With the Apache 2.0 license (open-source, production-ready), explore the repo and star it to follow development progress or join the community to contribute.

To try out secure-inference for yourself, see the "Enable secure LLM inferencing over protected enterprise content with access-controlled LoRA adapters".

The bottom line

The enterprise AI security paradox is solvable. You don't have to choose between AI productivity and data security, innovation velocity and compliance, or centralized efficiency and departmental autonomy. The secure-inference project lets you have all three by making access control a first-class concern at inference time, not an afterthought. The question isn't whether your organization will adopt confidential AI. The question is whether you'll do it securely.