Solution · Private knowledge

Private AI over confidential documents

“We need AI over confidential documents, but cannot send them outside our infrastructure.”

Assess this workload Buying questions
Short answer. Yes, you can run a document assistant entirely inside approved infrastructure. It only stays inside if every component stays inside: document extraction, embeddings, the vector store, inference, authentication, logging and updates, not just the language model. Most failures come from retrieval and permissions, not the choice of model, so start with one bounded workflow, enforce existing access rights at retrieval time, and measure answer quality on your own questions before scaling.
Private document assistant inside an approved boundary Documents are extracted, embedded and stored in a vector store with access-control metadata. A signed-in user's question passes through a permission filter, fed by the identity provider, before reaching a local language model, which returns an answer with citations. Everything sits inside the approved boundary; outbound traffic to hosted APIs, telemetry and update services is blocked and tested. YOUR APPROVED BOUNDARY Ingest Query Documents SharePoint, shares Extraction parse and OCR Embeddings local model Vector store chunks + ACLs User question signed-in user Permission filter retrieve by access Local LLM approved hardware Answer with citations Identity provider user + groups Hosted APIs, telemetry, updates egress blocked and tested Private document assistant inside an approved boundary Documents flow through extraction, embeddings and a vector store. The user's question and identity feed a permission filter before a local language model returns an answer with citations. Outbound traffic to hosted services is blocked and tested. YOUR APPROVED BOUNDARY Documents SharePoint, shares Extraction parse and OCR Embeddings local model Vector store chunks + ACLs Permission filter retrieve by access Local LLM approved hardware Answer with citations Question from the user Identity user + groups Hosted APIs, telemetry, updates egress blocked and tested
Every component runs inside the boundary you approve. Permissions are enforced at retrieval time, and outbound traffic is blocked and tested, not assumed.

When this works

  • Every processing step has a local or approved alternative. OCR, embeddings and re-ranking are often overlooked.
  • Existing document permissions can be read and enforced at query time, not only at ingestion.
  • There is a named IT owner and a sponsor for one specific workflow.
  • You can assemble 30–100 representative questions with known good answers for testing.

The questions to settle before you buy

What leaves the network?

Nothing should, by design, but that has to be verified, not assumed. We map documents, prompts, embeddings, logs, outputs, telemetry, update checks and any optional cloud features, such as hosted models or web search in some runtimes, and test the deployed system with egress blocked.

How are existing access permissions respected?

Retrieval is filtered by the requesting user's identity and group memberships before anything reaches the model. Permission tests check that a user cannot surface content from documents they cannot open directly. This is a release gate, not a nice-to-have.

What happens when it cannot answer?

It should say so and show which sources it searched, rather than guess. We test refusal and "not found" behaviour explicitly, and require citations to source documents for answers that matter.

How is answer quality measured?

Against an agreed test set of your questions, scored for correctness, citation accuracy and completeness, with retrieval quality measured separately so problems can be traced to the right component.

Who maintains it?

Your team, us, or a partner, but it must be someone named, with a regression suite that runs before any model, runtime or index change.

Where processing happens

On-device

Processing happens on the user's own computer or phone. Nothing is shared between users unless you build that in.

On-premises

Processing happens on infrastructure your organisation owns and operates, in your building or your colocation space.

Private hosted or hybrid

Explicitly approved external infrastructure, or a defined mix, with every external flow named in writing.

How we evaluate it

  1. Map the data boundary and every external dependency
  2. Build an agreed question set with expected answers and permission cases
  3. Compare two or three candidate configurations (model, embeddings, chunking, runtime) on the same set
  4. Measure answer quality, permission leakage, time to first token, end-to-end latency and resource use
  5. Record versioned configurations so results can be reproduced

Every published result is labelled measured, reproduced, vendor-reported or estimated. Client data and results stay private unless the client approves publication.

When local is not the answer

Stay hosted, or go partly local, if:
  • Your approved hosted provider already meets your contractual and residency requirements, and the blocker is something else
  • The document corpus is small and rarely queried, so a hosted service with a suitable data-processing agreement costs less to run
  • You cannot staff an owner for updates and incidents, and no managed option fits

Ways to engage

Private AI Fit Assessment

Find out whether, and how, a specific AI workload can run within your constraints, before you buy hardware or sign an implementation contract.

From $1,500Typically 1–2 weeks

Workload Proof Pilot

A bounded implementation of one workload, tested against examples and acceptance criteria you agree up front.

From $8,000Typically 3–6 weeks

Production Deployment

A hardened deployment with integrations, access controls, operational documentation, training and acceptance testing.

From $20,000Typically 6–12 weeks

Dipankar Sarkar · Engagement owner, Neul Labs Limited
One named person is accountable for scope, commitments and acceptance on every engagement, including work delivered with specialist partners.

Further reading

Sources

Frequently Asked Questions

What leaves the network?

Nothing should, by design, but that has to be verified, not assumed. We map documents, prompts, embeddings, logs, outputs, telemetry, update checks and any optional cloud features, such as hosted models or web search in some runtimes, and test the deployed system with egress blocked.

How are existing access permissions respected?

Retrieval is filtered by the requesting user's identity and group memberships before anything reaches the model. Permission tests check that a user cannot surface content from documents they cannot open directly. This is a release gate, not a nice-to-have.

What happens when it cannot answer?

It should say so and show which sources it searched, rather than guess. We test refusal and "not found" behaviour explicitly, and require citations to source documents for answers that matter.

How is answer quality measured?

Against an agreed test set of your questions, scored for correctness, citation accuracy and completeness, with retrieval quality measured separately so problems can be traced to the right component.

Who maintains it?

Your team, us, or a partner, but it must be someone named, with a regression suite that runs before any model, runtime or index change.

Does running the model locally make us GDPR compliant?

No. Keeping processing in-house can remove a third-party transfer and simplify your position, but accountability, lawful basis, transparency, security and individual rights still apply. We document the technical controls; your advisers own the compliance opinion.

Can it run fully offline?

It can, provided every required component, including extraction, embeddings, retrieval, inference, authentication, updates and support access, is available inside the offline environment. We check each one.

How many users can one server support?

It depends on model size, context length, concurrency and latency targets, not staff count. Use the sizing calculator for a provisional range, then validate it under a measured load.