Solution · Deployment rescue

Make a local AI prototype production-ready

“Our prototype worked in the demo, but it is slow, unreliable or unsafe with real users.”

Assess this workload Buying questions
Short answer. Most prototypes fail in production for predictable reasons: memory that grows with context length and parallel requests, a runtime built for single-user use, no permission checks in retrieval, and no tests to catch regressions. Fixing it starts with measuring the real load, then choosing serving, retrieval and access controls to match, rather than swapping models at random.
From prototype to production Before: a prototype serves one request at a time, so a queue builds up, memory sits near its limit and the first token is slow. After a measure, diagnose, fix and re-measure cycle: continuous batching serves concurrent requests, memory has headroom, and authentication, monitoring and regression tests are in place. BEFORE Prototype Model 1 at a time Queue grows under load Memory near limit Time to first token slow, rising 1. measure2. diagnose3. fix4. re-measure AFTER Production Serving runtime continuous batching Memory headroom ✓ Auth ✓ Monitoring ✓ Regression tests Named owner, runbooks, data boundary From prototype to production Before: one request at a time, a growing queue, memory near its limit and slow first tokens. After measuring, diagnosing, fixing and re-measuring: continuous batching, memory headroom, and authentication, monitoring and regression tests. BEFORE Prototype Queue grows under load 1 at a time Memory near limit Time to first token slow 1. measure2. diagnose3. fix4. re-measure AFTER Production Serving runtime continuous batching Memory headroom ✓ Auth ✓ Monitoring ✓ Regression tests
Qualitative picture, not a benchmark. Every rescue starts by measuring the real load, and ends by re-measuring on the same profile.

When this works

  • There is a working prototype and a defined workload
  • You can share configuration, logs and a description of real usage
  • Someone can approve changes and own the result afterwards

The questions to settle before you buy

Why is it fast for me and slow for everyone?

Concurrent requests share GPU memory and bandwidth. Context length and parallelism settings change memory needs sharply, and some runtimes queue requests by default. We measure before changing anything.

Do we need to switch runtime?

Sometimes. Desktop-first runtimes are excellent for development; high-concurrency serving may need a server-oriented runtime with continuous batching. The decision follows the measurements.

What makes it "production"?

Authentication, permission-aware retrieval, monitoring, a documented data boundary, a regression suite, runbooks and a named owner, as well as speed.

Where processing happens

On-device

Processing happens on the user's own computer or phone. Nothing is shared between users unless you build that in.

On-premises

Processing happens on infrastructure your organisation owns and operates, in your building or your colocation space.

Private hosted or hybrid

Explicitly approved external infrastructure, or a defined mix, with every external flow named in writing.

How we evaluate it

  1. Reproduce the failing behaviour under a defined load profile
  2. Profile memory (weights and KV cache), batching, retrieval and infrastructure
  3. Agree a remediation plan with an expected effect for each change
  4. Implement, re-measure on the same profile, and document

Every published result is labelled measured, reproduced, vendor-reported or estimated. Client data and results stay private unless the client approves publication.

When local is not the answer

Stay hosted, or go partly local, if:
  • The workload is bursty and rare enough that a managed hosted endpoint is cheaper and simpler
  • The required model quality is only available from hosted frontier models

Ways to engage

Performance Rescue

Diagnosis and agreed remediation of latency, memory, concurrency, reliability or cost problems in an existing deployment.

From $3,000Typically 1–3 weeks

Production Deployment

A hardened deployment with integrations, access controls, operational documentation, training and acceptance testing.

From $20,000Typically 6–12 weeks

Managed Private AI

Defined monitoring, maintenance, regression testing, incident handling and upgrade support for an accepted deployment.

From $750/monthRolling, with a minimum term agreed in the scope

Dipankar Sarkar · Engagement owner, Neul Labs Limited
One named person is accountable for scope, commitments and acceptance on every engagement, including work delivered with specialist partners.

Further reading

Sources

Frequently Asked Questions

Why is it fast for me and slow for everyone?

Concurrent requests share GPU memory and bandwidth. Context length and parallelism settings change memory needs sharply, and some runtimes queue requests by default. We measure before changing anything.

Do we need to switch runtime?

Sometimes. Desktop-first runtimes are excellent for development; high-concurrency serving may need a server-oriented runtime with continuous batching. The decision follows the measurements.

What makes it "production"?

Authentication, permission-aware retrieval, monitoring, a documented data boundary, a regression suite, runbooks and a named owner, as well as speed.

Can you guarantee a latency improvement?

Not before measuring. We commit to a measured baseline, a diagnosis and a plan with expected effects, then re-measure on the same load.