A locally operated AI system is more than a model on a GPU. It needs an inference runtime, a secured API, identity and permissions, optionally a knowledge layer, monitoring, and a defined update path. I assemble those layers on your own hardware, a suitable appliance such as NVIDIA DGX Spark, or a dedicated server.
Your team can use the system through a browser interface similar to ChatGPT. Internal applications can connect through an OpenAI-compatible endpoint, keeping business logic from becoming tightly coupled to one runtime.
Technical architecture
- Model & inference: open-weight models are selected for the task, language quality, licence, context length, and available memory — not a generic leaderboard.
- Serving layer: Ollama fits smaller installations and straightforward model management. vLLM is an option for concurrent users, continuous batching, and a capable OpenAI-compatible API endpoint. I consider TensorRT-LLM where NVIDIA-specific optimisation justifies the extra operational complexity.
- Access: Open WebUI or an existing internal frontend with personal login, roles, and separated workspaces.
- Knowledge: when needed, a RAG pipeline supplies approved documents; Qdrant or Postgres with pgvector handles vector retrieval.
- Operations: containers, network segmentation, TLS, secrets, logs, metrics, backups, and a documented model and update process.
Ollama or vLLM?
- Ollama is often the pragmatic start for a small team, low concurrency, and uncomplicated model changes.
- vLLM becomes relevant when several users or applications connect at once, throughput and GPU utilisation matter, or a stable API service is required.
- Hosted APIs remain sensible when a frontier model, infrequent use, or a fast start is economically better. Local and cloud models can be combined behind a routing layer.
I decide after a short load test with your real inputs. Model size alone does not answer quality, latency, or total cost.
What is measured before release
- quality on a representative set of your tasks
- time to first token, generation speed, and behaviour under concurrent requests
- GPU and system memory use at a realistic context length
- failures, timeouts, and fallback to another model or a manual process
- useful logging without collecting unnecessary confidential prompt content
What’s included
- compare hardware and operating models, optionally before procurement
- configure and harden the base system, GPU drivers, and container runtime
- tune the model, quantisation, context limits, and inference parameters against tests
- set up the browser interface and OpenAI-compatible API access with roles
- optionally integrate RAG, company login, model routing, or external API fallbacks
- document monitoring, backup, update, and recovery procedures
- onboard administrators and end users
Realistic limits
Local does not automatically mean no data ever leaves the company network. Updates, telemetry, external APIs, integrations, and remote support are therefore reviewed and restricted explicitly. A local model is also not a blanket replacement for the most capable hosted models. I compare local and hosted options on the same tasks and recommend the architecture that fits technically and economically.
Hardware, electricity, maintenance, and support remain operating costs. An appliance is justified not by the label, but when data class, utilisation, latency, availability, or integration needs support the case.