Serving LLMs on-premise: GPU, quantization, and throughput
A technical guide to sizing GPU and memory, applying quantization, continuous batching, and KV cache when serving language models inside the company.
8 publications
A technical guide to sizing GPU and memory, applying quantization, continuous batching, and KV cache when serving language models inside the company.
Lexical plus vector search with result fusion, source-inherited ACL applied at query time, revocation, multi-tenancy, and automated leakage tests.
Structure-aware chunking, embedding versioning, reindexing, reranking, and retrieval evaluation — the technical decisions that determine RAG quality.
Answer quality is a data and permission problem, not a model problem. What changes when the index must respect who can see what.
How to build a regression dataset from real cases, pick metrics that matter, and use an LLM-as-judge calibrated against human review to block bad deploys in CI.
Prompts and system instructions treated with the same discipline as code: repository, PR review, automated tests, canary rollout, and rollback.
What actually replaces the per-turn prompt in agentic systems — and where the risk goes when no one reviews the wording anymore.
Model gateway, orchestration, retrieval, memory, identity, observability, and audit trail: the technical design behind an enterprise AI that isn't locked to a single vendor.
Put it to work
Everything we cover here — AI governance, legacy system integration, audit trails and preserved corporate knowledge — is available on the e.works platform at eworks.cloud.
e.works infrastructure on AWS
A managed environment protected by e.works on AWS, with encryption, per-company isolation, backup and high availability.
On-premises, in your environment
The same platform running in your company's data center or private cloud, when data sovereignty requires that nothing leaves your perimeter.
In either model your data stays yours — with access control, audit logging, configurable retention and guaranteed availability.
Newsletter
Analysis on automation, industrial data and technology adoption. No spam.