Private GPU Inference on Kubernetes: A Shared-GPU Service for Document Automation

October 2026 • White paper

Executive summary

A single shared NVIDIA V100 on CloudAcropolis Kubernetes now serves a private, API-key protected language model that turns invoice PDFs into structured accounting data, without any document leaving the customer's tenant.

We deployed a document-extraction model (NuExtract3) as a two-replica llama-server deployment on shared GPU slices. Each replica gets a fixed share of the card, enforced by the platform. Workloads inside and outside the project call it through an OpenAI-compatible endpoint.

The first consumer is the CloudAcropolis accounting bundle. Invoice PDFs uploaded in a tenant portal are read by n8n, extracted by the model, checked by gatekeeper workflows and booked in Odoo with the original PDF attached.

This paper describes the platform pattern, the service, the security model, the production workflow, and the problems we hit and solved along the way.

1. Why a private GPU service

Invoices carry supplier names, VAT numbers and amounts, so sending them to a public AI API is a data-protection decision, not only a technical one. Running the model inside the customer's own Kubernetes project keeps documents in the tenant and in the datacenter that already holds the accounting data.

A whole GPU per workload is rarely justified for extraction. Kube-DC lets a project use a defined fraction of a card, so a Pro Pool + GPU package includes a 50% share of a Tesla V100 (32 GB) that can be split between replicas. That is enough for a compact structured-extraction model to serve a tenant continuously.

The job itself suits a small, specialised model: read the text of one document, return one JSON object that follows a given template. It needs predictable output and low cost per document more than open-ended conversation.

2. Architecture overview

Four parts make up the service, and only extracted text crosses to the model.

Documents stay in the tenant; only extracted text reaches the GPU service.

The portal and n8n run on the customer's accounting server. n8n sends text, a template and the bearer key to the two model replicas in the Kube-DC project, receives JSON back, and books the result in the tenant's Odoo.

3. Shared GPU on Kubernetes

A GPU pod on Kube-DC claims its slice through the Kubernetes Dynamic Resource Allocation (DRA) API, and the platform enforces the slice size on the card. In our tests the container saw one Tesla V100-PCIE-32GB, and the HAMi-core log confirmed a limit of 25% compute and 8 GiB of memory per slice.

Kube-DC GPU nodes run in one of three exclusive modes: pod-hami (shared, container level), vm-passthrough and vm-vgpu. This service uses the shared container mode. Platform GPU support is currently in Pilot status on the public roadmap.

A workload is accepted only if it follows the platform's convention:

A ResourceClaimTemplate that references the project's device class (here kube-dc-nvidia-v100-shared-8g, visible in the project's resource quota).
The labels and annotations kube-dc.com/gpu-backend, kube-dc.com/gpu-profile, kube-dc.com/gpu-workload-hash, kube-dc.com/gpu-memory-mib and kube-dc.com/gpu-core-percent.
Native nvidia.com/gpu requests are rejected by admission policy; that is intended, not a fault.

CPU and memory quota is pooled across everything in the project. A managed cluster and a VM together used about 7.5 of 8 vCPU in one test, and the GPU pod could not start until capacity was freed. Plan the CPU budget before adding GPU workloads.

4. Model serving

The model runs as a Kubernetes Deployment named llama-server with two replicas, each holding one GPU slice. Clients call a standard chat-completions endpoint (/v1/chat/completions) and select the model nuextract3, so any OpenAI-compatible client or HTTP node works unchanged.

NuExtract3 is built for template-driven extraction. The request carries the document text as the user message, and the extraction is controlled through chat_template_kwargs:

template: a JSON skeleton whose values name the type to extract, such as verbatim-string, date, number, or a list of allowed values for classification.
instructions: plain-language rules, for example how to tell a sales invoice from a vendor bill, or never to guess a VAT rate that is not printed on the document.
enable_thinking: false, with temperature 0.2 and max_tokens 4000, for short and repeatable answers.

The response is a JSON object in the shape of the template. Because the model can occasionally return malformed numbers (for example 1.183.08 for 1,183.08), the calling workflow repairs thousands separators, validates the JSON, and calls the model again, up to three attempts in total, before flagging the document for a person.

5. Access and security

Every request must carry a bearer key. The key lives in a Kubernetes Secret in the project and is read only by the people and workflows that need it; it is entered at the point of use and never stored in documents, chat or source files.

The service is exposed through a LoadBalancer on an address in the cloud's internal range, not on a dedicated public IPv4. Consumers are workflows on servers inside the same datacenter. A public address can be added deliberately, as we did for a separate external test, but it is not the default.

Tenant isolation comes from the platform. A tenant role cannot list nodes or namespaces at cluster scope and cannot exec into pods, so troubleshooting is done with one-shot Jobs, logs and the REST API. This limits what a compromised tenant account can reach.

Good practice we apply or recommend:

One key per environment, rotated on a schedule and after any exposure.
Quota and replica limits set on the deployment, so one consumer cannot exhaust the GPU slices.
Documents are sent as extracted text; the original file stays in the customer's own system.

6. Use case: invoice extraction for accounting

The accounting bundle gives an accountant ten independent Odoo tenants per server, each with its own upload portal, plus one n8n instance per server. A PDF dropped in the portal reaches Odoo as a booked invoice or vendor bill in five steps:

The portal posts the file to the server's n8n webhook, protected by a shared secret.
n8n extracts the PDF text and asks the tenant's Odoo for its company VAT and chart of accounts.
The text, a JSON template and the instructions go to the GPU model, which classifies the document as a vendor bill or a sales invoice and extracts supplier or customer, number, date, currency, lines and totals.
A gatekeeper workflow checks the result against Odoo and rejects documents with missing fields or duplicates.
Odoo creates the invoice or bill, and the original PDF is attached to it.

Three behaviours made the pipeline dependable in daily use:

Retries. The portal retries every failed upload up to three times. A document that is already booked is not retried, but it is still listed in the failed-files box with the reason.
Duplicate protection. A request id makes a repeated call idempotent, and the duplicate check no longer depends on the total, because the model reads totals inconsistently. A replay is reported to the user as “Already booked as invoice N”.
Attachments. n8n stores binary data on disk, so the workflow reads the real bytes through the n8n helper before sending them to Odoo. Without this the invoice was booked but the PDF was missing.

On a new server the provisioning script writes that server's own n8n and Odoo addresses into every portal and into n8n itself, so each server reads from its own workflow engine.

7. Lessons learned

Most problems came from sizing and defaults, not from the model. These are the ones worth knowing before you build the same thing.

Symptom	Cause	Fix
The model server ignored the GPU	The latest Ollama image dropped CUDA compute capability 7.0 (V100)	Pin an older, V100-compatible image tag
Pod killed with exit 137 on first model load	The pod memory limit (512 Mi) was too small for host-side overhead; VRAM was fine	Raise the container memory request and limit, independent of the GPU slice
Garbled or malformed agent answers	A default context of 4,096 tokens silently truncated a prompt of about 14,700 tokens	Set the context length to 16,384 and check the server log for truncation
Pod would not schedule	CPU quota is pooled across clusters, VMs and containers	Budget CPU for the GPU pod first, or free capacity
Package builds hung inside a VM	Docker MTU 1500 against a project network MTU of 1400	Set the Docker daemon MTU to 1400
An internal VLAN could not reach the cloud's public GPU address	No NAT hairpin path to the routed public subnet	Add firewall policies between that VLAN and the public subnet, without translation
A `LoadBalancer` did not get the dedicated public IP	The default service binds a shared internal address	Create a dedicated `EIp` and bind the service to it with an annotation

Two habits paid off: check the server log for truncation before blaming the model, and keep one-shot Jobs and logs ready, since exec is not available to a tenant.

8. Results and limits

The service works end to end, from a shared GPU slice to a booked invoice with its PDF attached. We verified the provisioning of the GPU package, shared-slice scheduling and enforcement, persistent storage, a dedicated public IPv4, S3-compatible storage, nested clusters with real autoscaling under load, and GPU-backed inference used by consumers both inside and outside the project.

What we have not yet measured, and do not claim:

Throughput and latency per document under concurrent load.
Extraction accuracy on a labelled sample of real invoices.
Behaviour of the service when both replicas are busy.

Known limits:

GPU support on Kube-DC is in Pilot status, so interfaces and policies may change.
A slice is fixed in size (8 GiB of memory in our profile), which limits model size and context length.
Invoices without a supplier VAT number are protected from duplicates by the request id only.
Pulled models and caches should live on a persistent volume, not on emptyDir, or a pod restart wipes them.

9. Getting started

You can run the same pattern on the Kube-DC Pro Pool + GPU package: 8 vCPU, 24 GB RAM, 160 GB NVMe, 100 GB object storage, one dedicated IPv4, a 50% shared GPU and nested Kubernetes clusters through KubeVirt.

Order the package in the CloudAcropolis storefront; the quota appears in the customer portal when the order is active.
Create a project in the portal. The package grants the quota, and the project itself is a manual step.
Open the project console, a browser terminal with kubectl already configured.
Deploy your model with the claim and labels from section 3, set a bearer key in a Secret, and expose it with a LoadBalancer.
Point your workflow engine at the endpoint and start with a small template.

If you want the accounting bundle that uses this service, visit FiscFort. For help sizing a GPU workload, contact CloudAcropolis.

See the platform

The Kube-DC Pro Pool + GPU package runs on CloudAcropolis Kubernetes.

Explore CloudAcropolis Kubernetes

Private GPU Inference on Kubernetes