Skip to main content
Kubernetes Your app points OPENAI_BASE_URL at one of three endpoints: your Mac, your cluster, or managed Apprentice does not run a cluster for you and never asks for access to yours. Instead, the agent skill picks up after training and writes a standard vLLM Deployment + Service into your own repo, mirroring your existing manifests: your registry, your ingress pattern, your naming. You review and apply it like any other change.

​
Hand it to your agent

The fastest path: install the skill and let your coding agent write the manifests against your real repo.
Then paste:
The skill mirrors your registry, ingress pattern, and naming. You review the diff and apply it like any other change; nothing is applied for you. The sections below are the review knowledge: what lands in the cluster and which knobs matter when you read that diff.

​
What gets deployed

One pod running the vLLM OpenAI-compatible server with the fine-tuned model, exposed by a ClusterIP service on port 8000. Your app pods stay untouched:
Traffic never leaves the cluster, and a model-pod crash never takes your API down. Rollback is flipping the env var back.

​
Sizing, honestly

The precondition the skill will surface: at least one GPU node in the cluster. If you have none, requesting that node group is the only step that is not YAML.

​
The knobs that matter

On the vllm serve command line:
  • --gpu-memory-utilization: how much of the GPU vLLM claims for weights + KV cache (default 0.9). Lower it if the pod shares the GPU.
  • --max-model-len: cap the context to what your task actually needs; shorter contexts free memory for more concurrent requests.
In the manifest, three things the official vLLM guide includes that are easy to forget:
  • GPU requests and limits: nvidia.com/gpu: "1" in both.
  • Readiness and liveness probes with a generous initial delay (60s+): model load takes a while, and a probe that fires too early crash-loops a healthy pod.
  • A PersistentVolumeClaim for the Hugging Face cache (mounted at /root/.cache/huggingface) so the pod does not re-download weights on every restart, plus a shared-memory volume (emptyDir with medium: Memory).

​
Privacy note

This path exists for exactly one reason: inference requests carry your production data, and in this setup they never cross your network boundary. The only thing that came from outside is the model weights.