Part of Index of Inference Engineering. Book: Ch 7.5 (Client Code). Your unfair advantage: Java+K8s gateway in front of Python inference core. Python OR Go postings accept this with proven vLLM.
Architecture
Client → Spring Boot gateway (auth, rate-limit, validation, Kafka/Redis queue) → FastAPI/vLLM OpenAI-compat (AsyncLLMEngine) → stream back → Prometheus metrics → Grafana
Java owns: JWT/API keys, per-tenant budgets, request validation (GuidedDecodingParams JSON-schema passthrough), queueing (Kafka/SQS), retries/fallback (LiteLLM), audit logs, $/MTok accounting. Python owns: token loop only.
Client code matters too (Book §7.5)
- Client latency overhead: the network hop + client-side parsing/rendering can dominate perceived latency, especially for small responses. Measure end-to-end, not just server TTFT/ITL.
- Streaming + protocols: SSE/WebSocket streaming hides total generation time behind first-token latency; choose the protocol per client (browser SSE vs gRPC service-to-service).
- Asynchronous inference: for long jobs, accept → queue → return
job_id→ poll/webhook. Fits batch/offline work (see 00b-Prerequisites-Product-Model-Selection) and pairs with the Kafka path above.
Minimal Python glue to learn (nothing else)
FastAPI proxy + transformers.AutoTokenizer (count/truncate) + prometheus_client (queue, TTFT) + openai client for tests. No training, no custom kernels.
Spring sketch
POST /v1/chat→ validate → enqueue withrequestId + model + budget→ worker calls vLLM/v1/chat/completionsstreaming → SSE back.- Filters:
RateLimitFilter (Bucket4j/Redis),AuthFilter (JWT),BudgetFilter (tokens × $/MTok per model). - K8s: gateway Deployment (CPU, HPA on RPS/latency) separate from vLLM Deployment (GPU, KEDA on queue). Never co-schedule.
- Config:
VLLM_URL,HUGGING_FACE_HUB_TOKEN,VLLM_API_KEYall Secrets; model allowlist in ConfigMap.
Do
P3 gateway (see 11-Projects-P1-P2-P3): 200-line Spring gateway + 100-line FastAPI router proving auth→queue→vLLM→stream→metrics path with p95 log.