MLOps Engineer – AI Inference Platform (LLM Serving & Optimization)

About FPT Smart Cloud 

FPT Smart Cloud (FCI) – a member of FPT Corporation, pioneers AI & Cloud solutions in Vietnam. FCI was founded with the mission to generating an immense leap in productivity and agility in business operations.

FPT Smart Cloud aims at leading the industry by focusing on building a firm technological foundation, developing diversified ecosystem products, and reaching global connectivity.

• Customized to specific needs: Providing cloud-based products and solutions customized to each industry.
• All-in-one Platform: Consolidating FPT Smart Cloud technology and diverse business solutions all in one platform. AI & Cloud services are a Unify eco-system.
• Local market leadership: Outstanding Cloud and AI technology infrastructure and platform to help local businesses grow their products and services online.
• Deliver the future: Help customers achieve business outcomes faster by integrating world-class processes and technology.

Key Responsibilities 

• Design, deploy, and operate LLM inference services on the Model-as-a-Service (MaaS) marketplace, serving open-source foundation models (Llama 4, Qwen3, DeepSeek V3/R1) and multi-modal workloads (text generation, TTS, STT, vision, embeddings) at production scale across Vietnam and Japan regions.
• Tune and optimize inference engines (vLLM, TensorRT-LLM, SGLang, NVIDIA Dynamo, LMCache) using continuous batching, PagedAttention, Prefix Caching, Chunked Prefill, Speculative Decoding, and FlashAttention, FlashInfer to reduce TTFT, increase throughput, and lower cost-per-token.
• Build and maintain OpenAI-compatible API gateways (Kong, LiteLLM) supporting streaming SSE, function calling, multi-LoRA hot-swap, per-tenant rate limiting and quota, token-accurate billing telemetry, and cross-region routing.
• Solve LLM cold-start and serverless challenges for 70B+ parameter models: fast model loading (safetensors mmap, tensor streaming, Run:AI Model Streamer, Tensorizer), warm pool management, predictive scaling, and snapshot/restore (CRIU, Nydus, lazy container layers).
• Operate distributed inference at scale: Tensor / Pipeline / Expert Parallelism (for MoE models like DeepSeek V3, Qwen3-MoE), disaggregated prefill/decode workers, KV cache sizing and CPU/disk offload across H100 / H200 / B300 GPU fleets; apply FP8, FP4 quantization with measured accuracy/latency trade-offs.
• Run Kubernetes + GPU infrastructure with GitOps (ArgoCD, Helmfile): GPU operator, KEDA and custom HPA on token throughput, multi-tenant GPU sharing (MIG), canary and A/B rollouts for new model versions, and chaos testing (GPU failure injection, OOM scenarios).
• Build LLM-specific observability and SLOs: GPU metrics (NVIDIA DCGM), inference metrics (TTFT, ITL, tokens/sec, KV cache hit rate, queue time), per-token billing accuracy, response quality monitoring, plus security and safety controls (prompt injection defense, prompt/KV cache isolation between tenants, content moderation, PII redaction, audit logging for cross-border compliance including Japan APPI). Own on-call response with runbooks for LLM failure modes (loop generation, KV cache corruption, model load timeout).

Requirements 

• 5+ years of experience in MLOps / Platform / Infrastructure Engineering, with at least 2 years operating LLM or large-scale model serving in production.
• Production experience deploying LLMs ≥ 7B parameters with verifiable metrics — QPS, P95 TTFT/ITL, and cost per 1M tokens — not just calling pre-built APIs.
• Hands-on tuning expertise in vLLM, TensorRT-LLM, or SGLang beyond default configuration; deep understanding of continuous batching, PagedAttention, prefix caching, and KV cache management.
• Strong Kubernetes skills with production GPU scheduling experience: GPU operator, KEDA, custom HPA, node feature discovery, MIG, and multi-tenant GPU isolation.
• Ability to measure and optimize TTFT (Time To First Token), ITL (Inter-Token Latency), throughput, and cost-per-token using DCGM, Prometheus, and Grafana — not standard HTTP latency.
• Working knowledge of GPU architecture (H100/H200, NVLink, HBM3, CUDA Cores, Tensor Cores) and CUDA basics — able to read CUDA code and debug; kernel writing not required.
• Experience with distributed inference: Tensor Parallelism, Pipeline Parallelism, Expert Parallelism for MoE models, and prefill/decode disaggregation.
• (Preferred) Production experience with multi-LoRA serving (S-LoRA, Punica, vLLM multi-LoRA) and MoE models (DeepSeek V3, Mixtral, Qwen3-MoE).
• (Preferred) Experience building OpenAI-compatible gateways from scratch or via LiteLLM / vLLM router, supporting streaming, tool calling, vision and audio.
• Proficient in Python at source-reading level (able to read and debug vLLM / TensorRT-LLM internals); Go or Rust for high-performance gateway/proxy services is a strong plus. Expert in Bash, Docker, and Linux performance tooling.
• (Preferred) FP8 / FP4 Weight Quantization, KV Cache Quantization deployed in production; open-source contributions to vLLM / SGLang / TGI, or technical blog/talks on LLM inference.

Top Benefits

• Salary: Competitive, pay according to ability. Negotiation during the interview.
• Social insurance and health insurance according to labor laws.
• Creative, open-minded working environment that respects individuals
• FPT Premium Care package
• Activities and culture with FCI and FPT Corporation
• Study support package for children of FCI union
• Sponsor related courses and certifications

Working Environment

• Working Location:
  • Site Hanoi: 7th Floor, FPT Tower, no. 10 Pham Van Bach Street, Dich Vong ward, Cau Giay district, Hanoi
• Working hours:
  • 8h30 AM – 12h00 PM
  • 1h00 PM – 5h30 PM

• Working days: Monday to Friday (weekends off)

Contact Person

Pham Thi Ha My (Ms.) – Talent Acquisition Team Leader
Email: [email protected]
Phone: 0962456194
FPT Smart Cloud (FCI) Co., LTD
Address: 7th Floor, FPT Tower, No. 10 Pham Van Bach, Cau Giay Dist, Hanoi
Websites: FPT Cloud | FPT AIBusiness Analyst (BSS)