ML Inference Engineer – Terminal-Bench Task Designer (vLLM / SGLang / TensorRT-LLM)

Toptal is looking for highly skilled ML inference and serving engineers to contribute to a specialized project: Terminal-Bench. This initiative focuses on creating rigorous, real-world benchmarks to evaluate how effectively AI agents can handle complex, expert-level tasks within a terminal environment. Rather than theoretical exercises, the company aims to translate genuine professional challenges—the kind experienced by engineers in the trenches—into reproducible benchmarks.

This is a task-based opportunity, meaning your compensation is directly tied to the challenges you design and successfully implement. It is an ideal role for engineers who enjoy architecting complex systems and want to contribute to the evolution of AI agent evaluation.

About the role

In this position, you will be responsible for translating your professional expertise into structured, command-line-solvable challenges. You won't just be writing code; you will be designing the entire ecosystem in which an AI agent operates. This involves setting up specific environments, defining file structures, and ensuring that model serving—including streaming and high-concurrency scenarios—is accurately represented.

Each challenge must be a self-contained unit of work that takes an expert roughly 60 to 75 minutes to complete. A critical component of your output will be the creation of automated validation scripts. These scripts must be sophisticated enough to detect subtle failures, such as numerical drift or metric inaccuracies, that might otherwise be missed by superficial checks. Because this project is focused on high-level AI testing, your work will be subject to peer review to ensure the challenges are fair, objective, and maintain an expert-level of difficulty.

• **Design complex challenges:** Transform real-world ML inference tasks from your professional background into reproducible "Terminal-Bench" scenarios.
• **Environment configuration:** Package tasks using Docker and Git, ensuring that every dependency, data point, and system requirement is fully documented and functional.
• **Develop validation frameworks:** Write reference solutions and build automated testing suites that confirm an agent has arrived at the correct result.
• **Quality assurance:** Iterate on your designs based on feedback from the Toptal review team and performance data from AI agents, ensuring tasks are neither too easy nor ambiguously defined.

What they're looking for

To be considered for this role, you must demonstrate a significant history of hands-on experience in the ML space. This is not a role for those who have only interacted with ML via API calls or educational notebooks.

• **Practical Expertise:** At least three years of active, hands-on experience in ML inference or serving, or a PhD paired with active, relevant research.
• **Technical Proficiency:** Deep familiarity with PyTorch or JAX internals. You should have direct experience with production-grade serving stacks such as vLLM, SGLang, or TensorRT-LLM.
• **Systems Thinking:** You must be able to explain complex failure modes in ML pipelines—specifically numerical drift or non-obvious metric bugs—and possess the ability to architect systems that catch these issues.
• **Infrastructure Skills:** Strong command-line fluency and a proven ability to manage complex file systems, databases, and APIs in a production-like setting.

For South African engineers, this role offers a unique way to participate in the global AI research community while working remotely. As a task-based engagement, it provides significant flexibility. While international freelance rates for specialized ML engineering can vary widely, experts in this niche in South Africa typically see hourly or project-based compensation that reflects the scarcity of these deep-learning infrastructure skills. Roles involving GPU-intensive serving stacks are increasingly sought after by local tech hubs and international firms alike, making this an excellent addition to a portfolio for anyone looking to transition into high-end AI research and development.

• 3+ years doing ML inference or serving work hands-on (not only studying, selling or managing it), or a PhD plus active research in the subfield.
• Did this work in the last 3 years and can name the systems, tools, data and rules used.
• Trained, served or evaluated models at real scale (multi-GPU or production serving), not only notebooks or API calls.
• Hands-on with PyTorch or JAX internals and serving stacks such as vLLM, SGLang or TensorRT-LLM.
• Can explain how ML results go wrong without errors (numerical drift, metric bugs) and how to catch them.
• Technical fluency in Python and the command line, and with files, CSV/JSON, APIs or databases.
• Availability to start by early November 2026 and complete tasks of about 60 to 75 minutes each.