Principal Software Engineer, Harness

Responsibilities

We are looking for a Principal Engineer to define and deliver the core Harness system at Atoms. You will build the execution, routing, caching, concurrency, evaluation, and quality feedback systems that help Atoms deliver complex AI-native capabilities reliably, with higher success rates, lower costs, and lower latency. What we are looking for - 10+ years building and operating products or systems in production. You have led critical engineering systems, automation platforms, evaluation systems, task schedulers, or complex infrastructure and remain deeply hands-on. - Strong backend, platform, or full-stack engineering skills, with proficiency in Python, Go, TypeScript, Rust, or similar languages. Ability to design reliable, observable, scalable systems that handle high concurrency. - Deep understanding of AI agents, LLM applications, and AI coding tools, including model and tool calls, context management, prompt caching, execution environments, failure modes, and evaluation challenges. - Ability to design routing across models, agent architectures, and toolchains, choosing execution strategies based on task type, context complexity, cost, latency, success rate, and quality requirements. - Deep understanding of concurrency, asynchronous jobs, queues, scheduling, state management, retries, and resource isolation for large-scale agent workloads. - High standards for code quality, explainability, and automation. Use systematic methods to make issues easier to reproduce, diagnose, and fix. - Ability to bring clarity to ambiguity and collaborate with low ego. Use structured communication and documentation to align Algorithms, Product, Engineering, and QA and keep work moving.

Qualifications

What you will do Build Atoms' agent execution and quality feedback systems so agent capabilities can be invoked reliably, tested under real conditions, and continuously improved for users worldwide. - Design, build, and maintain the core Harness system for task execution, model routing, prompt caching, concurrent scheduling, automated testing, result evaluation, failure attribution, traffic replay, and quality analysis. - Improve prompt cache hit rates for long-context, multi-turn, and repeated tasks to reduce model costs and latency. - Build routing across models, harnesses, and toolchains so tasks can select the right model, agent architecture, execution path, and fallback strategy. - Build an asynchronous, highly concurrent agent execution framework for task decomposition, queue scheduling, parallel execution, state recovery, retries, resource isolation, and result aggregation. - Build end-to-end agent evaluations that reproduce complex tasks, real user scenarios, and production issues for automated runs and ongoing comparison. - Work with Algorithms, Product, Engineering, and QA to turn production feedback, failures, evaluation results, and user behavior into improvements to models, prompts, toolchains, routing, and product experience. - Help define engineering quality practices, evaluation standards, and AI-native development workflows so teams can ship faster with more predictable quality.

Apply