Senior Staff Software Engineer, High Performance Inference System

Senior Staff Software Engineer – High Performance Inference System

Mission: 

Join the team that builds and operates Groq’s real-time, distributed inference system delivering large scale inference for LLMs and next-gen AI applications at ultra-low latency. Your work will optimize for heterogeneous hardware, dynamic global workloads, and extreme performance—all while running code at the edge of physics.

Responsibilities & opportunities in this role:

• Distributed Systems Engineering: Design and implement scalable, low-latency runtime systems that coordinate thousands of GroqChips across a software-scheduled interconnect.
• Low-Level Optimization: Develop deterministic, hardware-aware abstractions that prioritize execution speed, fault tolerance, and reliability.
• Performance & Diagnostics: Build tools and infrastructure to support real-time system observability, diagnostics, and SLO improvements.
• Future-Proofing: Evolve Groq’s system stack to support emerging silicon, topologies, and heterogeneous accelerators (e.g., FPGAs).
• Cross-Functional Collaboration: Partner with teams across compiler, infra, cloud, hardware, and data centers to align architecture and drive shared progress.

Ideal candidates have/are:

• Consistently ship high-impact, production-ready systems code.
• Have deep knowledge of computer architecture, operating systems, algorithms, and hardware-software interfaces.
• Are fluent in low-level systems languages such as C++ or Rust, and comfortable with hardware-aware programming.
• Rigorously profile and optimize for latency, throughput, and resource efficiency—every cycle counts.
• Believe in automation and CI/CD best practices—you don’t ship untested code.
• Thrive across the stack—from kernel internals to hardware integration to cloud load balancers.
• Communicate clearly, make pragmatic technical decisions, and write maintainable code for the long term.
• Ensures code stays fast, scales well, and takes ownership of outcomes.

• Operating large-scale distributed systems for real-time, high-traffic services.
• Deploying and optimizing ML or HPC workloads in production environments.
• Hands-on experience with GPUs, FPGAs, or ASICs in performance-critical systems.
• Familiarity with ML frameworks (e.g., PyTorch) or compiler tools (e.g., MLIR).
• Experience delivering complex projects in fast-paced, high-impact environments.

Attributes of a Groqster:

• Humility - Egos are checked at the door
• Collaborative & Team Savvy - We make up the smartest person in the room, together
• Growth & Giver Mindset - Learn it all versus know it all, we share knowledge generously
• Curious & Innovative - Take a creative approach to projects, problems, and design
• Passion, Grit, & Boldness - no limit thinking, fueling informed risk taking

If this sounds like you, we’d love to hear from you!

Compensation: At Groq, a competitive base salary is part of our comprehensive compensation package, which includes equity and benefits. For this role, the base salary range is $248,710 - $336,490, determined by your skills, qualifications, experience and internal benchmarks.