Operate and optimize large-scale LLM pre-training on 1,000+ GPU clusters using PyTorch, DeepSpeed, or Megatron-LM. Improve networking (InfiniBand/RDMA), memory management, checkpointing, and failure recovery. Manage SLURM/Kubernetes GPU clusters and apply systems engineering (C++, CUDA, Python) and 3D parallelism techniques.
We are seeking a highly skilled LLM Pre-training & Distributed Systems Engineer. This role is essential for orchestrating large-scale machine learning training runs and optimizing distributed infrastructure. The ideal candidate will have a deep understanding of GPU clusters and extensive experience in system engineering to ensure efficient and reliable training processes.
Responsibilities:
- Orchestrate distributed training runs across 1,000+ GPUs using PyTorch, DeepSpeed, or Megatron-LM.
- Optimize networking (InfiniBand/RDMA) and memory management to prevent out-of-memory errors.
- Automate checkpointing and failure recovery during month-long training runs.
Required Skills:
- Deep expertise in 3D parallelism (Data, Tensor, Pipeline).
- Experience managing SLURM or Kubernetes-based GPU clusters.
- Strong systems engineering background (C++, CUDA, Python).
Similar Jobs
.png)
Artificial Intelligence • Blockchain • Fintech • Financial Services • Cryptocurrency • NFT • Web3
Own and lead Coinbase’s developer infrastructure platforms, including CI, build systems, and deployment orchestration. Set technical direction for complex cross-team initiatives, improve build and deployment performance, manage reliability and observability, and drive platform migrations and adoption. Partner with engineering leaders, operate production systems, mentor engineers, and deliver scalable tools used across the organization.
Top Skills:
Build SystemsCi/CdDeployment OrchestrationDistributed SystemsGenerative AiGoObservability
AdTech • Artificial Intelligence • Big Data • Machine Learning • Marketing Tech • Mobile • Software
Own the strategy, business case, experimentation roadmap, and launch of Liftoff’s zero-to-one e-commerce advertising platform. Lead product development from beta toward general availability, partnering with engineering, machine learning, legal, finance, sales, and customers. Responsibilities include optimizing funnel performance, defining traffic quality standards, managing identity and attribution architecture, overseeing e-commerce integrations, guiding go-to-market strategy, and making data-driven scaling, pivot, or shutdown decisions.
Top Skills:
AttCustomer Privacy ApiGoogle Enhanced ConversionsIncrementality TestingItpMeta CapiMmmMtaPixelsProbabilistic MatchingServer-To-Server PostbacksShopifyWebhooks
Big Data • Fintech • Mobile • Payments • Financial Services
Owns delivery of backend money movement and card ledger systems, leading engineers through complex problems and ambiguity. Collaborates with product, design, analytics, and engineering stakeholders; defines scalable technical solutions; supports system availability, monitoring, on-call operations, code quality, and engineering standards. The role also includes mentoring and developing team talent while building highly available distributed systems for Affirm’s financial products.
Top Skills:
AWSKotlinKubernetesMySQLPython
What you need to know about the Charlotte Tech Scene
Ranked among the hottest tech cities in 2024 by CompTIA, Charlotte is quickly cementing its place as a major U.S. tech hub. Home to more than 90,000 tech workers, the city’s ecosystem is primed for continued growth, fueled by billions in annual funding from heavyweights like Microsoft and RevTech Labs, which has created thousands of fintech jobs and made the city a go-to for tech pros looking for their next big opportunity.
Key Facts About Charlotte Tech
- Number of Tech Workers: 90,859; 6.5% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Lowe’s, Bank of America, TIAA, Microsoft, Honeywell
- Key Industries: Fintech, artificial intelligence, cybersecurity, cloud computing, e-commerce
- Funding Landscape: $3.1 billion in venture capital funding in 2024 (CED)
- Notable Investors: Microsoft, Google, Falfurrias Management Partners, RevTech Labs Foundation
- Research Centers and Universities: University of North Carolina at Charlotte, Northeastern University, North Carolina Research Campus
