Photon Logo

Photon

QA Engineer (Performance)- Dallas, TX

Reposted 3 Months Ago
Be an Early Applicant
In-Office or Remote
Hiring Remotely in United States
38K-133K Annually
Senior level
In-Office or Remote
Hiring Remotely in United States
38K-133K Annually
Senior level
Benchmark and optimize latency and throughput for an agentic AI platform. Stress-test multi-agent concurrency, RAG/vector DB retrieval, token throughput, orchestration hand-offs, and integrate automated performance regression tests into CI/CD while balancing cost versus performance.
The summary above was generated by AI

We are looking for a Performance QA Engineer to specialize in benchmarking and optimizing our Agentic AI platform. You will be the gatekeeper of the "User Experience of Thought," ensuring that as our AI agents plan, reason, and execute tasks, they do so within acceptable timeframes and cost-efficiency boundaries.

Your mission is to stress-test the entire AI pipeline—from the initial prompt to the final autonomous action—identifying bottlenecks in LLM response times, RAG (Retrieval-Augmented Generation) retrieval speeds, and third-party API orchestration.

Key Responsibilities

  • Latency Benchmarking: Measure and optimize TTFT (Time to First Token) and Total Request Latency for complex agentic workflows that involve multiple reasoning steps.
  • Agentic Loop Stress Testing: Simulate high-concurrency environments to see how the system handles hundreds of autonomous agents running simultaneously, particularly focusing on API rate limits and GPU/compute bottlenecks.
  • RAG Performance Analysis: Test the speed and efficiency of the vector database retrieval process. Identify how increasing the "context window" size impacts overall system performance.
  • Token Throughput Monitoring: Analyze the "tokens per second" (TPS) metrics and identify when model-switching (e.g., from a large model to a smaller one) is necessary to maintain performance.
  • Cost vs. Performance Optimization: Create reports that balance performance gains against token costs, helping the team find the "sweet spot" for production-grade agents.
  • Orchestration Bottleneck Identification: Use profiling tools to find delays in the "hand-off" between different agents or between the agent and external tools (APIs, databases).
  • Automated Performance Regressions: Integrate performance testing into the CI/CD pipeline to ensure that new prompt versions or architectural changes don't degrade the agent's speed.

Required Skills & Qualifications

  • Experience: 8+ years in Performance Engineering, with a specific focus on AI/ML applications or high-concurrency distributed systems.
  • Tooling Proficiency: Expert-level experience with performance testing tools like Locust, JMeter, or k6, specifically customized for Python-based AI backends.
  • Python Mastery: Strong ability to write custom scripts to simulate complex, multi-step user/agent interactions.
  • AI Infrastructure Knowledge: Understanding of LLM-specific performance factors, such as quantization, KV caching, and the impact of different model architectures on latency.
  • Observability Expertise: Experience with tools like Prometheus, Grafana, LangSmith, or Weights & Biases to monitor system health and AI-specific metrics.
  • Database Performance: Experience testing the query latency of Vector Databases  under heavy load.

Compensation, Benefits and Duration

Minimum Compensation: USD 38,000
Maximum Compensation: USD 133,000
Compensation is based on actual experience and qualifications of the candidate. The above is a reasonable and a good faith estimate for the role.
Medical, vision, and dental benefits, 401k retirement plan, variable pay/incentives, paid time off, and paid holidays are available for full time employees.
This position is not available for independent contractors
No applications will be considered if received more than 120 days after the date of this post

Similar Jobs

23 Minutes Ago
Remote or Hybrid
65K-90K Annually
Mid level
65K-90K Annually
Mid level
AdTech • Cloud • Digital Media • Information Technology • News + Entertainment • App development
Supports IT services for remote operations and live events, including endpoint devices, network connectivity, infrastructure kits, inventory, documentation, dashboards, and vendor coordination. Partners with broadcast engineering, digital, and corporate security teams to integrate broadcast and IT technologies. Implements technology and AI initiatives, manages projects, and provides onsite support at remote events. Requires extensive travel, overtime, weekends, and five-day onsite office work.
Top Skills: DesktopsGrafanaLan/WanLaptopsMicrosoft CopilotMobile DevicesRoutersSelectoraiSevoneSnmpSwitches
23 Minutes Ago
Remote or Hybrid
120K-170K Annually
Senior level
120K-170K Annually
Senior level
AdTech • Cloud • Digital Media • Information Technology • News + Entertainment • App development
Design, develop, and maintain high-performance Go-based software for global VOD, live, and linear streaming delivery. Build reliable services for dynamic ad insertion and manifest manipulation, partner with SRE on highly available multi-region systems, use observability and analytics for performance and incident analysis, and contribute to multi-CDN strategies, architecture, code reviews, and engineering standards.
Top Skills: AkamaiAWSAzureCdnsCi/CdCloudflareCloudfrontDashDockerFastlyGCPGitGoHlsKubernetesPythonSre
24 Minutes Ago
Remote
Florida, USA
110K-253K Annually
Senior level
110K-253K Annually
Senior level
Artificial Intelligence • Cloud • Information Technology • Consulting
Designs, configures, troubleshoots, and maintains enterprise firewalls, security policies, and network segmentation. The role remediates policy and network risks, supports Juniper segmentation and NSPM capabilities, evaluates firewall changes, develops Layer 7 controls, investigates connectivity issues, and collaborates across security, network, cloud, and infrastructure teams. It also maintains technical documentation and supports secure, compliant network operations.
Top Skills: AutomationDnsEnterprise FirewallsIntrusion PreventionJuniperLayer 7 FirewallNatNetwork MonitoringNetwork Security Policy Management (Nspm)Network SegmentationPacket CaptureRoutingScriptingSIEMSsl InspectionSubnettingSwitchingTcp/IpUrl FilteringVpnVulnerability Management

What you need to know about the Charlotte Tech Scene

Ranked among the hottest tech cities in 2024 by CompTIA, Charlotte is quickly cementing its place as a major U.S. tech hub. Home to more than 90,000 tech workers, the city’s ecosystem is primed for continued growth, fueled by billions in annual funding from heavyweights like Microsoft and RevTech Labs, which has created thousands of fintech jobs and made the city a go-to for tech pros looking for their next big opportunity.

Key Facts About Charlotte Tech

  • Number of Tech Workers: 90,859; 6.5% of overall workforce (2024 CompTIA survey)
  • Major Tech Employers: Lowe’s, Bank of America, TIAA, Microsoft, Honeywell
  • Key Industries: Fintech, artificial intelligence, cybersecurity, cloud computing, e-commerce
  • Funding Landscape: $3.1 billion in venture capital funding in 2024 (CED)
  • Notable Investors: Microsoft, Google, Falfurrias Management Partners, RevTech Labs Foundation
  • Research Centers and Universities: University of North Carolina at Charlotte, Northeastern University, North Carolina Research Campus

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account