Staff Software Engineer - Full Stack / SRE (Remote)

This is a remote position.

Ad Hoc is a technology company that empowers organizations to deliver scalable, impactful digital services. Using modern, agile methods, our team creates products that meet people’s needs and transform their experience of government.

Work on things that matter

Our collaborations have shaped some of the defining moments in public-sector service delivery. We’ve helped build products that connect Veterans to tailored services, help millions access affordable health care, and support important programs like Head Start. As we work with agencies to deliver critical services, we’re also changing how the government approaches technology.

Built for a remote life

Our culture, communications, and tools are built for remote work, enabling us to bring together top talent nationwide. At Ad Hoc, remote life empowers our teams to design work environments that fit their lives and that foster flexibility and collaboration to achieve positive outcomes for our customers.

Committed to high expectations and a welcoming culture

Ad Hoc values acceptance, accountability, and humility. We aren’t heroes. We learn from our mistakes and improve the process for the next time. We build small, inclusive teams to collaborate closely with our partners to solve the right problems and deliver software that works.

The Veterans Affairs business unit helps transform the VA into a modern digital services organization where Veteran outcomes are at the center of every effort. We partner with the VA to design and deliver seamless user experiences for Veterans, their families and caregivers, and VA employees. By applying better practices in service design, product management, and technology, we enable the VA to increase the use, quality, and reliability of services and decrease the time Veterans spend waiting for outcomes.

Primary Responsibilities:

Staff Software Engineer - Full Stack serves as an emerging subject matter expert, while serving as an individual contributor that exhibits a high level of trust, autonomy and empowerment. With minimal oversight from leadership, you will be responsible for leading and monitoring the ability to meet scope, schedule and delivery requirements and may support adjacent programs within the business unit. A Staff Software Engineer - Full Stack exhibits strong influential skills to represent the software engineering practice in planning sessions, advocating for the best technical solution, while communicating tradeoffs. You will serve as the primary lead and proactively communicate with stakeholders; while utilizing strong influential skills to drive improvements in software engineering processes and practices. Primary expectations of a Staff Software Engineer - Full Stack include:

• Plans and executes on roadmaps for new projects without explicit guidance and direction from technical supervisors

• Actively participates in conversations and planning sessions with partners and key stakeholders

• Periodically travels to work with and present to clients, partners, and stakeholders

• Elaborates on and evolves complex and ambiguous products to uncover constraints and new opportunities

• Reduces ambiguity in the systems they work with, including adding documentation, refactoring, and automated testing

• Effectively communicates on existing systems, design decisions, past performance, and a major history of the projects that they’ve been part of for bid-writing, tech demos, and other potentially client-facing communications

• Participates in technical depth interviews with new candidates

• Presents on technical topics effectively, articulating implementation complexity and other costs to inform business decisions

The Watchtower Team is a specialized SRE team supporting va.gov. We're not your typical SREs. We act as both consultants and hands-on problem solvers, working across multiple teams to improve the reliability, performance, and overall health of the va.gov platform. This is a high-impact role where you'll be directly contributing to the experience of Veterans accessing critical services.

We're looking for a highly skilled and adaptable engineer who can diagnose and resolve complex issues in a large web application. You'll be using your expertise in observability, incident response, and software development to improve the stability and performance of the platform. You'll also collaborate with other engineering teams, providing guidance and support to prevent future issues.

• Troubleshoot and Resolve Production Issues: Diagnose and fix performance bottlenecks, errors, and other issues within the va.gov application (primarily a Ruby on Rails monolith, including Sidekiq background jobs, but familiarity with similar frameworks is valuable).

• Observability & Monitoring: Utilize DataDog (and potentially Dynatrace) to monitor application performance, identify anomalies, and proactively address potential problems. Develop and maintain relevant dashboards and alerts.

• Incident Response and On-Call Rotation ("The Watch"): Participate in our on-call rotation approximately once per month. Unlike traditional pager-driven on-call, "The Watch" involves reviewing the previous day's alerts and ensuring no silent failures occurred (such as background jobs exhausting without an alternate submission path). During your on-call week, expect to work 2-4 hours each day on the weekend to maintain system reliability.

• Code Contributions: Write and review code to improve observability and fix bugs (Ruby on Rails), implement improvements, and maintain internal tools (JavaScript/SvelteKit, and Python).

• Consulting & Collaboration: Work closely with other engineering teams to provide guidance on best practices for observability, reliability, and performance. Communicate technical issues clearly to both technical and non-technical audiences.

• Process Improvement: Identify and implement improvements to our monitoring, alerting, and incident response processes. Contribute to documentation and runbooks.

• Maintain Internal Tools: Contribute to the development and maintenance of a small SvelteKit application used for tracking team metrics and success.

• Bachelor’s degree and 9+ years of engineering experience or Site Reliability Engineer

  • Relevant years of experience may be substituted for education

• 3+ years of experience with backend web application development in a production environment. Strong preference for Ruby on Rails experience, but candidates with demonstrable experience in other dynamic languages (e.g., Python/Django/Flask, Node.js/Express, PHP/Laravel) or compiled languages with web frameworks (e.g., Java/Spring, C#/.NET) will be considered.

• Experience with Sidekiq or other background job processing framework. If not Sidekiq, experience must be with a comparable system in their chosen language/framework (e.g., Celery for Python).

• Proven experience with application performance monitoring (APM) tools, specifically DataDog and/or Dynatrace. Ability to interpret metrics and identify root causes of performance issues.

• Demonstrated experience in incident response and troubleshooting complex production issues.

• Experience with at least one modern JavaScript framework (React, Angular, Vue, Svelte, etc.).

• Excellent communication, collaboration, and consulting skills.

• Ability to work effectively in a fast-paced, dynamic environment.

• Experience working within an Agile environment.

• Experience with vets-api.

• Prior experience working within the VA/OCTO environment or any large government software deployment that integrates with multiple legacy services.

• Experience with Python for scripting, API interactions, and ETL/data engineering tasks.

• General understanding of DevOps concepts (containerization, virtualization, networking).

• Familiarity with GitHub Actions.

• Experience with the U.S. Web Design System (USWDS).

To learn more about working at Ad Hoc, please visit:https://adhocteam.us/join

Benefits:

• Company-subsidized health, dental, and vision insurance


• Flexible PTO


• 401K with employer match


• Paid parental leave after one year of service


• Employee Assistance Program

Ad Hoc LLC is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, national origin, ancestry, sex, sexual orientation, gender identity or expression, religion, age, pregnancy, disability, work-related injury, covered veteran status, political ideology, marital status, or any other factor that the law protects from employment discrimination.

We value the unique skills gained through military service and encourage veterans and transitioning service members to apply.

In support of various state and city equal pay transparency laws, Ad Hoc job descriptions feature the starting range we reasonably expect to pay to candidates who would join our team with little to no need for training on the responsibilities we've outlined above. Actual compensation is influenced by a wide range of factors including but not limited to skill set, level of experience, and responsibility. The range of starting pay for this role is $140,000 - $155,000. Our recruiters will be happy to answer any questions you may have, and we look forward to learning more about your salary requirements.