Software Engineer III

Job Brief: 

The Data Acquisition team operates at the core of our platform, engineering efficient systems that empower all our clients through all our products. We design and run large scale, highly distributed pipelines capable of scheduling, orchestrating, and executing tens of millions of data collection tasks daily across a diverse and constantly evolving landscape of sources. This involves tackling complex challenges in high-throughput web crawling, adaptive extraction, intelligent batching, fault tolerant processing, and seamless distribution of mission critical datasets. We are seeking a Software Engineer who thrives on building resilient, scalable systems, solving real world data engineering puzzles, and pushing the boundaries of automation and efficiency in large-scale information retrieval. 

Essential Function & Responsibilities: 

• Design and implement advanced capabilities to detect and counter anti-bot protection methods. 
• Engineer complex network requests that accurately mimic real user behavior and environments. 
• Develop network services leveraging HTTP(S) protocol versions 1.1, 2, and 3. 
• Implement and manage egress proxy networks with MITM traffic interception and routing optimization. 
• Extend Chrome DevTools Protocol (CDP) with custom commands to support specialized use cases. 
• Build and maintain scalable WebSocket infrastructures optimized for short-lived, high-volume connections. 
• Design distributed caching systems to support high throughput containerized worker pools. 
• Deploy and manage geographically distributed proxies with automated failover and load balancing. 
• Optimize compute, storage, and traffic costs for high-volume cloud-based services. 
• Proficiency with data parsing, transformation, and storage technologies. 
• Unwavering focus on delivering measurable value to customers