Anti-Bot Engineer (Remote, Full-Time) [HR177]

Responsibilities

• Design, build, and maintain large-scale Python-based scraping systems targeting highly protected websites (including Google-like environments).
• Architect resilient extraction systems capable of handling dynamic, JavaScript-heavy pages using browser automation and hybrid approaches.
• Continuously adapt systems to frequent changes in page structures, request flows, and anti-bot mechanisms.
• Build robust, production-grade data extraction pipelines with strong emphasis on accuracy, observability, and fault tolerance.
• Implement advanced strategies such as proxy rotation, fingerprinting, session management, and request routing to ensure stability at scale.
• Monitor system health, proactively detect anomalies, and debug complex production failures across distributed systems.
• Optimise scraping infrastructure for performance, cost efficiency, and reliability at scale.
• Collaborate with data engineering and product teams to ensure scraped data is structured, validated, and trusted.
• Operate and improve systems running continuously in cloud environments.
• Document system architecture, scraping logic, and operational procedures for long-term maintainability.
• Contribute to improving resilience, automation, and adaptability in adversarial environments.

Requirements

• 7+ years of professional software engineering experience, with a strong focus on backend systems, data engineering, or distributed systems.
• Proven experience building and operating large-scale production web scraping systems.
• Deep hands-on experience scraping Google or similarly complex, heavily protected / anti-bot environments.
• Strong expertise in Python (or comparable production languages such as Go, Rust, or JavaScript).
• Strong understanding of HTTP internals: headers, cookies, TLS, redirects, sessions, and browser networking behaviour.
• Experience with browser automation frameworks such as Playwright, Selenium, Puppeteer, or equivalent.
• Strong knowledge of HTML parsing, DOM traversal, and high-performance data extraction techniques.
• Proven experience handling anti-bot systems, including rate limiting, CAPTCHAs, IP rotation, and fingerprinting.
• Experience designing asynchronous and concurrent systems for high-throughput workloads.
• Strong debugging skills across distributed, failure-prone production systems.
• Experience running cloud-based systems at scale with strong operational ownership.
• Comfortable working in highly adversarial and fast-changing technical environments.

Nice to Have

• Experience with Docker and Kubernetes in production environments.
• Exposure to distributed task queues or large-scale job orchestration systems.
• Experience with monitoring, anomaly detection, or data quality validation systems.
• Background in search intelligence, advertising tech, or competitive intelligence platforms.
• Experience building systems that operate against high-defence web environments at scale.
• Familiarity with observability tooling and production-grade system monitoring.
• Exposure to AI-assisted development workflows or agentic coding tools.

Benefits

• Fixed Shifts: 12:00 PM - 9:30 PM IST (Summer) | 1:00 PM - 10:30 PM IST (Winter)
• No Weekend Work: Real work-life balance, not just words
• Day 1 Benefits: Laptop and full medical insurance provided
• Support That Matters: Mentorship, community, and forums where ideas are shared
• True Belonging: A long-term career where your contributions are valued