Job Summary
GrabOn is looking for a highly skilled Python Developer with deep expertise in large-scale web scraping, browser automation, and distributed systems . The ideal candidate will design and maintain high-resilience, fault-tolerant scraping infrastructure capable of operating reliably against modern anti-bot defenses.
You will work on distributed, cloud-native scraping pipelines , collaborate on agentic and
autonomous systems , and continuously optimize for scale, cost, and reliability .
Key Responsibilities
Scraping & Automation
- Design, build, and maintain high-resilience web scraping systems at scale
- Implement advanced Selenium / Playwright automation (headless, stealth, browser fingerprint control)
- Handle anti-bot mechanisms , including :
- IP rotation & proxy orchestration
- CAPTCHA detection & mitigation
- Browser fingerprinting and evasion strategies
Distributed Systems & Cloud
Architect and maintain distributed scraping pipelines using :AWS LambdaSQSEC2 (auto-scaling worker fleets)Build retry-safe, idempotent, and fault-tolerant pipelinesEnsure graceful handling of failures, throttling, and partial outagesPerformance, Monitoring & Optimization
Monitor scraping throughput, failure rates, and infrastructure healthDebug production issues across distributed workersOptimize AWS cost, latency, and system throughputAgentic & Autonomous Systems
Collaborate on agent-based architectures for :Autonomous task executionDecision-making workflowsSelf-healing data pipelinesEngineering Practices
Write production-grade Python code (clean, testable, maintainable)Work comfortably in Linux server environmentsFollow best practices for concurrency, rate-limiting, and queue-based systemsMandatory Requirements (Non-Negotiable)
Strong Python expertise (production systems, not notebooks or scripts)Deep hands-on experience in web scraping at scaleExpert-level Selenium and / or Playwright knowledgeProven experience with AWS Lambda, SQS, and EC2Strong understanding of :Concurrency & parallelismQueues and distributed workflowsRetries, backoff strategies, and rate limitsComfortable managing and debugging Linux servers & cloud infrastructureSkills Required
Aws Lambda, Concurrency, Ec2, Linux, Sqs, Selenium, Python