SDE - Infrastructure Engineer

Company Description :

ClearDemand is the  leader in AI-driven price and promotion optimization for retailers . Our platform transforms pricing from a challenge to a competitive advantage, helping retailers make smarter, data-backed decisions across the entire pricing lifecycle. By  integrating competitive intelligence, pricing rules, and demand modeling , we enable retailers to  maximize profit, drive growth, and enhance customer  loyalty — all while maintaining pricing compliance and brand integrity. With ClearDemand, retailers stay ahead of the market, automate complex pricing decisions, and unlock new opportunities for growth.

Why This Role Matters :

Data is the foundation of our business, and your work will ensure that we continue to deliver high-quality competitive intelligence at scale. Web platforms are constantly evolving, deploying sophisticated anti-bot measures—your job is to stay ahead of them. If you thrive on solving complex technical challenges and enjoy working with real-world data at an immense scale, this role is for you.

We seek a Software Development Engineer with expertise in cloud infrastructure, Big Data and web crawling technologies. This role bridges system reliability engineering with scalable data extraction solutions, ensuring our infrastructure remains robust and capable of handling high-volume data collection. You will design resilient systems, optimize automation pipelines, and tackle challenges posed by advanced bot-detection mechanisms.

Key Responsibilities :

• Architect, deploy, and manage scalable cloud environments (AWS / GCP / DO) to support distributed data processing solutions to handle terabyte-scale datasets and billions of records efficiently
• Automate infrastructure provisioning, monitoring, and disaster recovery using tools like Terraform, Kubernetes, and Prometheus.
• Optimize CI / CD pipelines to ensure seamless deployment of web scraping workflows and infrastructure updates.
• Develop and maintain stealthy web scrapers using Puppeteer, Playwright, and headless chromium browsers.
• Reverse-engineer bot-detection mechanisms (e.g., TLS fingerprinting, CAPTCHA solving) and implement evasion strategies.
• Monitor system health, troubleshoot bottlenecks, and ensure 99.99% uptime for data collection and processing pipelines.
• Implement security best practices for cloud infrastructure, including intrusion detection, data encryption, and compliance audits.
• Partner with data collection, ML and SaaS teams to align infrastructure scalability with evolving data needs
• Research emerging technologies to stay ahead of anti-bot trends including technologies like Kasada, PerimeterX, Akamai, Cloudflare, and more.

Required Skills :

Why Join Us :

At ClearDemand, you’ll be at the forefront of innovation in the data engineering space, working with cutting-edge technologies and a talented team. If you're passionate about building scalable systems, handling large-scale distributed data, and solving complex data challenges, we’d love to have you on board.