Forage AI - Data Pipeline Engineer - Python / ETL / SQL

Description : Data Pipeline Engineer Web Services, WebCrawling, ETL, NLP(spaCy / LLM), AWS. Experience Level : 5-7 years of relevant experience in data engineering.

About Forage AI :

Forage AI is a pioneering AI-powered data extraction and automation company that transforms complex, unstructured web and document data into clean, structured intelligence. Our platform combines web crawling, NLP, LLMs, and agentic AI to deliver highly accurate firmographic and enterprise insights across numerous domains. Trusted by global clients in finance, real estate, and healthcare, Forage AI enables businesses to automate workflows, reduce manual rework, and access high-quality data at scale.

About the Role :

We are seeking a Data Pipeline Engineer to develop, optimize, and maintain production-grade data pipelines focused on web data extraction and ETL workflows. This is a hands-on role requiring strong experience with Python (as the primary programming language), spaCy, LLMs, webcrawling, and cloud deployment in containerized environments.

Youll have opportunities to propose, experiment with, and implement GenAI-driven approaches, innovative automations, and new strategies as part of our product and pipeline evolution. Candidates should have 5-8 years of relevant experience in data engineering, software engineering, or related fields.

Key Responsibilities :

• Design, build, and manage scalable pipelines for ingesting, processing, and storing web and API data.
• Develop robust web crawlers and scrapers in Python (Scrapy, lxml, Playwright) for structured and unstructured data.
• Create and monitor ETL workflows for data cleansing, transformation, and loading into PostgreSQL and MongoDB.
• Apply spaCy for NLP tasks and integrate / fine-tune modern LLMs for analytics.
• DriveGenAI-based innovation and automation in core data workflows.
• Develop and deploy secure REST APIs and web services for data access and Integrate RabbitMQ,Kafka, SQS(for distributed queueing), and Redis (for caching) into data workflows; also proficient with distributed queue tools such as Celery, TaskIQ.
• Containerize and deploy solutions using Docker on AWS(EC2, ECS, Lambda).
• Collaborate with data teams, maintain pipeline documentation, and enforce data quality standards.
• Maintain and enhance legacy in-house applications as required.

Technical Skills & Requirements :

What We Offer :

(ref : hirist.tech)