Roles and Responsibilities
- Lead the design, development, and maintenance of data pipelines and ETL processes
architect and implement scalable data solutions using Databricks and AWS.
Optimize data storage and retrieval systems using Rockset, Clickhouse, and CrateDB.Develop and maintain data APIs using FastAPI.Orchestrate and automate data workflows using Airflow.Collaborate with data scientists and analysts to support their data needs.Ensure data quality, security, and compliance across all data systems.Mentor junior data engineers and promote best practices in data engineering.Evaluate and implement new data technologies to improve the data infrastructure.Participate in cross-functional projects and provide technical leadership.Manage and optimize data storage solutions using AWS S3, implementing bestpractices for data lakes and data warehouses.
Private and Confidential www.fissionlabs.com [HIDDEN TEXT]
Implement and manage Databricks Unity Catalog for centralized data governanceand access control across the organization.
Qualifications Required
Bachelor's or Master's degree in Computer Science, Engineering, or related field5+ years of experience in data engineering, with at least 2-3 years in a lead roleStrong proficiency in Python, PySpark, and SQLExtensive experience with Databricks and AWS cloud servicesHands-on experience with Airflow for workflow orchestrationFamiliarity with FastAPI for building high-performance APIsExperience with columnar databases like Rockset, Clickhouse, and CrateDBSolid understanding of data modeling, data warehousing, and ETL processesExperience with version control systems (e.g., Git) and CI / CD pipelinesExcellent problem-solving skills and ability to work in a fast-paced environmentStrong communication skills and ability to work effectively in cross-functional teamsKnowledge of data governance, security, and compliance best practicesProficiency in designing and implementing data lake architectures using AWS S3Experience with Databricks Unity Catalog or similar data governance and metadatamanagement tools
Skills and Experience Required
Tech Stack
Databricks, Python, PySpark, SQL, Airflow, FastAPI, AWS (S3, IAM, ECR, Lambda), Rockset,
Clickhouse, CrateDB
Why you'll love working with us :
Opportunity to work on business challenges from top global clientele with high impact.Vast opportunities for self-development, including online university access andsponsored certifications.
Sponsored Tech Talks, industry events & seminars to foster innovation and learning.Generous benefits package including health insurance, retirement benefits, flexiblework hours, and more.
Supportive work environment with forums to explore passions beyond work.This role presents an exciting opportunity for a motivated individual to contribute to the
development of cutting-edge solutions while advancing their career in a dynamic and
collaborative environment.
Skills Required
Pyspark, Databricks, Python, Sql, data engineering , Cloud Services
