Data Architect

Key Responsibilities :

Design, develop, and optimize scalable data pipelines and workflows using Databricks and Apache Spark on AWS.

Collaborate with data scientists, analysts, and other stakeholders to transform data into actionable insights.

Implement data ingestion processes from various sources including S3, RDS, Redshift, and external APIs.

Optimize Spark jobs for performance and cost efficiency.

Monitor and troubleshoot data workflows, ensuring data quality and integrity.

Automate deployment and orchestration of data pipelines using AWS services like Lambda, Step Functions, or Glue.

Maintain security and governance standards in cloud data environments.

Document processes, data lineage, and architecture designs.