Data Engineer - WGDT

We are seeking a highly motivated and detail-oriented Data Engineer with expertise in designing, constructing, and maintaining robust data architectures and infrastructure. In this critical role, you will contribute to the successful implementation of digital government policies and programs in India. You will be instrumental in developing scalable and efficient systems to manage large volumes of data, making it accessible for analysis and decision-making, and ultimately driving innovation and optimizing operations across various government ministries and state departments.

Key Responsibilities

• Data Architecture Design: Design, develop, and maintain scalable data pipelines and infrastructure for ingesting, processing, storing, and analyzing large volumes of data efficiently, translating business requirements into technical solutions.
• Data Integration: Integrate data from various sources (databases, APIs, streaming platforms, third-party systems), ensuring reliable and efficient collection while maintaining data quality and integrity as per government standards.
• Data Modeling: Design and implement data models (dimensional modeling, normalization, denormalization) to organize and structure data for efficient storage and retrieval based on project requirements.
• Data Pipeline Development / ETL: Develop ETL processes to extract, transform, and load data into target systems, including writing scripts or using ETL tools to automate workflows and ensure data accuracy.
• Data Quality and Governance: Implement data quality checks and governance policies, designing and tracking data lineage, data stewardship, metadata management, and building business glossaries.
• Data Lakes or Warehousing: Design and maintain data lakes and data warehouses for structured, semi-structured, and unstructured data at scale, integrating with big data processing frameworks (e.g., Apache Hadoop, Apache Spark, Apache Flink), and with machine learning and data visualization tools.
• Data Security: Implement security practices, technologies, and policies to protect data throughout its lifecycle, including access control, encryption, data masking, data loss prevention, and compliance with regulations like DPDP and GDPR.
• Database Management: Administer and optimize both relational and NoSQL databases to effectively manage large data volumes.
• Data Migration: Plan and execute data migration projects, ensuring data consistency and minimal downtime.
• Performance Optimization: Optimize data pipelines and queries for performance and scalability, identifying bottlenecks, tuning configurations, and implementing caching and indexing strategies.
• Collaboration: Work with data scientists, analysts, and other stakeholders to understand data requirements and provide access to necessary resources, and collaborate with IT operations for deployment and maintenance.
• Documentation and Reporting: Document data models, data pipelines/ETL processes, and system configurations, providing training to ensure system sustainability and maintainability.
• Continuous Learning: Stay updated with the latest technologies and trends in data engineering, engaging with the community and participating in relevant programs to enhance skills.

Qualifications

• A Bachelor's or Master's degree in Computer Science, Software Engineering, Data Science, or an equivalent field.

Skills & Expertise

• Database Management: Strong expertise in SQL databases (e.g., MySQL, PostgreSQL) and NoSQL databases (e.g., MongoDB, Cassandra).
• Big Data Technologies: Familiarity with big data technologies such as Apache Hadoop, Spark, and related ecosystem components.
• ETL Tools: Experience with ETL tools (e.g., Apache NiFi, Talend, Apache Airflow, Talend Open Studio, Pentaho, Infosphere) for designing and orchestrating data workflows.
• Data Modeling and Warehousing: Knowledge of data modeling techniques and experience with data warehousing solutions (e.g., Amazon Redshift, Google BigQuery, Snowflake).
• Data Governance and Security: Understanding of data governance principles and best practices for ensuring data quality and security, including compliance with DPDP, GDPR, HIPAA, or PCI DSS.
• Cloud Computing: Experience with cloud platforms (e.g., AWS, Azure, Google Cloud) and their data services.
• Streaming Data Processing: Familiarity with real-time data processing frameworks (e.g., Apache Kafka, Apache Flink).
• Analytical thinking and problem-solving aptitude for identifying and resolving bottlenecks and optimizing systems.
• Strong communication skills for effective collaboration and documentation.

Skills Required
Data Modeling, Database Management, Etl Tools, Big Data Testing, Data Warehousing, Cloud Computing