DevTalent Job Board - DAT-030

DAT-030 Azure Data Engineer

Tri-State, NJ

Our client is seeking a skilled data processing specialist to design and develop innovative solutions on a leading cloud platform using a popular Python-based data processing framework. The goal is to leverage extensive experience in this framework and hands-on expertise in cloud computing to navigate complex data challenges, ensuring adherence to industry regulations while driving meaningful outcomes. Our client aims to rapidly adapt to emerging technologies and create scalable data solutions that enhance operational efficiency and informed decision-making.

Responsibilities:

  • Automate data workflows for both real-time and batch processing
  • Integrate data from various sources and sync data to various systems
  • Implement data quality checks and ensure data integrity, consistency and reliability
  • Help build the required infrastructure and DevOps pipelines
  • Monitor and troubleshoot data processing jobs, ensuring high availability and performance.
  • Implement best practices for data governance and security within the cloud platform.
  • Stay up-to-date with emerging trends in data management, cloud computing, and the client's industry.

Requirements:

  • Azure/ Power platform Data technologies like Data Factory, Data Fabric, Data Lake Storage. SQL Database, Data Explorer, Synapse Analytics, Purview
  • Azure Identity management such as AKV, UAMI.
  • Spark, Kafka and Hive, and related
  • setting up CI/CD pipeline like GitLab/Teamcity/Jenkins.
  • Proven experience working with the cloud platform's services (including data integration, data engineering, and analytics tools).
  • Prior experience in a related sector is mandatory.
  • Proficient in the Python-based framework for data processing and ETL tasks.
  • Experience with relational and non-relational databases.
  • Familiarity with data storage concepts and tools.
  • Knowledge of cloud services related to data management and analytics.

Preferred, but not required:

  • Experience with virtualization technologies (e.g., containerization, orchestration).
  • Familiarity with predictive modeling frameworks (e.g., machine learning libraries, deep learning tools).