Data Pipeline Engineer

Responsibilities

• Monitor and maintain the health and efficiency of data pipelines
• Troubleshoot and perform root cause analysis for data discrepancies and pipeline issues
• Communicate with data providers to manage changes in data delivery
• Implement fixes and enhancements to improve data quality and pipeline performance
• Collaborate with data scientists and analysts to implement effective data solutions
• Develop strategies for data validation and quality assurance
• Document and manage data pipeline architectures and maintenance protocols
• Ensure compliance with data governance and security standards

Requirements

• Bachelor’s Degree
• Minimum one year of research computing experience
• Proficiency in Linux administration (system layout, file permissions, shell, utilities)
• Experience with Apache Airflow
• Experience with database management, specifically Postgres
• Proficiency in Python and Bash scripting
• Experience with team software development tools including Git/GitHub

Preferred Qualifications

• Experience with Apache Airflow version 3.0
• Knowledge of Docker and Docker Compose
• Experience with pandas, Flask, and PyPI publishing
• Familiarity with Elastic, Kibana, and FileBeat
• Proficiency with GitHub Actions and Jira Software

Benefits

• Comprehensive medical, prescription, dental, and vision insurance
• Generous retirement savings program with employer contributions
• Tuition benefits
• Ample paid time off and observed holidays
• Life, accidental death, and disability insurance
• Access to fitness center facilities

About the Company

Carnegie Mellon University is a private, global research university and a leading hub for research and education in artificial intelligence and machine learning. The Machine Learning Department (MLD) focuses on developing innovative algorithms and models to address complex problems in diverse fields such as robotics, healthcare, and finance.