Data Engineer

About the Company

We build infrastructure that delivers massive amounts of web data to the companies training the world’s most powerful AI models. We power and support Grass, a bandwidth-sharing network that operates a massive distributed crawler, providing unique access to high-quality public web data at global scale.

Responsibilities

• Design, build, and optimize scalable data pipelines for real-time or batch processing
• Develop and manage ETL/ELT workflows to transform raw data into structured formats
• Integrate and configure database infrastructure for performance, scalability, and security
• Automate data workflows and infrastructure setup using tools like Apache Airflow or Terraform
• Collaborate with data scientists and analysts to ensure data accessibility
• Monitor, troubleshoot, and improve the performance of data pipelines and infrastructure
• Manage cloud infrastructure (AWS, GCP, Azure) for databases, storage, and compute
• Implement best practices for data governance, security, and disaster recovery

Requirements

• Bachelor’s degree in Computer Science, Information Systems, or a related technical field
• Extensive experience with database systems like Redshift or Snowflake
• Advanced proficiency in SQL and query optimization
• Hands-on experience with data pipeline tools such as Apache Airflow or AWS Glue
• Solid understanding of ETL/ELT processes and data integration best practices
• Experience with infrastructure automation tools like Terraform or CloudFormation
• Proficiency in Python, Scala, or Java
• Familiarity with containerization (Docker) and orchestration (Kubernetes)
• Strong analytical and problem-solving skills