Senior Data Engineer

Responsibilities

• Design and implement a multi-zone enterprise data lake on Amazon S3
• Build batch and streaming data ingestion pipelines using AWS Glue and Amazon Kinesis
• Write PySpark and Python ETL code for AWS Glue jobs
• Implement data transformation and orchestration using AWS Step Functions
• Implement AWS Lake Formation for fine-grained data governance
• Configure Amazon Athena for serverless SQL querying and Amazon DynamoDB for lookups
• Develop and deploy AWS Lambda functions using AWS Lambda Powertools
• Write and maintain Terraform or CloudFormation modules for infrastructure as code
• Integrate CI/CD pipelines using GitHub Actions
• Support Azure Data Lake migration via AWS DataSync
• Design entity resolution pipelines for customer record deduplication
• Build data models to support Customer 360 views and Amazon QuickSight dashboards

Requirements

• 5+ years of data engineering experience
• 2+ years of experience in AWS cloud environments
• Strong proficiency in Python and SQL
• Hands-on experience with PySpark or Scala
• Experience with AWS Glue, Step Functions, Lambda, and Amazon S3
• Proficiency with AWS Lambda Powertools
• Working knowledge of Apache Iceberg
• Experience with Terraform, CloudFormation, or AWS CDK
• Experience with AWS Lake Formation
• Understanding of DynamoDB data modeling
• Experience with Amazon Athena performance tuning
• Experience with GitHub Actions or comparable CI/CD tooling

Preferred Qualifications

• Experience with Azure Data Lake Storage (ADLS)
• Familiarity with AWS Entity Resolution service
• Exposure to Amazon Bedrock or Amazon SageMaker
• Knowledge of Amazon QuickSight
• AWS Certification (Data Analytics, Database, or Solutions Architect)
• Background in real estate or property management data

About the Company

Capnexus is a comprehensive services provider specializing in designing, building, and supporting retail software. They operate as a build-as-a-service provider with a culture focused on outcomes, delivery, and continuous learning.