Senior/Lead Data Engineer – AI-Native Aftermarket Platform

Required Qualifications

• Expertise in SQL and dimensional modeling methodologies, including medallion architecture, SCDs, and grain management.
• Proven ability to design idempotent pipelines utilizing incremental, checkpoint, and replaceWhere strategies.
• Extensive experience with production-grade Python engineering, including type hints, pytest, and ruff.
• Strong capability to diagnose and resolve failing Spark / PySpark jobs utilizing tools like Spark UI.
• Deep understanding of Delta Lake features such as MERGE, OPTIMIZE, Z-ORDER, and time travel.
• Hands-on expertise with dbt, including models, tests, and exposures.
• Experience authoring and deploying jobs using Databricks Asset Bundles (DAB) and operating within a Unity Catalog environment.
• Commitment to data quality via pre-write asserts, schema checks, and maintaining dbt relationship and uniqueness tests.
• Strong adherence to disciplined Git workflows, conventional commits, and strict documentation practices.
• Experience provisioning and utilizing Service Principals, GitHub environment secrets, and secret management tools like Azure Key Vault or Databricks secret scopes.
• Strong written technical communication skills for PR descriptions and runbooks, with the ability to translate pipeline work into business metrics.
• Proven decision-making abilities to navigate ambiguity and balance trade-offs between cost, latency, and reliability.
• Experience leading technical initiatives, establishing architectural standards, and contributing to interview rubrics is preferred.
• Experience reading or modifying Azure Data Factory (ADF) pipelines and familiarity with Azure Data Lake storage is highly preferred.
• Familiarity with dbt observability tools, such as Elementary, is a plus.
• Awareness of PII detection and masking best practices is preferred.
• Experience with multi-tenant configuration patterns to onboard new tenants with zero code changes is a strong plus.
• Proficiency in reading and editing GitHub Actions workflows for Databricks deployment is preferred.
• Ability to make cost-aware compute decisions, selecting the appropriate cluster shape per workload, is a plus.
• Proficiency in AI-assisted development tools like Claude Code for daily work and code review is preferred.
• Experience writing incident post-mortems and coordinating feature handovers with Data Science teams is a plus.
• Expertise in SQL and dimensional modeling methodologies, including medallion architecture, SCDs, and grain management.
• Proven ability to design idempotent pipelines utilizing incremental, checkpoint, and replaceWhere strategies.
• Extensive experience with production-grade Python engineering, including type hints, pytest, and ruff.
• Strong capability to diagnose and resolve failing Spark / PySpark jobs utilizing tools like Spark UI.
• Deep understanding of Delta Lake features such as MERGE, OPTIMIZE, Z-ORDER, and time travel.
• Hands-on expertise with dbt, including models, tests, and exposures.
• Experience authoring and deploying jobs using Databricks Asset Bundles (DAB) and operating within a Unity Catalog environment.
• Commitment to data quality via pre-write asserts, schema checks, and maintaining dbt relationship and uniqueness tests.
• Strong adherence to disciplined Git workflows, conventional commits, and strict documentation practices.
• Experience provisioning and utilizing Service Principals, GitHub environment secrets, and secret management tools like Azure Key Vault or Databricks secret scopes.
• Strong written technical communication skills for PR descriptions and runbooks, with the ability to translate pipeline work into business metrics.
• Proven decision-making abilities to navigate ambiguity and balance trade-offs between cost, latency, and reliability.
• Experience leading technical initiatives, establishing architectural standards, and contributing to interview rubrics is preferred.
• Experience reading or modifying Azure Data Factory (ADF) pipelines and familiarity with Azure Data Lake storage is highly preferred.
• Familiarity with dbt observability tools, such as Elementary, is a plus.
• Awareness of PII detection and masking best practices is preferred.
• Experience with multi-tenant configuration patterns to onboard new tenants with zero code changes is a strong plus.
• Proficiency in reading and editing GitHub Actions workflows for Databricks deployment is preferred.
• Ability to make cost-aware compute decisions, selecting the appropriate cluster shape per workload, is a plus.
• Proficiency in AI-assisted development tools like Claude Code for daily work and code review is preferred.
• Experience writing incident post-mortems and coordinating feature handovers with Data Science teams is a plus.