BrightDrop - Senior Data Engineer – Delivery Intelligence Group

Descripción

As a Senior Data Engineer within our Delivery Intelligence Group, you will be responsible for designing, developing, and maintaining efficient data workflows and associated cloud infrastructure that support BrightDrop’s core analytic products and services. You will work closely with our data scientists, DevOps, and software engineers to automate pipelines that transform data in the BrightDrop data lakehouse into deployable data models that power our automated fleet insights and visualizations as well as emerging machine learning, optimization, and AI applications. The ideal candidate will have experience standing up ELT pipelines to handle massive volumes of spatiotemporal data, designing enterprise data warehouse models, implementing robust data quality tests, and using data pipeline-as-code integration systems to ensure the smooth and efficient orchestration of the workflows powering our analytics platform.

Responsibilities:

• Architect, implement, and maintain ELT pipelines in our cloud-native data lakehouse platform.

• Work with data scientists to turn exploratory analyses into production data transformation workflows within our multihop data lakehouse architecture.

• Support enterprise-wide use of BI tools and assist in developing internal and external analytics products (e.g., dashboards).

• Generate and deploy feature engineering pipelines to power internal and customer-facing machine learning-based products.

• Maintain data quality testing and monitoring tools.

• Work with information security, DevOps, and DataOps to maintain data classification, auditing, and access and cost control policies.

Additional Description

Required:

• Bachelors degree (Master’s preferred) or equivalent experience in computer science, data science, engineering, or related quantitative field

• 5+ years of industry experience developing, implementing, and maintaining solutions for Big Data or data warehousing systems

• 3+ years of industry experience working in a cloud environment (Azure preferred)

• 3+ years experience working with SQL query authoring for automated data transformation (familiarity with dbt preferred, but not required).

• 2+ years of experience developing streaming data loading pipelines (use of Spark/pySpark preferred)

• Basic understanding of machine learning/statistical learning principles

• Experience implementing and maintaining data workflow orchestration and integration tools (e.g.  Airflow/Astro, Prefect, dbt cloud, etc.)

• Understanding of and experience with application of data quality tools integrated with CI/CD automation frameworks in functional deployment environments (e.g., Github Actions/Azure DevOps pipelines).

• Familiarity with data quality testing frameworks (Great Expectations, Deequ)

• Self-driven with an interest in on-the-job learning.

Preferred:

• Familiarity with enterprise warehouse data modeling techniques (e.g., Kimball)

• Experience integrating simulation systems with distributed, data-intensive processing or analytics applications

Desired:

• Working familiarity with terraform

• Domain knowledge in transportation and energy systems, graph algorithms, convex optimization, and/or reinforcement learning

• Engagement with modern data stack community (open source and commercial)

• Passion for transportation decarbonization

• High degree of attention to software craftsmanship and professionalism

• Experience working with containerization technologies and orchestration platforms (specifically Docker and Kubernetes)

#LI - Hybrid

The expected base compensation for this role is: ($106,460 - $161,240 USD Annual). Actual base compensation within the identified range will vary based on factors relevant to the position.

Bonus Potential: An incentive pay program offers payouts based on company performance, job level, and individual performance.

Benefits: GM offers a variety of health and wellbeing benefit programs. Benefit options include medical, dental, vision, Health Savings Account, Flexible Spending Accounts, retirement savings plan, sickness and accident benefits, life insurance, paid vacation & holidays, tuition assistance programs, employee assistance program, GM vehicle discounts and more.

Como contratamos

Este es un ejemplo de un contenido de trabajo común que se puede mostrar en la parte inferior de cada descripción de trabajo. Se agrega en el CMS y luego se muestra en cada trabajo. Se puede utilizar para complementar el contenido del trabajo que proviene del ATS.

Egestas faucibus lacus a ac aptent ac condimentum risus iaculis a parturent a enim suscipit semper hendrerit feugiat suspendisse lobortis facilisis vel at dolor ornare rutrum a elementum mi. Rhoncus mollis curae penatibus scelerisque suspendisse faucibus phasellus porttitor maecenas amet a amet hac facilisi a urna a vestibulum vestibulum maecenas per adipiscing ultrices.