Lead Data Engineer Job at Reuben Cooley Inc., Cary, NC

  • Reuben Cooley Inc.
  • Cary, NC

Job Description

Programming & Data Engineering

  • Expert - level proficiency in Python, Scala, and PySpark, with a strong track record of designing and delivering production-ready, modular, and well-tested solutions; developing and troubleshooting Spark workloads; and optimizing large-scale batch and streaming data pipelines using Delta Lake and Spark technologies.
  • Strong SQL and data modelling dimensional and normalised; schema design and data contract definition.
  • Databricks expertise Delta Lake, Unity Catalog, Jobs & Workflows, cluster and pool management, performance tuning, Model Serving.
  • Azure data stack — ADLS Gen2 (zone design, ACLs, lifecycle), Azure Data Factory (parameterized / metadata-driven frameworks, error handling), Azure Event Hubs.

AI & Machine Learning

  • 3+ years designing and shipping LLM-based systems in production: RAG pipelines, agentic / tool-calling workflows, structured output, chunking and embedding strategy, vector and hybrid retrieval, and prompt engineering.
  • Evaluation discipline — golden datasets, regression suites, accuracy and hallucination tracking, human-in-the-loop feedback loops; you measure AI quality, not assert it.
  • Hands-on experience with LangChain, LlamaIndex, or LangGraph, plus at least one provider stack (Azure OpenAI, OpenAI, or Databricks Model Serving).
  • Metadata-driven thinking — schema inference, data profiling, lineage, catalogs, and configuration-driven frameworks that onboard the next source without new code.

Architecture & Governance

  • 12–18 years of total experience in data engineering, data platform delivery, or related disciplines.
  • Proven delivery of a medallion/lakehouse architecture at enterprise scale — not just familiarity with the concept.
  • Azure security and governance: Entra ID, managed identities, RBAC, POSIX ACLs on ADLS Gen2, Key Vault, private endpoints, and PII handling.
  • CI/CD and infrastructure as code: Azure DevOps, Terraform, Databricks Asset Bundles, and automated testing of data pipelines.
  • Clear technical writing and the ability to present and defend a design to both engineers and non-technical stakeholders.

Job Tags

Full time, Contract work

Similar Jobs

ProSet Construction Inc

General Contractor Construction Superintendent Job at ProSet Construction Inc

 ...Commercial Construction Project Management & General Contracting Experience Highly Desireable Proset Construction is seeking an...  ...things right. Proset Construction is a Colorado-based general contractor provoking change in the way structures are built. We deliver... 

Hyperion Solutions Emco LLC

Machinist Job at Hyperion Solutions Emco LLC

 ...Job Description Job Description Hyperion Solutions is currently seeking experienced manual machinists for our Titan Machine Corp. machine shop as we expand our team of metalworking craftsmen. This position will focus heavily on the machining, repair, and reconditioning... 

LocumTenens.com

CRNA Needed for Locum Tenens Coverage at OR in Springfield, Illinois Job at LocumTenens.com

 ...a Glance Job Reference Id: ORD-215735-CRNA-IL Title: CRNA Dates Needed: ASAP -...  ...Certification Required: Negotiable Job Duration: Locums About the Facility This hospital...  ...call coverage is required for this locum tenens assignment. Additional Job Details... 

Public

Research Assistant Job at Public

 ...Bring your 3 years of experience to a Research Assistant role that rewards initiative and fresh thinking. With 5 years of experience under your belt, you'll step into a remote position paying $77,000 - $110,000 where ownership and momentum matter. Key Responsibilities... 

Air Evac Lifeteam

Helicopter Pilot: $15K Sign-On Bonus Job at Air Evac Lifeteam

 ...you to join a team of highly skilled emergency medical experts, pilots and mechanics! Our mission of providing care to the world at...  ...operations involving Company aircraft are in compliance with Federal Aviation Regulations, Air Evac EMS Operations Manual, and all state and...