Endava Senior Data Engineer \\\| SmartRecruiters Google Chrome Microsoft Edge Apple Safari Mozilla Firefox . Senior Data Engineer Full-time Company Description Technology is our how. And people are our why. For over two decades, we have been harnessing technology to drive meaningful change. By combining world-class engineering, industry expertise and a people-centric mindset, we consult and partner with leading brands from various industries to create dynamic platforms and intelligent digital experiences that drive innovation and transform businesses. From prototype to real-world impact - be part of a global shift by doing work that matters. Job Description Lead onshore technical engineers Act as the lead engineer for each domain Collaborate with Solution Architects to ensure understanding needed for engineering level guidance to nearshore teams Translate architecture design to engineering level guidance Own specifying architectural designs into sprint-ready engineering tasks for assigned domain Ensure delivery to the solution architecture within domain Answer design questions for sprint teams Participate in backlog refinement sessions to align team with the architecture Act as the internal technical expert supporting Product design and Engineering teams on demonstrations and technical discussions. Create/Update documentation such as solution design patterns and engineering standards. Lead technical discussions around APIs, integrations, data flow, compliance and security. Serve as a trusted advisor on best practices, technical implementation, and scalability. Design and implement highly performant and scalable ETL/ELT data pipelines using Databricks (Spark) and orchestration tools. Develop data ingestion, transformation, and enrichment workflows using Scala, Python, and advanced SQL. Support real-time and batch processing needs with optimized workflows, adhering to best practices for scalability and maintainability. Leverage Databricks for developing and managing Spark-based data pipelines and analytics workloads. Implement Unity Catalog for centralized data governance, access control, lineage, and auditability. Manage Delta Lake schemas and optimize data lake performance using Z-ordering, compaction, and partitioning strategies. Knowledge about testing and validation frameworks into pipelines using tools such as Great Expectations or custom frameworks. Monitor data flows for anomalies, failures, and integrity issues and proactively troubleshoot and resolve them. Knowledge about automated deployment pipelines for data workflows using Azure DevOps, GitHub Actions, or similar. Apply version control and collaborative development practices for notebooks, pipelines, and data models. Partner with Data Scientists, BI teams, and Product stakeholders to understand data needs and translate them into scalable solutions. Mentor junior engineers and help standardize practices across the data engineering team. Qualifications Required 5+ years of professional experience in data engineering roles. Experience with Azure Cloud and some hands-on experience with cloud services (such as ADF, Data Lake, Key Vault, etc.) Hands-on expertise with: - Databricks (including Spark tuning and cluster management) - Scala, Python, and SQL - Azure SQL, Cosmos DB, PostgreSQL Knowledge of databases (SQL and/or NoSQL) and data modeling. Proven experience building and managing data pipelines in production. Strong understanding of Data Lakehouse, Delta Lake, and medallion architecture. Experience with data governance, access control, and lineage using Unity Catalog. Must be able to work onsite 2-3 days a week Desired: Pharmacy Business domain experience Exposure to DevOps, CI/CD pipelines, and containerization (Docker, Kubernetes Familiarity with: - dbt for transformations and documentation. - Azure Event Hubs or Kafka for streaming data ingestion. - Airflow, Synapse, or Power BI for orchestration and reporting integration. - Experience with Infrastructure-as-Code tools like Terraform or Bicep for provisioning Azure data resources. - Exposure to data observability platforms (e.g., Monte Carlo, OpenLineage). - Knowledge of data mesh and distributed data ownership models. Additional Information Discover some of the global benefits that empower our people to become the best version of themselves: Finance: Competitive salary package, share plan, company performance bonuses, value-based recognition awards, referral bonus; Career Development: Career coaching, global career opportunities, non-linear career paths, internal development programmes for management and technical leadership; Learning Opportunities: Complex projects, rotations, internal tech communities, training, certifications, coaching, online learning platforms subscriptions, pass-it-on sessions, workshops, conferences; Work-Life Balance: Hybrid work and flexible working hours, employee assistance programme; Health: Global internal wellbeing programme, access to wellbeing apps; Community: Global internal tech communities, hobby clubs and interest groups, inclusion and diversity programmes, events and celebrations. Additional Employee Requirements Participation in both internal meetings and external meetings via video calls, as necessary. Ability to go into corporate or client offices to work onsite, as necessary. Prolonged periods of remaining stationary at a desk and working on a computer, as necessary. Ability to bend, kneel, crouch, and reach overhead, as necessary. Hand-eye coordination necessary to operate computers and various pieces of office equipment, as necessary. Vision abilities including close vision, toleration of fluorescent lighting, and adjusting focus, as necessary. For positions that require business travel and/or event attendance, ability to lift 25 lbs, as necessary. For positions that require business travel and/or event attendance, a valid drivers license and acceptable driving record are required, as driving is an essential job function. \\*If requested, reasonable accommodations will be made to enable employees requiring accommodations to perform the essential functions of their jobs, absent undue hardship. USA Benefits (Full time roles only, does not apply to contractor positions) Robust healthcare and benefits including Medical, Dental, vision, Disability coverage, and various other benefit options Flexible Spending Accounts (Medical, Transit, and Dependent Care) Employer Paid Life Insurance and AD&D Coverages Health Savings account paired with our low-cost High Deductible Medical Plan 401(k) Safe Harbor Retirement plan with employer match with immediately vest At Endava, were committed to creating an open, inclusive, and respectful environment where everyone feels safe, valued, and empowered to be their best. We welcome applications from people of all backgrounds, experiences, and perspectivesbecause we know that inclusive teams help us deliver smarter, more innovative solutions for our customers. Hiring decisions are based on merit, skills, qualifications, and potential. If you need adjustments or support during the recruitment process, please let us know. I'm interested I'm interested I'm interested Refer a friend share this job Share on LinkedIn Share on Facebook Share on Twitter Share via email Share on Xing Share on WeChat Share to WeChat × Copy the link and open WeChat to share. Copy to clipboard Open WeChat Share to WeChat × Use Scan QR Code in WeChat and click ··· to share. Senior Data Engineer Deerfield, IL, USA Full-time I'm interested I'm interested

The role requires a minimum of 5 years of professional experience in data engineering. Candidates must have experience with Azure Cloud and its services, including ADF, Data Lake, and Key Vault. Hands-on expertise is needed with Databricks, Spark tuning, cluster management, Scala, Python, SQL, Azure SQL, Cosmos DB, and PostgreSQL. A strong understanding of databases (SQL/NoSQL), data modeling, Data Lakehouse, Delta Lake, and medallion architecture is essential. Experience with data governance, access control, and lineage using Unity Catalog is also required.