Data Engineer Consultant

Working as a Data Engineer Consultant, you will:

• Design and implement data ingestion pipelines connecting external APIs and business systems with the Databricks data platform.

• Develop reliable processes for retrieving and processing historical data, including mechanisms for rate limiting, checkpointing, retries and safe reprocessing.

• Build and maintain the Bronze layer of the data platform, covering data standardisation, typing, historical tracking and document storage.

• Develop processes for linking digital documents with corresponding business records while ensuring data consistency and idempotent processing.

• Use Databricks capabilities to classify documents and extract structured information from PDFs and images, including confidence scoring and workflows for manual review.

• Contribute to the development of the Silver layer using appropriate entity modelling, CDC and incremental processing patterns.

• Implement deterministic entity resolution and source-priority logic across multiple data sources.

• Apply data quality controls within the existing framework and ensure that quality results are properly stored, monitored and available for governance purposes.

• Support the development of analytical and machine-learning data products, including feature and metric stores and batch scoring workflows.

• Work with MLflow and related processes for model registration and promotion where required.

• Improve automation, monitoring and reliability across data pipelines and contribute to CI/CD processes.

• Collaborate with other technical stakeholders to deliver agreed project milestones within a defined consulting scope.

Chabre IT Services is a global professional IT services provider, building long-lasting relationships with Enterprises. We specialize in the delivery of tailor-made solutions,  smart outsourcing, try&hire, and success fee services. We are a smart IT boutique with unique knowledge, which will deliver your ideas into reality.

Our client is a global technology services organisation delivering data and digital transformation projects for enterprise customers. The project focuses on building and enhancing a modern data platform based on Databricks, with a strong emphasis on data engineering, governance, automation and analytics.

The successful candidate will join a project involving multiple data sources, API integrations, document processing and layered data architecture. The environment combines traditional data engineering with modern Databricks AI capabilities and machine-learning workflows, providing an opportunity to work across several stages of the data lifecycle.

• Recent, hands-on experience delivering production solutions with Databricks.
• Strong knowledge of PySpark, Delta Lake and Unity Catalog.
• Practical experience with the Databricks medallion architecture and production data pipelines.
• Experience with Spark Declarative Pipelines / Delta Live Tables and Databricks Workflows.
• Very good Python and advanced SQL skills.
• Experience integrating REST APIs into data engineering solutions, including authentication, pagination, rate limiting and incremental or checkpointed data retrieval.
• Strong understanding of data modelling, including SCD Type 2 and current-state / CDC patterns.
• Experience designing data quality controls, reconciliation processes and mechanisms for handling invalid or quarantined data.
• Knowledge of Git-based development and CI/CD for data platforms, preferably with Azure DevOps or GitHub Actions.
• Experience implementing monitoring and observability for production data pipelines.
• Ability to work within a fixed project scope, estimate work, meet milestones and take ownership of agreed deliverables.
• Strong communication skills and the ability to work independently in a consulting environment.
• Excellent English and Polish skills (min. B2 level).

• Rate up to 240,00 PLN/h + VAT
• Remote work
• Subsidy for peripherals in the amount of 500,00zł
• Working tool (MacBook Pro or Lenovo Legion 5)
• Co-financing of courses related to the position
• Benefits: MultiSport, Medicover