ADITYA KHOBRAGADE
Pune
Summary
Lead Data Engineer with 10+ years of experience designing, developing, and optimizing scalable cloud-native data platforms, ETL/ELT pipelines, and distributed data processing solutions. Expertise in Databricks, Snowflake, PySpark, Apache Spark, Python, SQL, BigQuery, Kafka, Airflow, Hive, AWS, and GCP, delivering high-performance batch and real-time data pipelines for multi-terabyte datasets. Proven success in modernizing legacy data ecosystems, building enterprise data lakes and warehouses, and implementing cloud solutions that reduced infrastructure costs by 30% and improved pipeline performance by 40%. Strong experience in data architecture, business intelligence, cloud migration, data governance, CI/CD, DevOps, and performance optimization. Skilled at leading cross-functional engineering teams, mentoring developers, collaborating with stakeholders, and delivering secure, scalable, and reliable data solutions that drive business value.
Overview
10
10
years of professional experience
1
1
Certification
Work History
Lead - Technology
Synechron Technologies
Pune
05.2024 - Current
- Project: HSBC – Event Streaming Platform (ESP)
- Situation: The organization required a high-throughput, multi-region streaming platform to process high volumes of financial market messages securely without data drops.
- Engineered a scalable Source -> Kafka Cluster -> Pub/Sub -> Dataflow -> BigQuery streaming architecture. Developed core Tagging and Rules Engine components in Python to classify incoming events and implemented enrichment logic utilizing STM mappings before data warehouse loading.
- Engineered end-to-end real-time ingestion and transformation data pipelines, enhancing reliability and delivery efficiency.
- Successfully stabilized a high-throughput streaming ecosystem across multiple global regions, utilizing Google Cloud Monitoring and Logging to eliminate bottlenecks and handle heavy real-time traffic proactively.
- Project: HSBC – Globallinks
- Situation: Critical financial data was confined to localized Microsoft SQL Server databases, limiting advanced data analytics and unified business reporting.
- Built and automated an end-to-end data ingestion pipeline using Apache Sqoop, GCS, Apache Spark, and BigQuery. Developed custom Python scripts and Spark transformations to automate schema conversion, and orchestrated weekly incremental ingestion tables using Cloud Composer.
- Migrated large-scale financial datasets to Google BigQuery on GCP, automating mapping workflows to enable advanced data analytics across multiple financial markets.
- Streamlined legacy warehouse migration across multiple financial markets, enabling data integration into Looker Studio dashboards for automated, actionable business insights.
- Led cross-functional teams to deliver technology solutions for clients.
- Developed software architecture and design specifications for various projects.
- Coordinated with stakeholders to gather requirements and define project scope.
Senior Data Engineer
Quantiphi Analytics Pvt. Ltd.
Bangalore
10.2020 - 05.2024
- Project: HCA Healthcare
- Situation: Critical clinical datasets were locked across disparate legacy Epic and Cerner EHR architectures, creating bottlenecks for distributed medical data analysis.
- Migrated healthcare datasets from RDBMS and FTP-based flat files into GCP BigQuery, ensuring data quality through cleaning and validation.
- Designed data ingestion pipelines using Apache Beam and Dataflow. Developed custom Python-based Spark applications to clean and parse raw CSV/TSV datasets, and structured downstream analytics inside BigQuery using optimized data partitioning and clustering strategies.
- Improved ETL performance for large medical datasets and orchestrated automated retries using Airflow (Composer) to ensure data availability.
- Project: Vivint
- Situation: The client’s existing framework struggled to process an influx of 2+ TB of daily log data, causing severe analytical delays and bloated infrastructure costs.
- Re-engineer a scalable big data processing pipeline on GCP to ingest data from various RDBMS, API, and file sources, transforming raw data into structured analytics formats.
- Deployed a high-performance processing pipeline using PySpark on Dataproc clusters. Migrated legacy NiFi pipelines into Cloud Composer (Airflow) DAGs, implemented schema evolution via Avro, engineered Type 2 data handling, and optimized cluster configurations by integrating spot instances.
- Reduced daily log processing runtime by 40% and slashed cloud infrastructure costs by 30% while establishing a highly resilient, automated ETL flow loaded into BigQuery.
- Project: So-Energy
- Situation: Customer behavioral and energy usage indicators were siloed across Relational Databases (Oracle, SQL Server) and Freshdesk APIs.
- Consolidate and migrate historical data from 150 legacy MS-SQL tables into GCP BigQuery to drive machine learning and business analytics.
- Formulated an ingestion framework incorporating MS-SQL -> Sqoop -> GCS -> Spark -> BigQuery. Developed incremental weekly Sqoop jobs, automated schema conversions using Python scripts, and scheduled end-to-end ML and ingestion pipelines using Cloud Composer.
- Reduced end-to-end processing time, delivering a unified data platform for collaboration with ML teams on deep energy insight extraction.
Senior Data Engineer
Pluto7
Bangalore
07.2019 - 10.2020
- Project: BharatiAXA, TVS (Data Migration)
- Situation: E-commerce and operational systems required unified historical data structures to prepare for analytical models and secure recommendation systems.
- Led data engineering team to design CDC-based migration paths from MS SQL Server to BigQuery, automating ingestion from secure external APIs.
- Architected ETL pipelines on Google Cloud using BigQuery, Cloud Functions, and Dataproc to streamline data processing. Built custom Airflow HTTP operators to securely consume token-based APIs for automated customer profiles and order datasets, and developed an OpenCV and Google Cloud Vision API PoC for document text extraction.
- Centralized multi-source data streams achieving schema alignment and high data accuracy, eliminating manual document extraction cycles through vision pipeline.
Big Data Engineer + Intern
Katerra Technology
Bangalore
07.2016 - 05.2019
- Engineered geospatial pipelines using Python, AWS S3, RDS, and Lambda for streamlined data processing.
- Built automated workflows to ingest and serve clean geospatial insights to executive dashboards.
- Optimized parcel processing by applying Python multiprocessing for enhanced spatial metrics calculations.
- Deployed high-performance Flask REST APIs to deliver data to Mapbox and ArcGIS frontends.
- Accelerated transformation speeds of geospatial files, enabling instant proximity analysis on property dashboards.
Education
Master of Technology (M.Tech) - Computer Science (Specialized in Big Data Analytics)
VIT University
Chennai, India05-2017
Bachelor of Engineering (B.E.) - Computer Science
PIET, Nagpur University
Nagpur, India07-2014
Skills
- Google Cloud Platform (GCP): BigQuery, Dataflow, Dataproc, Pub/Sub, Cloud Function/Run, Composer (Airflow), GCS
- Amazon Web Services (AWS): S3, EMR, Lambda, CloudWatch, RDS, Redshift, Nifi
- Databricks
- Apache Beam
- Apache Spark
- Apache Sqoop
- Python
- SQL
- GitHub Actions
- Terraform
- Docker and Kubernetes
- File Formats: Parquet, Avro, JSON, CSV/TSV, Delta Lake
Certification
- Google Cloud Certified: Professional Cloud Architect
- Google Cloud Certified: Professional Data Engineer
Additional Details
English, Hindi, Marathi, Married
Timeline
Lead - Technology
Synechron Technologies
05.2024 - Current
Senior Data Engineer
Quantiphi Analytics Pvt. Ltd.
10.2020 - 05.2024
Senior Data Engineer
Pluto7
07.2019 - 10.2020
Big Data Engineer + Intern
Katerra Technology
07.2016 - 05.2019
Master of Technology (M.Tech) - Computer Science (Specialized in Big Data Analytics)
VIT University
Bachelor of Engineering (B.E.) - Computer Science
PIET, Nagpur University