Shubham Memane
Pune
Summary
Data Engineer with 5+ years of experience in designing, building, and optimizing scalable ETL pipelines and big data solutions. Proficient in PySpark, Databricks, Snowflake, and AWS. Strong expertise in data modeling, data warehousing and cloud-based data engineering solutions. Adept at improving data processing performance and ensuring high data quality.
Overview
6
6
years of professional experience
1
1
Certification
Work History
Consultant ( Data Engineering )
Deloitte-USI
Pune, India
02.2024 - Current
- Designed and implemented ETL pipelines in Databricks & PySpark to process terabytes of structured and semi-structured data for health-care domain.
- Developed dashboarding solution by processing ~17mil user's data using Databricks Pyspark, Snowflake and Tableau.
- Developed and optimized Snowflake-based data warehousing solutions, reducing query execution time by 40%.
- Used Autosys for job scheduling and monitoring. Databricks for performance enhancement, log creation and maintenance.
- Enhanced data integration strategies, effectively consolidating data sources by 15%, which streamlined reporting functions.
- All solutions implemented by maintaining the data integrity and security.
Product Engineer ( Data Science & Engineering )
LTI-Mindtree
Pune, India
07.2022 - 02.2024
- Analytical feature development using Pyspark, Python, Machine learning and monitoring tool like Airflow.
- Build Machine learning models for key driver analysis and test the accuracy of different models and deploy.
- Implemented feature for Anomaly detection in time series data using Pyspark and ML approach.
Assistant Manager ( Data Engineering )
TATA-insights and quants
Remote
07.2021 - 07.2022
- Develop dashboarding solution for medical insurance domain by creating data pipeline using Pyspark and SparkSQL.
- Pyspark pipeline used for fetching data stored in sql database and perform analysis to calculate KRA's on data and store result back to database.
Product Engineer ( Data Engineering )
C-DAC
Hyderabad, India
03.2020 - 06.2021
- Create a spark (Using Pyspark) pipeline for fetching data stored in hdfs, perform analysis on data and store result back to database.
- Worked on ML and DL libraries like nltk, spacy, sklearn etc. to build text classification.
Education
PG Diploma - Big Data Analytics
CDAC
Pune01.2020
BE - Computer Engineering
Pune University
Pune01.2018
HSC - Science
Bhonsala Military College
Nashik, India01.2014
SSC - Science
Adarsh Madhyamik Vidyalaya
Nashik01.2012
Skills
- Databricks
- Snowflake
- Pyspark
- Python
- PostgreSQL
- MySQL
- AWS
- Azure
- CI/CD (Jenkins, GitHub)
- Team Management
Certification
Databricks data engineering associate
Awards Activities
- Spot award FY-24 for best process implementation and management
- Best Snowflake implementation team award FY-24
Timeline
Consultant ( Data Engineering )
Deloitte-USI
02.2024 - Current
Product Engineer ( Data Science & Engineering )
LTI-Mindtree
07.2022 - 02.2024
Assistant Manager ( Data Engineering )
TATA-insights and quants
07.2021 - 07.2022
Product Engineer ( Data Engineering )
C-DAC
03.2020 - 06.2021
BE - Computer Engineering
Pune University
PG Diploma - Big Data Analytics
CDAC
HSC - Science
Bhonsala Military College
SSC - Science
Adarsh Madhyamik Vidyalaya