Abhishek Parmar
Pune
Summary
Motivated and result-driven Data Engineer with a strong background in designing and optimizing data pipelines for high-performance processing in the payments industry. Leveraging technologies like Spark, Apache Hudi, and AWS to ensure data integrity and drive operational efficiencies. Dedicated team player with a passion for innovation and commitment to contributing significantly to business success.
Overview
3
3
years of professional experience
1
1
Certification
Work History
Data Engineer
NomuPay
09.2022 - Current
- Part of Data warehouse team at a digital payments firm which specializes in cross border and multi currency payments
- Led the development of a cross-border forex payment functionality that enabled NomuPay to process payments and disbursements in multiple currencies for clients and third-party ISOs, resulting in significant increase in the customer base and profits in Southeast Asia driven by forex fees and expanded market reach
- Executed a critical design change in the transaction processing pipeline that reduced the overall processing time by 10%
- It also enabled on time payouts by meeting SLAs, improved efficiency and quality of the pipeline, and reduced costs significantly
- Engineered a comprehensive data monitoring framework using Spark and Apache Airflow, delivering crucial insights into data ingestion and downstream pipeline processes, significantly enhancing data visibility and operational efficiency
- Managed transactional data integrity by handling diverse data streams from various payment processors, utilizing Apache Hudi as the backbone for reliable and consistent data management
- Developed and optimized data pipelines for fraud detection, leveraging Visa data to analyze and decode fraud patterns
- This initiative empowered merchants to reduce operational costs by proactively identifying and mitigating fraudulent transactions
- Designed and implemented multiple reporting pipelines using a robust combination of technologies, including Spark, Airflow, HUDI, AWS Lambda, Kinesis, and SQS
- These pipelines provided detailed reports that enabled the management to analyze merchant balances and make informed decisions, leading to increased company profits
- Created a localized testing framework for payments data, facilitating unit testing in a controlled environment
- Additionally, developed an end-to-end pre-production data pipeline that enabled the team to verify core business logic before deployment, ensuring seamless and reliable production releases
- Gained deep expertise in payments processing and digital payments technologies, with hands-on experience in Spark, Apache HUDI, and a range of AWS services.
Data Engineer
Barclays
08.2021 - 08.2022
- Part of Data Engineering Team in Next-Gen Consumer Banking Platform
- Engineered 10+ high-performance real-time ETL pipelines using Spark Structured Streaming and AWS services, transforming data processing by enabling seamless, real-time data upserts to the data lake, which significantly enhanced data accessibility to downstream consumers
- Built a highly scalable and adaptive end-to-end outbound data transfer framework, capable of transferring terabyte-level of data monthly
- This framework, orchestrated using AWS CloudWatch and triggers, streamlined massive data transfers with increased efficiency and reliability
- Conducted in-depth research on Spark and HUDI configurations to tackle complex challenges, such as merging and evolving schemas
- Created frameworks for extensive unit testing, load testing, and schema management.
Education
Information Technology -
K.J.Somaiya College Of Engineering
06.2021
Skills
- Apache Spark
- Apache HUDI
- AWS
- Python
- Data Warehouse
- SQL
- Data Analysis
- Apache Hadoop
- Problem-Solving
- Spark performance tuning
- Jenkins
- Data Warehousing
- Data Modeling
- ETL development
- Database Design
Certification
Google Digital Marketing, 2021
Publications
IEEE paper on using blockchain based voting system, 2021
Hobbies
Swimming and Trekking
Timeline
Data Engineer
NomuPay
09.2022 - Current
Data Engineer
Barclays
08.2021 - 08.2022
Information Technology -
K.J.Somaiya College Of Engineering
Similar Profiles
Suman GiriSuman Giri
Software Developer at Agile Soft techSoftware Developer at Agile Soft tech
Ved ShuklaVed Shukla
Senior System Engineer at InfosysSenior System Engineer at Infosys
Mohil Pushpraj DhadgeMohil Pushpraj Dhadge
Senior DevOps Engineer at WiproSenior DevOps Engineer at Wipro