Pooja Dhaygude

Project Name: Planned Parenthood Dashboard

• Developed and maintained AWS Glue jobs using PySpark to efficiently load and transform large volumes of patient appointment data into Amazon Redshift, ensuring timely and accurate data processing.
• Designed and implemented data pipelines to handle diverse data sources and formats such as json and parquet, optimizing performance and scalability for the organization's evolving needs.
• Collaborated closely with stakeholders to understand requirements and translate them into scalable and maintainable data engineering solutions on the AWS platform.
• Implemented data quality checks and monitoring processes to ensure the reliability and integrity of patient appointment data throughout the ETL process.

Project Name: Entity Resolution Engine

• Collaborated with stakeholders to define and refine rules for entity resolution, adapting them to suit specific business needs and data characteristics.
• Developed a rule-based entity resolution engine using PySpark, aimed at identifying and consolidating duplicate or inconsistent data to establish a reliable master data source.
• Utilized PySpark's distributed computing capabilities to efficiently process large datasets and identify potential matches or discrepancies.
• Provided guidance and support to teams on utilizing the entity resolution engine effectively and maintaining data quality standards.
• Continuously refined and optimized the engine based on performance feedback and evolving business requirements, driving ongoing improvements in data management processes.

Project Name: Travelers data model migration

• Led migration project from mainframe to AWS cloud as a data engineer.
• Designed and implemented robust data pipelines for efficient ETL processes.
• Utilized AWS services such as Athena, Glue, and Lambda for data extraction, transformation, and loading into redshift.
• Collaborated with cross-functional teams to understand legacy data structures and mitigate risks.
• Ensured data integrity and consistency throughout migration process.
• Provided guidance, training, and support to team members.
• Leveraged best practices in cloud architecture, automation, and monitoring.

Project: Travelers Production Discrepancies

• Proficient in recreating scenarios in lower regions to effectively trace and troubleshoot issues, ensuring swift resolution.
• Skilled in analyzing data flow and identifying failing code or programs in mainframe systems, enabling rapid root cause identification.
• Experienced in implementing necessary code changes or table adjustments to address issues efficiently and restore system functionality.
• Capable of preparing comprehensive test cases and executing thorough unit testing to validate code modifications, ensuring quality and reliability.
• Proficient in coordinating the migration of fixes to the production environment, minimizing downtime and maintaining operational continuity.