Ankush Wanjari

Ubnatech

Pune

03.2022 - Current

Designed and developed an end-to-end ETL pipeline on AWS to process and analyze data in the healthcare domain.
Utilized AWS Glue Studio (Visual ETL) to create scalable data pipelines for ingesting raw datasets from Amazon S3, performing data cleaning, transformation, and schema standardization.
Implemented PySpark-based transformations within Glue jobs for efficient distributed processing, including dropping duplicates, handling missing values, renaming columns, and deriving features for analytics.
Employed AWS Glue Crawler to catalog data and integrate it into the AWS Glue Data Catalog, enabling seamless querying using Amazon Athena.
Secured data pipelines using IAM roles and KMS encryption policies to manage access and ensure compliance with data security standards.
Automated ETL workflows by integrating AWS Lambda functions to trigger Glue jobs upon S3 data arrival, and orchestrated end-to-end data pipelines using AWS Step Functions.
Loaded curated and processed datasets into Amazon Redshift for advanced analytical querying, reporting, and integration with downstream BI tools.
Gained experience in schema evolution management, data partitioning strategies, and optimizing ETL jobs for cost-effective performance on AWS.
Documented pipeline architecture, data flow, and cleaning logic to ensure maintainability and reproducibility of the data engineering process.

Similar Profiles