Minal Das

● Designed multiple Datafusion pipelines to improve Business output.

● Pipelines created leveraging composer Airflow DAGs.

● Designed Automation script to enable disable DAGs.

Creating DAGS to load raw data in BQ.

• Design, build and large scale enterprise data solutions and applications using one or more of GCP data and analytics services in combination with Composer, Dataflow, Dataproc and Big Query
• Developing DAG in python leveraging Airflow to orchestrate
• Customized java framework for real-time Ingestion.

• Design, build and large scale enterprise data solutions and applications using one or more of AWS data and analytics services in combination with 3rd parties - Spark, EMR, DynamoDB, EC2, Parameter Store
• Design and build production data pipelines from ingestion to consumption within big data architecture using Scala programming language
• Proficient in AWS data processing services (S3, Glue, Athena)
• Experience in creating and automating ETL pipelines via shell scripts.

• Design and develop data pipelines to move data into various layers in big data ecosystem
• Develop Hive scripts on the data and load to target systems for use by the data analysts for reporting
• Performance tuning of the Data pipelines to meet the desired outcome
• Troubleshoot and debug issues independently
• Understanding, gathering and analyzing the client's problem statement
• Monitor daily execution, diagnose and log issues, and fix business critical pipelines to ensure SLAs are met with stakeholders
• Experience with Linux command line, version control software (Git).

• Develop and implement data pipelines that extracts, transforms and loads data into an information product that helps to inform the organization in reaching strategic goals
• Data will be stored in Hadoop file system and processed using Spark Scala
• Involved extensively with Sqoop for importing metadata from RDBMS into the staging Layer
• Perform analysis of vast data stores and uncover insights.