Maya Bangar

Big Data Developer with 3+ years of experience designing, building, and optimizing cloud-based data pipelines using Apache Spark (Scala, PySpark, Spark SQL) on AWS and Azure platforms. Strong expertise in data lake and lakehouse architectures using Amazon S3, Delta Lake, and Parquet, with hands-on experience in ETL pipelines, performance optimization, and workflow orchestration using Apache Airflow. Proven ability to process large-scale structured and semi-structured datasets.

• Utilized Sqoop to transfer data from MySQL to HDFS.
• Employed Spark for data processing and transformation to ensure data quality and consistency.
• Understanding knowledge to Import data into HDFS and Hive using Sqoop and managed data within the environment.
• Loaded data into Hive from Spark RDDs and data frames for further processing.
• Loaded and transformed large sets of semi structured data like JSON, Avro, Parquet.
• Queried data using Spark SQL on top of Spark engine for faster datasets processing.
• Created multiple Hive tables, running hive queries in those data, implemented Partitioning, Dynamic Partitioning, and Bucketing in Hive for efficient data access.
• Created EC2 instances and EMR clusters for development and testing.
• Performed step execution in EMR clusters for the job deployment as per requirements.
• As per the business requirement storing the spark processed data in S3 with appropriate file formats.
• Creating different types of tables like Managed Table and External table.
• Skilled in using Spark SQL persistence and caching mechanisms to reduce data processing overhead and improve query performance.
• Familiarity with Spark SQL schema and data type operations, such as creating, modifying, and dropping tables and handling null values.
• Knowledge of Spark optimization techniques, such as cost-based query optimization, column pruning, and predicate pushdown, and their impact on query performance and resource utilization.
• Experience working with Spark in production environments and implementing performance monitoring and alerting systems to detect and resolve performance issues proactively.
• Proficient in processing serialized data in Spark using various formats, such as Avro, Parquet, ORC, and their features.
• Skilled in working with textual data formats in Spark, such as CSV, JSON, and XML, and their serialization using Spark DataFrames and RDDs.
• Maintained and monitored Spark clusters on AWS EMR, ensuring high availability and fault tolerance.
• Designed and implemented data lake architectures on Amazon S3 and columnar formats such as Parquet to optimize query performance.
• Optimized Spark jobs and data processing workflows for scalability, performance, and cost efficiency using techniques such as partitioning and caching.
• Designed and developed Spark applications to implement complex data transformations and aggregations for batch processing jobs, leveraging Spark SQL and DataFrames.
• Loaded and transformed large sets of data.