Madhan Lingareddy
Vetted Talent
Madhan Lingareddy
Vetted Talent
Data Engineer with 6 years of experience working on various types of data, different ETL and BI tools. Technically strong in Python, Apache Spark (PySpark) and SQL. Responsive expert experienced in monitoring database performance, troubleshooting issues and optimising database environment. Possesses strong analytical skills, excellent problem-solving abilities, and deep understanding of database technologies and systems.
- Role
Sr Data Engineer
- Years of Experience
7 years
Skillsets
- Airflow - 3 Years
- REST API - 1 Years
- Oracle
- Apache Spark - 5 Years
- Cloud - 5 Years
- SQL - 6 Years
- AWS - 3 Years
- GitLab - 4 Years
- Pandas - 2 Years
- CI/CD - 3 Years
- Redshift - 3 Years
- PostgreSQL - 3 Years
- Mssql - 1 Years
- Data Migration - 2 Years
- AWS Expertise - 4 Years
- Etl development - 4 Years
- SQL Proficiency - 6 Years
- Data Analysis - 2 Years
- MySQL - 3 Years
- GCP - 3 Years
- BigQuery - 3 Years
- Tableau - 2 Years
- APIS
- Python - 5 Years
- ETL - 4 Years
- Snowflake - 2 Years
- Pyspark - 5 Years
- Data lineage - 1 Years
- Data cataloging - 1 Years
- Data lake - 3 Years
- Bi development - 3 Years
- Data Governance - 2 Years
- Data Engineering - 4 Years
Vetted For
13Skills
- Roles & Skills
- Results
- Details
Data Engineer || (Remote)AI Screening- 78%
- Skills assessed :Airflow, Data Governance, machine learning and data science, BigQuery, ETL processes, Hive, Relational DB, Snowflake, Hadoop, Java, PostgreSQL, Python, SQL
- Score: 70/90
Professional Summary
7Years
- Apr, 2022 - Present3 yr
Sr Data Engineer
Pasarpolis - Apr, 2021 - Apr, 20221 yr
Sr Data Engineer
ADF Data Science - Feb, 2018 - Mar, 20213 yr 1 month
BI Developer
Wipro
Applications & Tools Known
Python
SQL
.png)
Apache Spark
AWS
GCP
Airflow
Amazon Redshift
Snowflake
BigQuery
AWS Glue
AWS Lake Formation
Tableau
Quicksight
.png)
Cloud Composer
GitLab
AWS S3
Pentaho
Spline
Work History
7Years
Sr Data Engineer
PasarpolisApr, 2022 - Present3 yr
- Designed and implemented ETL architecture using Cloud Composer on GCP and documented the process.
- Created Airflow Workflows for ETL and generating reports
- Architected a Data archival system and authored base scripts for the python
- Repository.
- Migrated BigQuery Datasets across regions successfully for Compliance
- Reasons.
- Setup a real-time data streaming pipeline using managed services like Data Streams and BigQuery.
- Designed warehouse table for Tableau Dashboard that displays SLA Quartiles
- and Customer Lifetime values.
- Working on POC for a cost effective real time data system, focusing majorly on tools offered by AWS and Open Source.
- Updated ETL logic for Huge tables to bring down the load times by 20 %.
Sr Data Engineer
ADF Data ScienceApr, 2021 - Apr, 20221 yr
- Orchestrated data pipelines for migrating data from MySQL to AWS Redshift
- using AWS DMS
- Authored scripts using Python and PySpark for building Historical data from Change data capture (CDC) files in AWS S3
- Authored an ETL workflow in Pentaho to detect daily Schema changes
- Designed metadata and calendar dimension as per business requirements
- Installed an open source Data Cataloging tool called Amundsen for data governance and cataloging
- Drafted a Python code to validate the counts by parsing Bin Log files of MySQL
- Enabled data lineage tracking for Spark workloads using Spline which is also an open source tool
- Designed and Developed data pipeline using Python and Spark on AWS Glue for building Warehouse Tables
- Have setup monitoring and alerts using AWS CloudWatch to notify on Slack in case of errors.
- Optimized Analytic queries to bring down the runtime from 2 Hrs to 2 Mins by taking advantage of Redshift data distribution and query tuning techniques.
- Built a scalable Data Lake on AWS S3 using AWS Glue, Apache Hudi and Lake Formation, which enables easier Data Shares with clients across different AWS accounts without the need for any ETL.
- Built numerous dashboards on Quicksight and taken care of Admin activities
- Led data analysis initiatives to extract insights and trends from large datasets.
BI Developer
WiproFeb, 2018 - Mar, 20213 yr 1 month
- Collected and analyzed data requirements that enable building dashboards and documentation around them
- Worked as a BI Developer and Data Engineer for Broadcom Inc., Collected and analyzed data requirements from clients
- Built Data models and ETL workflows using Incorta
- Data Enrichment using python pandas and Rest APIs
- Optimized code using Multithreading and Multiprocessing whilst using pandas framework to reduce execution times from over 6Hrs to less than a couple of Hours
- Developed numerous dashboards on Incorta
- Authored Python Scripts to automate Dashboard delivery using Incorta's REST API and pandas for data cleaning
- Developed PTD dashboards on Incorta, which were previously running on Oracle Publisher Optimized these queries for a huge jump in performance
- Designed and developed a series of dashboards for tracking employee performance at various levels.
Achievements
- Captain of the winning team, Cricket in the University Sports Meet
- Customer Charmer Award, Wipro: 2021
Education
Bachelor of Technology: Electronics And Communications Engineering
IIIT, RGUKT (2017)
Certifications
Incorta fundamentals for admins and developers: 2019
Python intermediate, hackerrank: 2020
Aws cda, udemy: 2020