Akashdeep Gupta

Akashdeep Gupta

Sign in Subscribe

Shuffle-less Join, a.k.a Storage Partition Join in Apache Spark - Why, How and Where?

data engineering

Shuffle-less Join, a.k.a Storage Partition Join in Apache Spark - Why, How and Where?

A Deep Dive into Shuffle-less joins (Storage Partitioned Joins) in Apache Spark to improve Join performance when using V2 Data Sources.

Enhancing Spark Job Performance with Multithreading

optimization-techniques

Enhancing Spark Job Performance with Multithreading

It covers a Spark Job Optimization technique to enhance the performance of independent running queries using Multithreading in Pyspark.

EMRFS S3 Optimized Committer and Committer Protocol for Improving Spark Write Performance - Why and How?

EMRFS S3 Optimized Committer and Committer Protocol for Improving Spark Write Performance - Why and How?

What are EMRFS S3 Optimized Committer and EMRFS S3 Optimized Committer Protocol and how to use and identify if these are working for your Spark Jobs to improve write performance?

Copy-on-Write or Merge-on-Read? What, When, and How?

data engineering

Copy-on-Write or Merge-on-Read? What, When, and How?

Copy-on-Write or Merge-on-Read? Optimizing Row-level updates in Apache Iceberg Table by understanding both the approaches and deciding when to use which approach and its impact on the Read and Write speed of the table. How to identify these using Iceberg Metadata tables on AWS?

Apache Iceberg - Architecture Demystified

data engineering

Apache Iceberg - Architecture Demystified

A detailed explanation of Apache Iceberg Architecture and how it evolves when data is inserted into the table.

How to Implement Write-Audit-Publish Pattern with Apache Iceberg on AWS using WAP id

How to Implement Write-Audit-Publish Pattern with Apache Iceberg on AWS using WAP id

Detailed implementation of Write-Audit-Publish (WAP) Data Quality Pattern in AWS using Apache Iceberg WAP ID i.e. for Apache Iceberg < 1.2.0.

How to Implement Write-Audit-Publish Pattern with Apache Iceberg on AWS using Branches

How to Implement Write-Audit-Publish Pattern with Apache Iceberg on AWS using Branches

Detailed implementation of Write-Audit-Publish (WAP) Data Quality Pattern in AWS using Apache Iceberg Braches i.e. for Apache Iceberg > 1.2.0. It also covers the gotchas while using this pattern and using Athena as a query Engine.

PyDeequ - Testing Data Quality at Scale

data engineering

PyDeequ - Testing Data Quality at Scale

How to use PyDeequ to test your data quality on AWS

Building Discord Bot with AWS Serverless - Part 3

Building Discord Bot with AWS Serverless - Part 3

This blog post is part of the Let's Build Series, where we pick and build an idea. In the inaugural blog post of this series, we developed VerifyRequest Lambda, discussed the code design decisions made, deployed it along with API Gateway on AWS, and finally integrated it with

Building Discord Bot with AWS Serverless - Part 2

Building Discord Bot with AWS Serverless - Part 2

This blog post is part of the Let's Build Series, where we pick and build an idea. In the inaugural blog post of this Series, we looked into what we are building and why we are building, things to know before building, and discussed how request verification works.

Building Discord Bot with AWS Serverless - Part 1

Building Discord Bot with AWS Serverless - Part 1

This blog post is part of the Let's Build Series, where we pick and build an idea. In this 3 part blog series, we will be building a Discord bot that fetches the recently posted blogs from the AWS blog space. We will build this with AWS Lambda

How I cleared AWS Data Analytics Speciality Certification

How I cleared AWS Data Analytics Speciality Certification

On 28th August 2023, After preparing for like 2.5 months, I appeared on AWS Certified Data Analytics Speciality Certification and cleared it. This post will list all the resources that I used and helped me to prepare for this certification. It’s going to be a bit exhaustive list