Alkmaar     +31659380335

Data Engineering and Analytics

Month: January 2025

Deal with Nested Columns in Spark

By Farnam Iranpour

Flattening Nested Data in Spark with the Explode Function When working with big data, it’s common to encounter nested data structures like arrays or maps. These structures are useful for storing complex relationships but can be challenging to work with during analysis or when storing data in a tabular format. Enter Spark’s explode function—a simple…

Dynamically Calculating Yesterday’s Date

By Farnam Iranpour

Streamlining ETL Pipelines: Dynamically Calculating Yesterday’s Date As a data engineer, one of the most common tasks is building ETL pipelines that process data from the previous day. While it may seem simple to manually enter a date before running the pipeline, this approach is both inefficient and error-prone. A small mistake, such as entering…