Data serialization with Apache Avro — What is Apache Avro? Avro is an open-source language-agnostic data serialization framework. The schema of Avro files is specified in JSON format, making it easy to read and interpret. Files that store Avro data should always also include the schema for that data in the same file. Avro includes APIs for C, C++, C#…

Visualize observable changes in global temperature using NOAA’s historical weather data, Apache Spark, BiqQuery, and Data Studio — We have all read about and experienced the effects of climate change every day around us. We have seen numbers like: The current global average temperature is 0.85ºC higher than it was in the late 19th century, and each of the past three decades…

Create a Multi-Cloud Data Lake using Terraform and run a configuration driven Apache Spark data pipeline on COVID-19 data — Five years back when I started working on enterprise big data platforms, the prevalent data lake architecture was to go with a single public cloud provider or on-prem platform. Quickly these data lakes grew into several terabytes to petabytes of structured and unstructured data(only…