Writing as I learn. #dataengineering, #aws, #bigdata
I stumbled upon DuckDB recently when I was doing data analysis with python Pandas. The 3 steps - set up/installation, data exploration, and execution, and it was a very pretty quick implementation. In this post, we will be discussing DuckDB, and what...
With spark-snowflake connector writes We have discussed the internals of spark snowflake writes in the previous post, in this post lets observe how the schema changes such as missing column/extra columns, data type changes behaves with spark-snowfla...
Ensuring data quality checks are really important in data driven projects To make sure of data correctness for correct business decisions Validate the data beforehand to avoid broken production pipelines Validate data from disperse sources(ftp, data...
In this post I would like to walk you through on the steps for Snowflake AWS lambda integration “AWS Lambda is a serverless, event-driven compute service that lets you run code for virtually any type of application or backend service without provisio...
Snowflake have gained in popularity being a platform that offers data storage and its processing in cloud. And the trends also got reflected in the EDP design patterns (Enterprise Data Platform), Snowflake is getting mapped as the recommended DW laye...
