Clarifying the often-confused terms in data engineering: upstream refers to the processes or data sources that provide data to a particular process, while downstream refers to the processes or systems that consume data from a particular process. For example, if you have a data pipeline that collects data from multiple…

Data engineering is a crucial element of the data ecosystem, comprised of diverse professionals who play essential roles in managing and processing data. While the job title may be the same, as I’ve seen over the years, data engineers often fall into two distinct archetypes: the “businessy” data engineer and…

Imagine a world where the boundaries between science and computer science blur to the point where they almost vanish. François Chollet, the genius behind Keras and XceptionNet, predicts that within the next 10 to 20 years, this might just be our reality. Computational physics, chemistry, biology, and even archaeology are…

You know that feeling when you’re knee-deep in a project, trying to make sense of the chaos and desperately seeking clarity? Oh, believe me, I’ve been there too! …

Amazon Web Services (AWS) has revolutionized the way that businesses manage their IT infrastructure and applications. With AWS, organizations can quickly and easily deploy applications in the cloud, without the need for expensive hardware, software, and networking equipment. AWS offers a variety of services that help businesses meet their high…

Amazon S3 (Simple Storage Service) is a popular cloud-based storage service offered by Amazon Web Services (AWS) that allows businesses to store, retrieve, and manage large amounts of data. With its highly scalable and reliable infrastructure, AWS S3 has become a go-to solution for companies of all sizes and industries. …

Caution: please use the below knowledge with caution — all the tools described below cannot be used with blind copy-pasting. Be aware of that — you have been warned (commands are not working). (But if you do have any funny/scary stories, please share in the comments). One day, when working…

“What is your definition of a good software engineer?” — I like to ask this question to everyone who is somehow involved in software development. And I always get a different answer. The answer seems to depend on people and their attitude — managers think that good engineers are those…

I’ve been talking about Test Driven Development(TDD) practices in the past, but I didn’t give my opinion on that. Unfortunately, I am not a big proponent of TDD. It’s probably because I haven’t worked in places where it’s been a serious part of the culture, haven’t seen how it can…