The good, bad, and ugly ways of uploading large batches of data — Speed matters! In data pipelines, as in anywhere else. Dealing with massive datasets is an everyday thing for most data professionals. There would be no issue if they streamed into a database or a warehouse. But there are instances where we need to upload batch data that are heavy enough…

Decorators provide a new and convenient way for everything from caching to sending notifications — At first, every developer’s goal is to get things working. Slowly, we worry about readability and scalability. This is when we first start thinking about decorators. Decorators are an excellent way to give additional behavior to a function. …

An elegant way to perform computation within the context of a record — Imagine a situation where you run a nationwide electronic store. You must determine how much each store contributes to the state’s sales. If you’re starting on SQL, you’d think for a bit and develop an incredibly amazing query that may look like the following. SELECT s.state, s.store_name, s.total_sales, s.total_sales /…

Speed up your SQL learning curve — Are you curious about SQL but hesitant to dive in? Or maybe you’re already familiar with the basics of SQL but struggle to apply it to real-life projects. I know the feeling. When I first started learning SQL, I was intimidated by the vast amount of information. And even today…

Regex simplifies pattern-matching tasks on large amounts of text — Pandas makes it elegant — Regex is the most potent technique to clean and extract data. If you’ve ever worked with a large text dataset, you’d know how time-consuming and energy-draining it would be. I often use regex to clean phone numbers and emails and standardize addresses. But there are complex use cases as well. …

The Pandas defaults aren’t optimal. A tiny configuration can compress your dataframe to fit in your memory. — I never thought my code needed improvement. I’ve always complained that I don’t have enough memory or the dataset was too big to handle. My go-to solution was to put them on a Postgres DB and write SQL queries. After all, that is an acceptable way to handle large-scale datasets…

Nobody wants a messy codebase; few have the patience to clean it — Writing code is hard enough, but ensuring it's well-formatted and easy to read can be even more challenging. My coding skills have improved a lot in a decade. But the majority of them are not some fancy API usage or anything. It's how I format the code. In my earliest…

What skills you as a developer should have to be irreplaceable? — The world is simplified. The work of developers is to be. Ever since we started using machines, there has been programming. The algorithm for the Analytical Engine is considered the first computer programming language. The English mathematician Ada Lovelace created it in 1883 for Charles Babbage’s mechanical general-purpose computer.

This amazing platform has a different purpose. Use it wisely. — If you say YouTube is your last resort, I have no objection. But being free and super popular doesn’t make YouTube the best learning platform. Having tons of tutorial videos doesn’t either. YouTube is a social media built for videos. Over time it has proven valuable in many ways—people from…