ETLs don’t have to be complex. If that’s the case, use GitHub Actions. — If you’re into software development, you’d know what GitHub actions are. It’s a utility by GitHub to automate dev tasks. Or, in popular language, a DevOps tool. But people hardly use it for building ETL pipelines. The first thing that comes to mind when discussing ETLs is Airflow, Prefect, or…

Matching text where there’s no perfect match — The real world is not perfect. People use different forms of the same piece of information. Even well-established systems use different standards. You’d have seen city names misspelled, like “Santana” instead of “Santa Ana” or “St. Louie” instead of “St. Louis.” When working with real-world data, this is inevitable. Thus…

You don’t always need Python or R to fit models — Postgresql has covered the basics — SQL programmers hardly fit any ML models. Someone else will do it unless they have either Python or R knowledge. While Python and scikit-learn are often my go-to tools for machine learning, it is worth noting that SQL can also do some quick and dirty model fitting. Regression models are…

Simple hacks to try before moving to a different data model altogether — Several databases have emerged to solve specific software development and data science needs. Yet, relational databases are still the most used and preferred by many. The query speed is the most common reason people think of a different data model. …

Bringing the best of both worlds together — I’m keen on finding ways to speed up analytical tasks. I’ve written several posts in the past about measuring memory usage, running time, and tips to speed up data-intensive tasks. Since we use Pandas primarily daily, I keep researching for improved performance. How to Do Fuzzy String Matching in Pandas Dataframes Matching text where there’s no perfect matchtowardsdatascience.com

Pipes are an incredibly fantastic technique for more human-friendly coding — Your code is your documentation too. People say that great programmers do not add comments to their code. They believe that if the code is hard to create, it should also be challenging for others to comprehend and alter. Hence they write straightforward code. Although I don’t advocate for not…

The good, bad, and ugly ways of uploading large batches of data — Speed matters! In data pipelines, as in anywhere else. Dealing with massive datasets is an everyday thing for most data professionals. There would be no issue if they streamed into a database or a warehouse. But there are instances where we need to upload batch data that are heavy enough…

Decorators provide a new and convenient way for everything from caching to sending notifications. — At first, every developer’s goal is to get things working. Slowly, we worry about readability and scalability. This is when we first start thinking about decorators. Decorators are an excellent way to give additional behavior to a function. And there are little things we data scientists often need to inject…

An elegant way to perform computation within the context of a record — Imagine a situation where you run a nationwide electronic store. You must determine how much each store contributes to the state’s sales. If you’re starting on SQL, you’d think for a bit and develop an incredibly amazing query that may look like the following. SELECT s.state, s.store_name, s.total_sales, s.total_sales /…