Lazy Pandas and Dask
Speed up data analysis by parallelizing your DataFrames.
- Comparing Rows Between Two Pandas DataFrames
- Using Hierarchical Indexes With Pandas
- Reshaping Pandas DataFrames
- Data Visualization With Seaborn and Pandas
- Parse Data from PDFs with Tabula and Pandas
- Lazy Pandas and Dask
- Automagically Turn JSON into Pandas DataFrames
- Connecting Pandas to a Database with SQLAlchemy
- Dropping Rows of Data Using Pandas
- Merge Sets of Data in Python Using Pandas
- Another 'Intro to Data Analysis in Python Using Pandas' Post
Ah, laziness. You love it, I love it, everyone agrees it's just better.
Flesh-and-blood are famously lazy. Pandas the package, however, uses Eager Evaluation. What's Eager Evaluation, you ask? Is Pandas really judgey, hanging out on the street corner and being fierce to the style choices of people walking by? Well, yes, but that's not the most relevant sense in which I mean it here.
Eager evaluation means that once you call
pd.read_csv(), Pandas immediately jumps to read the whole CSV into memory.
"Wait!" I hear you ask. "Isn't that what we want? Why would I call the function if I didn't want it to happen?"
Eventually, yes that is what you want. But sometimes you want some time in between when you give the command and when the computer hops to it. That's why it's Lazy and not Inactive - it will get to the job at some point, it'll just procrastinate a bit.
For example, last week I was tasked with searching through the output of a government API. It had records since the 90s, and was about 300MB. Now, this isn't actually outside the realm of what Pandas can handle - it's quite optimized, and as long as the file can fit into memory, Pandas can mess with. However, it won't necessarily be fast. Furthermore, my laptop is old and I didn't feel like offloading what I was doing to a remote machine.
Furthermore, I knew I actually only needed a subset of the file. Here's where Laziness comes in handy. With Eager evaluation, Pandas would have to load the whole thing into memory, then filter based on my criteria.
Enter Dask: Dask is a very cool little library that seamlessly allows you to parallelize Pandas. Pandas by itself is pretty well-optimized, but it's designed to only work on one core. Dask, on the other hand, lets you split the work between different cores - both on a single machine, or on a distributed system. It doesn't implement every single thing you can do with Pandas, though, so only use it when you have to.
I probably should have titled this post "Parallel Pandas", but hey, too late now - plus "Lazy Pandas" more easily lends itself to a nice visual metaphor. Anyway, Laziness is part of the process. Because Dask is lazy by default (much like your humble narrator), we can define our fileout loading it, like so:
import dask.dataframe as dd df = dd.read_csv("giantThing.csv")
Pandas was taking a long time to parse the file. What's more is that this file had a few quirks - I'd figured out that it needed a special text encoding, and I wasn't sure if there was other weirdness going on. Was my computer just taking a long time to nom the file, or was I going to wait there for a few minutes to find an error message? It's a catch-22 - I needed to figure out how to cut the file down to size, but in order to do that I would have needed to be able to load it into memory. Dask to the rescue!
This file wasn't terribly well-documented - I had an inkling as to what column would tell me which rows I wanted, but I wasn't sure. So, first thing I did was check out the first few rows. Remember, in order to see these first 5 rows in Pandas, I'd have to load the whole thing into memory (which might or might not even work!).
With that, I was able to do a simple spot-check and see if there were any weird columns that might get in the way of parsing. Furthermore, I confirmed that the ID columns I was looking at contained something vaguely like what I was looking for. Even MORE interestingly, I found that it was formatted slightly irregularly. Even more use for laziness! Let's load just that one column into memory (you could do this with a loop, sure - but selecting a single column is a lot clumsier)
.compute() method at the end. That's necessary because of the Lazy Evaluation - just calling a column name doesn't make Dask think you necessarily want the thing now. I'm not sure why I didn't have to call it with
df.head(), though (that's the Hackers & Slackers Codeblogging vérité style!).
So, now that I've seen the formatting, I found out that I'm going to have to filter it with a call of a
str.contains() method instead of an exact value. Let's poke around a teensy bit more.
orgDF = df["ORG_NAME"] orgFiltered = corp[corp.str.contains("baseName", na=False)].compute().shape
Turns out it was only about 800 rows!
So, let's filter that and make a regular Pandas Dataframe (and probably a new CSV for later!)
df = dd.read_csv("giantThing.csv") orgFiltered = df[df["ORG_NAME"].str.contains("baseName", na=False)].compute() df2 = pd.DataFrame(orgFiltered) df2.to_csv("filteredThing.csv")
Note that I actually could have done this with base Pandas, through use of the iterator flag. However, I didn't realize that it's only wind up being so few rows. It also would have been slower - and the speed difference makes a huge difference in terms of how fluidly you can explore.
For instance, the
na=False flag was something I discovered would be needed because of a quirk in the file - again, this sort of thing becomes a lot easier do diagnose when you can iterate quickly, and you know you're not going to just timeout from running out of memory.
For comparison's sake, here's the code for filtering on the fly and loading into Pandas:
iter_csv = pd.read_csv("giantThing.csv", iterator=True, chunksize=1000) df = pd.concat([chunk[chunk["ORG_NAME"].str.contains("baseName", na=False)] for chunk in iter_csv])
On my computer, that took a little over 3 minutes. While the Dask code took about a minute.