In one corner we have Pandas: Python's beloved data analysis library. In the other, AWS: the unstoppable cloud provider we're obligated to use for all eternity. We should have know this day would come.
While not the prettiest workflow, uploaded Python package dependencies for usage in AWS Lambda is typically straightforward. We install the packages locally to a virtual env,
One of the many things you'll want to do when working with large datasets is rid yourself of filthy, filthy data. This can include empty, poorly formatted, or simply irrelevant data entries. While 'bad' data can occasionally be fixed or salvaged via transforms, in many cases it's best to do away with rows entirely to ensure that only the fittest
Let's say you have two obscenely large sets of data. These sets of data might report on a similar topic, such as the personal details of employees, but for some reason set 1 contains email address where set 2 does not, and set 2 contains addresses which are similarly missing in set 1.
Alternatively, these sets of data may contain
As you already know, Python is a pretty popular data science programming language. Besides being a general pleasure to use, a large part of this can be attributed to the vast collection of Python packages dedicated to data analysis. Of all these, Pandas is king.
Pandas is used to analyze and modify tabular data: data that can be represented as
