Creating ideas and insights.

Celery Database Bottlenecks 

The joys of performance whack-a-mole with distributed systems

I recently had to refactor some code which uses MongoDB and Celery to store results from a scraping process to a MongoDB collection. It involved a number of whack a mole type performance problems due to the distributed nature of the system, and indeed was leading to the Linux out …

Two approaches to scale your processing: Task Queues and Workflows 

PyCon 2017

I was very fortunate to present to the main track at PyCon Ireland 2017 on the topic of scaling your processing.

The talk was entitled “Two approaches to scale your processing: Task Queues and Workflows” and was aimed at an intermediate Python audience to introduce them to the Celery and …