Creating ideas and insights.Creating ideas and insights.Sat 31 Mar 2018
tags: python queues celery bottlenecks
I recently had to refactor some code which uses MongoDB and Celery to store results from a scraping process to a MongoDB collection. It involved a number of whack a mole type performance problems due to the distributed nature of the system, and indeed was leading to the Linux out …
Thu 14 Sep 2017
tags: python presentations data science queues celery airflow rabbitmq
I was very fortunate to present to the main track at PyCon Ireland 2017 on the topic of scaling your processing.
The talk was entitled “Two approaches to scale your processing: Task Queues and Workflows” and was aimed at an intermediate Python audience to introduce them to the Celery and …