Background Processing for Faster Batches

background processing

We had a client approach us with a specific functionality request. They wanted to capture news articles from around the Internet that mentioned members of their staff and display them on their site. We came up with a great solution that leveraged two technologies in an efficient combination.

First we utilized an extension of Google’s search engine, Google Alerts, to repeatedly search for and gather the kinds of articles we wanted to see on the site. Within WordPress, we created a custom post type to store and format that data for inclusion on the site. With that piece of the puzzle created, we could see that there could be a considerable amount of entries created in a single day. Imagine searching for someone like “David Jones”, there could be thousands of mentions a day for a common name like that.

We needed a way to quickly process the large volume of incoming without waiting for the normal cron job triggers. Our biggest concern was that the volume of data we would be processing would cause server timeouts during the normal cron process leaving us with an ever-growing pile of data that we couldn’t process. Ashley Rich of Delicious Brains recently created an open source background processing code library for WordPress that would allow us to batch process the Google Alerts results without fear of overloading the server. With our implementation of this library, we could have these alerts streaming in, and the background processing moved them smoothly into the right position and was able to keep up with the volume.

The last hurdle was creating an approval system for the custom post type so that the right mentions are displayed for the right user. Using the example above, if the wrong “David Jones” was mentioned we didn’t want to see that show up for our David Jones. Without an approval the article doesn’t get displayed on the front end.

All together, the Google Alerts, background processing, and approvals make this automated content generation feature a streamlined process. Even without using Google Alerts, the background processing library can be a great tool when large amounts of data need to be handled.

Leave a Reply

Your email address will not be published. Required fields are marked *