Background Processing for Faster Batches

Background Processing for Faster Batches

We had a client approach us with a specific functionality request. They wanted to capture news articles from around the Internet that mentioned members of their staff. Then, they would display these articles on their site. We came up with a great solution that leveraged two technologies in an efficient combination.

Setting the Stage

First of all, we utilized an extension of Google’s search engine, Google Alerts, to repeatedly search for and gather the kinds of articles we wanted to see on the site. Next, within WordPress, we created a custom post type to store and format that data for inclusion on the site. With that piece created, we could see that there could be a considerable amount of entries created daily. Imagine searching for someone like “David Jones”. There could be thousands of mentions a day for a common name like that.

Leveraging Background Processing

We needed a way to quickly process the large volume of incoming without waiting for the normal cron job triggers. Our biggest concern was that the volume of data we would be processing would cause server timeouts during the normal cron process. This would leave us with an ever-growing pile of data that we couldn’t process. Ashley Rich of Delicious Brains recently created an open source background processing code library for WordPress that would allow us to batch process the Google Alerts results without fear of overloading the server. With our implementation of this library, we could have these alerts streaming in. The background processing moved them smoothly into the right position and was able to keep up with the volume.

The Human Element

The last hurdle was  to create an approval system for the custom post type. Due to this, the right mentions show for the right user. Using the example above, if the wrong “David Jones” was mentioned we didn’t want to see that show up for our David Jones. The article doesn’t display on the front end without an approval.

All together, the Google Alerts, background processing, and approvals make this automated content generation feature a streamlined process. Even without using Google Alerts, the background processing library can be a great tool when large amounts of data need to be handled. We love web development challenges like these and were able to build a complete system.

Written by the Team at Pixel Jar

We hope you got something useful out of that post. If you'd like to read more we have an active blog with topics across the spectrum of website development. If you're researching information for a project we'd love to talk to you about it.

Cloudbleed and AWS – Cautionary Tales

Cloudbleed and AWS – Cautionary Tales

Recently, there have been two major events in the online world that have impacted the way people work – Cloudbleed and the Amazon Web Services outage. While neither of these ended up being an attack, they have certainly brought questions of security to the foreground. We’re going to take a look at the two events…
Read More
WordCamp U.S. 2017

WordCamp U.S. 2017 – Nashville

This year WordCamp U.S. was held in Nashville, Tennessee. The event happened at the Music City Center, an enormous conference center that dominates the center of downtown Nashville. As the second destination for the traveling WordCamp, Nashville was an amazing choice. Some of the Sites in Nashville Nashville is all about two things: music and…
Read More

Learn how to understand your website's strengths and weaknesses, and how to make changes for the best possible chance at success.

Download Now

How Can We Help You?

We want to build your next project.

Connect with Pixel Jar

Our Community

Subscribe to learn more about the goings on at Pixel Jar.

  • Note: Your email will be added to our CRM and be used to receive emails from Pixel Jar. You can unsubscribe at any time.

  • This field is for validation purposes and should be left unchanged.