*UPDATE - 6:00 PM (CDT)*Â Another backlog of job processing has occurred in the Midwest data center. We have flushed all jobs and restarted the processing system. The backlog is from the same issue - the Graphiq feed system is having sporadic service and the legacy RSS feed has issues with unreliable connections.
We have decided to disable automatic running of all legacy RSS jobs based Graphiq feeds to prevent further issues over the weekend.
A release will be made next week to correct the behavior by making the legacy feed switch to cURL instead of continuing to use the PHP implementation of HTTP fetching to prevent further outages. Additionally, steps will start to be taken to end-of-life the legacy feed and get more customers to move to the syndication tool which is more insulated from this kind of issue.
-----
The job system in our Midwest data center was backlogged from stalled connections in legacy RSS jobs to Graphiq. This caused all job processing to appear to halt sometime after 10:00 PM (CDT). All job processing was restored around 2:15AM (CDT) and is working through the backlog.
The faulty connection hung the processes which did not timeout on the socket. With no job slots available, other processing was just waiting in the queue for the other jobs to finish. Additional safe guards will be added to the legacy RSS job to provide a better safety net around this by switching to cURL away from fopen().
If you have any questions please contact TownNews.com at (800) 293-9576.
