System Status
Process Health
libsynPRO Application
|
Media Delivery
|
RSSFeeds |
Statistics Processing
|
Media Conversion
|
Media Uploading
|
Advertising System
|
Wizzard Player
|
Wizzard Mobile Apps
|
Incident Reports
(Subscribe to RSS feed)
All Systems Down
(7:30 AM) All system back. There was a major data center outage that latest between 2:30 AM EST and 6:30 AM EST this morning, 2/22/2010. The secondary data centers were online and working, however due to a DNS configuration error, were not picking up the traffic resulting in failures to deliver media files, RSS feeds, web players, allow logins, or uploading media.
A core aspect of our service was unavailable for 4 hours- and we do not take these issues lightly. Our backup and failover process will be thoroughly reviewed and revised as needed today to ensure this double failure does not happen again.
(6:00 AM) Partial connectivity has been restored. A router failure at the primary data center had caused the majority of issues experienced over the last few hours. The secondary datacenters are available however traffic is not going to that failover. The team is working on the issues.
(3:17 AM) The support team is working with the data center regarding a hardware and network related issue. All services are down at this point
(2:30 AM) Issues with media delivery and web players. Support team being notified
Media Delivery Issues
Update 5:40 PM EST
CDN issues should be resolved. Please let us know if you have any questions or problems.
(1:15pm) Some users have been reporting slow downloads which may timeout. We have opened a ticket with our CDN to look into the issue.
Please contact us at support@libsynpro.com with any questions or problems.
Intermittent system issues
(8:00am)
The team is working on server issues which are causing issues logging in and uploading new episodes
(9:30am)
Most of the systems are back up to normal working state, there may be some issues with cached RSS feeds which will be resolved shortly as the system recycles and comes back up with new copies of all cached feeds.
(10:00) All system back to full normal operation
Ads Temporarily Not Serving - Issue Resolved
Advertising has been temporarily paused while we run some maintenance on the system. Please direct concerns to support@libsynpro.com.
Update 8:58pm EST
The issue has been resolved. Thanks for your patience, and please contact support@libsynpro.com if you have any questions.
Wizzard Player Issues
There are some lingering issues from the recent update that are affecting the Wizzard Player. Item images continue to be incorrect for some users. The player is now operating with some new requirements that are causing the player to not be able to play media on certain sites. These issues are known bugs that we are investigating.
System Maintenance
libsynPRO Application
libsynPRO Uploads
Access to LibsynPro and uploads are currently disabled while we release LibsynPro3.
System Maintenance: Update 5:50pm
Our apologies for access to LibsynPro still being currently disabled. The update is taking a little longer than expected. We hope for access to be reenabled in the next couple of hours. Thank you for your patience.
System Maintenance: Update 8:20pm
We are testing and validating the data post conversion. We hope for access to be available this evening. Thank you for your continued patience.
System Maintenance: Update 9:35pm
We have brought a few networks back online while performing final testing. We hope to turn all networks back on in the next couple hours.
System Maintenance: Update 10:35pm
The update is complete and the system is nearly back to normal. There were a few players and feeds that were temporarily broken. These are issues that we're working out now.
System Maintenance: Update 2:00am
Some unforseen caching issues has caused further delays. The team is working hard to find the bug and get everyone back online asap.
System Maintenance: Update 4:00am
Having issues with content uploads, ad campaigns, and episode thumbnails. Running campaigns are temporarily paused while we work out the issues. There may be some errant thumbnail issues as well. The team is working to get everything right.
System Maintenance: Update 5:40am
Upload issues resolved, campaigns back on, mixed up images are being fixed. The biggest issue at the moment is the feed caching problems which are making RSS feed urls timeout as well as the wizzard flash players. Team is hard at work to resolve all issues.
System Maintenance: Update 8:00am 7/13/2009
Feed caching continues to cause problems which are making RSS feed urls take a very long time to load and wizzard flash players to timeout or have very long load times. Networks are being brought on one at a time to mitigate load issues while the caches warm up. If you have immediate publishing needs, please contact support@libsynpro and we can bring your network online. We appreciate your continued patience through this transition.
System Maintenance: Update 4:30pm
Access has been restored, and all uploading and publishing features have been brought back online. Please contact support@libsynpro.com if you have any questions or problems. We appreciate your patience.
Upload Errors
(3:00pm) We have received reports of errors when uploading media files. The issue was caused by one of our service providers. Our developers have opened a ticket with the service provider, and we will post updates on the situation as it progresses.
(3:30pm) The issue with the service provider has been resolved, and uploads are now working. We apologize for any inconvenience this may have caused – please email support@libsynpro.com if you have any questions.
Performing system maintenance tonight
(10:00pm)
We received short notice from a service provider that will be performing system maintenance tonight. We will be temporarily disabling logins between the hours of 12am and 1am, and we will be providing updates on the situation as it progresses.
(1:00am)
Logins reenabled - please email support@libsynpro.com if you have any questions.