total meltdown
Incident Report for Fused
Postmortem

Basically, what happened is that CPanel pushed a broken Apache package into their so-called “production” repository, which later was picked up by our update system and was installed during the nightly update cycle. This caused significant damage to the well-being of web and SMTP services across entire fleet. It took us some time to realize what exactly is wrong and the malfunctioning packages have been rolled back to previous versions. CPanel has apologized for the incident.

Posted Dec 18, 2019 - 09:56 UTC

Resolved
To conclude our evening drama, we'd like to apologize for the hiccup. We're still working with cPanel on analyzing what went wrong and will publish an update once we've got the full details in hand.

Thank you.
Posted Dec 17, 2019 - 19:11 UTC
Monitoring
Everything has been rolled back to previous state & that resolved the issues. We're still investigating what went wrong during routine Apache update and will publish the results shortly. Reach out to our support if anything isn't working as expected.
Posted Dec 17, 2019 - 10:48 UTC
Update
This is ultimately more broad than expected. It has impacted not only the SMTP part of mail system, but also all websites. Resolution is underway.
Posted Dec 17, 2019 - 10:29 UTC
Investigating
Stand by while we perform a quick fix for incompatibility introduced by the latest cPanel update.
Posted Dec 17, 2019 - 09:42 UTC