Summary

Between December 3rd 10:00PM MST and December 4th 7:00AM MST we experienced a gradual loss of connectivity on many of our hosting locations/servers with CentOS 6 and 7 due to Kernel Panics and causing these servers to be unavailable for many of our customers. Some other customers reported domain redirection issues. 

Root Cause

The main cause was a minor security upgrade from BitNinja Software that was affected by some dependences during this upgrade which was released on December 3rd, 2019. The full BitNinja's report can be found here: https://bitninja.io/blog/2019/12/04/bitninja-agent-v264-bug-update

Resolution

Once the root cause was identified we temporary disabled BitNinja until they released a minor upgrade/fix for this issue within the next hours. We gradually started to upgrade all our servers at 6:00 AM MST and they were back online after that, working as usual. 

Did this answer your question?