On the 26th of November we had an interruption in our servers located in Stockholm. This applies to VPN02, VPN03 and VPN04.
The reason for the interruption was that the transformer in the UPS in the data center broke down. The data center lost 80% of its electricity, which caused the majority of the servers in the data center to be shut down. This affected us, and all of our servers were shut down.
Since it was not our hardware that was broken we could not do anything to bring the servers back up quickly, instead, it was ISP that needed to fix the issue. They worked fast, and got the power back to the data center within an hour.
When the power came back, VPN03 & VPN04 were started by themselves and became available to use. However, VPN02 was still unavailable.
It seems that VPN02 fell victim to some kind of kernel bug which made the whole system hang - and started the usual debian kernels instead of those using Tresor. We therefore had to work on restoring the server, ensuring that the correct kernels were used. The total downtime for VPN02 was approximately 6 hours.
Our entire infrastructure is back to normal operation, and we deeply apologize for what has happened. If it turns out that we are unable to keep our uptime guarantee of 99.5% in November, all users with an active subscription get two minutes free for every minute that passes our uptime guarantee.
The reason for the interruption was that the transformer in the UPS in the data center broke down. The data center lost 80% of its electricity, which caused the majority of the servers in the data center to be shut down. This affected us, and all of our servers were shut down.
Since it was not our hardware that was broken we could not do anything to bring the servers back up quickly, instead, it was ISP that needed to fix the issue. They worked fast, and got the power back to the data center within an hour.
When the power came back, VPN03 & VPN04 were started by themselves and became available to use. However, VPN02 was still unavailable.
It seems that VPN02 fell victim to some kind of kernel bug which made the whole system hang - and started the usual debian kernels instead of those using Tresor. We therefore had to work on restoring the server, ensuring that the correct kernels were used. The total downtime for VPN02 was approximately 6 hours.
Our entire infrastructure is back to normal operation, and we deeply apologize for what has happened. If it turns out that we are unable to keep our uptime guarantee of 99.5% in November, all users with an active subscription get two minutes free for every minute that passes our uptime guarantee.