Past Incidents

Two of our SSD disks do report problematic wearout indicator values. For the moment, they still do run fine, but we will replace them out of caution nevertheless. During the rebuild, a higher disk i/o latency is to be expected.

Update 25/06/2023 10:00: Both disk have been replaced successfully.

Start maintenance window: 24/06/2023 18:00
Stop maintenance window: 26/06/2023 06:00

We will update Confluence to version 7.19.10.

Start maintenance window: 19/06/2023 23:00
Stop maintenance window: 20/06/2023 02:00

At 14/06/2023 15:44 we detected increased high latencies in the network during maintenance work. This delayed the internal data replication, which in turn led to further latencies in the system. As a result, certain services were no longer available. The situation slowly returned to normal after a structured restart of the affected services.

Our analysis showed that this was not an attack. We have identified a human error as the cause. The confidentiality and integrity of the data was guaranteed at all times.

As an immediate measure, first organisational measures have already been taken. In addition, further infrastructure optimisation is planned for the next few weeks to prevent similar problems in the future.

We apologise for the inconvenience experienced by some of our customers. Thank you for your patience and understanding.

Incident timeline

• 17:48 [Update] - Services are back in operation. We will closely monitor the situation over the next few hours and then analyse the root cause.
• 16:53 [Update] - We are still experiencing limited availability of certain services. The affected systems are gradually becoming available again
• 16:23 [Update] - Affected services are partially available again. But still suffering from poor performance
• 15:53 - We are experiencing a partial outage of our network and are currently investigating the cause

We will update the following packages to the latest version.

vim-common/oldoldstable 2:8.1.0875-5+deb10u5 all [upgradable from: 2:8.1.0875-5+deb10u4]
vim/oldoldstable 2:8.1.0875-5+deb10u5 amd64 [upgradable from: 2:8.1.0875-5+deb10u4]
vim-runtime/oldoldstable 2:8.1.0875-5+deb10u5 all [upgradable from: 2:8.1.0875-5+deb10u4]
vim-tiny/oldoldstable 2:8.1.0875-5+deb10u5 amd64 [upgradable from: 2:8.1.0875-5+deb10u4]
xxd/oldoldstable 2:8.1.0875-5+deb10u5 amd64 [upgradable from: 2:8.1.0875-5+deb10u4]

Start maintenance window: 13/06/2023 15:00
Stop maintenance window: 13/06/2023 16:00