Service degradation when displaying Request's changes

There was a service degradation of our reference server.

Severe Service Degradation: OBS Unavailable

There was a service degradation of our reference server.

Post-mortem: Service Degradation in the Notifications System

Post-mortem: Service Degradation for Pages with Comments

Post-mortem: Database Cluster Crashes

Post-mortem: Staging Workflow Unavailable

In our efforts to add a distribution (Debian 12), we accidentally deleted an entry in our architecture database table. That made all attempts to fetch this architecture through associations crash.

Post-mortem: Lack of Emails/Notifications from April 6 to April 11, 2023

After implementing the feature that allows the users to comment on specific lines in the changes of a request, we also introduced the possibility to notify about them. However, there was a corner case which caused the notifications to fail, so our reference server did not send any web notifications or emails during the Easter holidays (from 6th to 11th April).

Post-mortem: Downtime on February 23, 2023

Some maintenance operations plus a missing configuration rollback caused a 20 minutes certificate error on our reference server.

Post-mortem: Failing email deliveries on 6th February, 2023

Our reference server stopped sending out email notifications on February 3th, 2023. In the lines below you will find a detailed explanation of what happened.

Post-mortem: Downtime on January 5, 2023

Some maintenance operations caused a long downtime on our reference server. In the lines below you will find a detailed explanation of what happened.

Post-mortem: Downtime on December 5, 2022

After yesterday’s deployment, we faced a downtime on our reference server. In the lines below you will find a detailed explanation of what happened.

Post-mortem: Downtime on November 30, 2022

After yesterday’s deployment, we faced a downtime on our reference server. We want to share with you a detailed explanation of what happened.

Post-Mortem: Rack Gem Version Mismatch on May 31, 2022

There was a severe service degradation of our reference server. On 2022-05-31 a deployment of OBS failed and led to a downtime. We want to give you some insight into what happened.

Post-Mortem: Events Table Overflow on May 18, 2022

There was a severe service degradation of our reference server. On 2022-05-18 the events system of OBS started to fail, causing problems in different areas of the application. We want to give you some insight into what happened.

Post-Mortem: Conflict with ruby standard gems on May 10, 2022

There was a severe service degradation of our reference server. On 2022-05-10 a deployment of OBS failed and led to a downtime. We want to give you some insight into what happened.

Post-Mortem: Repository deletion broken on May 6, 2021

There was a severe service degradation of our reference server. We want to give you some insight into what happened.

Post-Mortem: Search Unavailable on September 3, 2020

There was a severe service degradation of our reference server. We want to give you some insight into what happened.

Post-Mortem: Severe Service Degradation on May 18, 2020

The https://build.opensuse.org OBS instance was unavailable for eleven minutes, outside of our maintenance window. We want to give you some insight into what happened.

Post-Mortem: Severe Service Degradation on April 8, 2020

After today’s deployment, our reference server suffered from a severe service degradation. We want to give you some insight into what happened.

Post-mortem: Downtime on June 13, 2019

After today’s deployment, we faced a downtime of our reference server for users in our beta program. We want to give you some insight into what happened.

Post-mortem: Downtime on November 11, 2018

After today’s deployment, we faced a downtime of our reference server. We want to give you some insight into what happened.

Post-mortem: Downtime on August 15, 2018

After today’s deployment we faced a downtime of our reference server. We want to give you some insight into what happened.

Post-mortem: Downtime on March 23, 2018

After today’s deployment we faced a downtime of our reference server. We want to give you some insight into what happened.

Post-mortem: Deployment on March 15, 2018

During yesterday’s deployment we faced some issues. We want to give you some insight into what happened.

Post-mortem: Deployment on January 17th, 2018

During today's deployment we faced some issues. We had to disable rabbitmq support in build.opensuse.org for some hours.

Post-mortem: Deployment on December 14, 2017

During yesterday's deployment we faced some issues. We had to monkey patch some fixes and we want to give you some insight into what happened.

Post-mortem: Extended Deployment time on August 28, 2017

During deployment, we were facing some issues and build.opensuse.org was not accessible for a couple of minutes.

This sucks and that's why we want to give you some insight in what happened.

Post-mortem: Extended Deployment time on July 19, 2017

We did it again! Yesterday, on 19th of July 2017, we had an extended deployment time because of an issue during the deployment. Though this time it "only" took 15 minutes;-)

This sucks and that's why we want to give you some insight in what happened.

Post-mortem: Extended Deployment time on June 30, 2017

On June 30, 2017 we had an extended deployment time of roughly 45 minutes for our reference server because of a couple of problems with one of the data migrations. We implemented a new feature, user notifications via RSS, that included a migration of data in our database. This migration was broken, causing this deployment to go terribly wrong.

The frontend team afterward met to do a post-mortem to identify the problems, solutions and possible take aways for the future. This is the first post-mortem meeting we held, hopefully but not likely the last. Here goes the report.