12/28/2020 Outage: Failure in removal of old add-ons

[Deebs]

Most of you noticed today that we had a small outage. This was due to an attempt of cleaning/removing unused addons, as we enhance the security posture of this forum.

My personal apologies for the time out you have all experienced. The good part of this is, no data was lost during this issue, though the site was unavailable for some time. We will have to dive deeper into addon interactions in the future, and work with developers when they are not fully uninstalled.

Once we removed the addon folders completely, it was apparent there was an issue with the forums. Immediately a ticket was opened to restore that folder from the snapshot. After waiting for support, with slow responses, we were able to grab the data ourselves from the prototype environment (once we had ftp access), and restore the site from that backup.

Apologies again for all the inconveniences as we try to address the hygiene related aspects of this forum.

Appreciate you all~

Deebs

 

[MtRainDog]

Testing in Production, eh? ?

 

[DavidDellaRocco]

Tried once :

Tried again :

Tried again 15 minutes later :

 

[Deebs]

Testing in Production, eh? ?

Limited testing done did not produce similar errors unfortunately. It was an assumption that an uninstalled component, would not mind it being removed. Turns out..bad assumption, transparency is taking the bad with the good. Just make sure to learn from the bad! In this case, it means a lot more testing!

 

[BH]

Your on it every time, can’t blame ya when it’s really out of your hands. Your def trying hard To make it work and that’s what everyone (members) appreciate .

keep it up staff!

 

[Deebs]

Your on it every time, can’t blame ya when it’s really out of your hands. Your def trying hard To make it work and that’s what everyone (members) appreciate .

keep it up staff!

Appreciate it @Burned Haze. Part of keeping us all secure is not just making sure we are up to date with patches, but also removing old packages that are not in use to eliminate any future security or application related issues with them.

 

[spyralout]

Deebs and I spent a good amount of time to resolve the issue with hosting. It was Deebs that came up with the idea and we did the fix in-house before they could. Good audible play call Deebs ?

 

[spyralout]

Your on it every time, can’t blame ya when it’s really out of your hands. Your def trying hard To make it work and that’s what everyone (members) appreciate .

keep it up staff!

Thanks BH. We are always striving to keep the site the best and most secure it can be. We have a great team that is always on top of this.

 

[Deebs]

Just to keep you all in the loop:

Thanks to everyone again for the understanding on this. Last night I was able to complete the necessary cleanup. This was after finding out through the development team that the order in which the addons needed to be removed was the issue. I tested this thoroughly in the early morning (in proto), and was successful. This was then cleaned up in production successfully, and tested. Onward we go!