Just as a bit of notice for #MastodonAppUk and #Universeodon folks
There will be an outage later this evening to facilitate the work I need to do here at the dc. Hoping it won't take too long and I'll let folks know when I have an idea on time.
I'm just waiting now for a dc staff member to let me through the front gate as it's a weekend so nobody's normally here 😂
Just to note as well for folks on #MastodonAppUK and #Universeodon
I am still looking into this issue with media uploads, I'm so far struggling to pin down a root cause as for most folks it's an intermittent error and I haven't actually found any errors in our logs yet other than what is returned to the apps / clients.
As soon as I can get to the bottom of it I'll get it sorted properly so media uploads work like normal again.
For folks on #Universeodon - I've now ran #Wrapstodon so if you take a look on the website you should be able to view your Wrapstodon update.
Some apps will likely show an unknown notification type and it does only work currently on the web!
Apologies for the issues this morning with #MastodonAppUK and #Universeodon - It looks like the Proxmox backups for our new DB Server decided to brick things. We should now be back up and running and I'll be keeping an eye.
Unfortunately due to these same backups (I think) the #Universeodon new DB server lost connection to the old server and never re-established the connection so the replication is once again broken and I need to re-run it. I'm going to start that now with the hope that later this afternoon I'll be able to switch us over and retire the old server which should hopefully improve things a bit more.
Okay, I think #Universeodon should now be up, running and stable. There were a couple of bits I missed when I had deployed everything ready for the DB migration so we ran into some issues where we just got crushed with load and the server couldn't handle it properly.
It has flagged some other issues that I wasn't expecting when it comes to capacity and load but those are things I should be able to solve in due course and as part of moving MastodonAppUK over.
I am aware of some stability issues on #Universeodon - I think this is a case of a badly tuned database backup that's causing things to lock up, part of this will be improved in the next week or so with some hardware upgrades to our servers but until then I'll do what I can to get things more stable.
We're going ahead and adding `miniwa.moe` to our list of suspended servers across #MastodonAppUK #Universeodon and #MedMastodon as there appears to be no active moderation taking place and reports of transphobic behaviour have been ignored by their moderators.
I'm not expecting this to impact the vast majority of folks across our instances but you might see a follower / following drop by 1 or 2.
I've made an adjustment to the #Universeodon infrastructure this afternoon to try to improve the traffic and the responsiveness of the site.
We seem to be having a lot of issues at the moment with connectivity between our new environment and the old environment.
I'm hoping this same change should reduce a lot of the traffic we're trying to route back and forth between the two and should hopefully make #MastodonAppUK a bit more responsive as well.
I am hoping to have some time on Sunday to make some further improvements ahead of the maintenance on Monday which should allow me to once and for all finish the migration in the near future!
MastodonAppUK and Universeodon Maintenance:
I'm going to be getting us upgraded to the latest patch release for Mastodon shortly, I don't expect this to cause any sort of noticeable downtime or impact as we can do the upgrades one server at a time.
Sorry for the blips there to #MastodonAppUK and #Universeodon - I managed to bork some of our HAProxy configuration in a bit of a stupid way.
MastodonAppUK is now running a single web server on the new cluster so I'm hoping that should make things feel a lot more responsive as I've removed some of the hops we were going through.
Tomorrow I'll get the other two temp web servers stood up, an additional for MastodonApp and an additional for Universeodon and get all the old servers on the old cluster decommissioned which should then mean all traffic is hitting our new owned dedicated hardware in the new environment!
Apologies folks on #MastodonAppUK and #Universeodon - It looks like we're seeing and have been seeing for the last week or two some issues with certain operating systems and clients trying to upload media to the sites.
We're currently (over the last 5 days) seeing around a 45% failure rate for iOS and 21% failure rate on MacOS devices.
It's not clear what could be causing this and it seems to be impacting multiple clients as far as I can see on my end.
I will continue to investigate and see if I can get to the bottom of the weirdness.
We may be seeing some network related issues across #MastodonAppUK and #Universeodon
I'm keeping an eye but our router is currently unresponsive.
The status page will be updated should we actually see connectivity drops - https://status.atlas-media.co.uk
