Email delivery interruption
— Ben
When & What was effected
Over several weeks Email sending was interrupted in the Acter App for all Matrix accounts on acter.global. In particular verifying emails addresses to the app or using them for password recovery didn't work: emails just didn't find their inboxes.
What happened
We first though this was a bug in the App and thus searched there for a quite a while, before we realized the cause was in the running infrastructure. In an attempt to break free from unnecessary servers we have recently consolidated the services we used to send emails through. We had already disabled mailgun and then also closed our sendgrid account in order to switch (temporarily) to Amazon SES, which worked fine in testing. What we didn't realise was that it was running in a sandbox-mode only allowing us to sent to acter.global email-addresses. Thus, our tests worked, but emails to external providers were not sent.
What was compromised/lost
Nothing was compromised or lost. Just emails were not delivered.
Measures taken to prevent this in the future
We have finished the switch and are now sending emails through sweego, a French company with Data Centers in France. Email sending has been restored successfully.
Additionally, we have been planning to add a more extensive monitoring system to our infrastructure and will for sure add the ability to get notified when the email sending isn't working anymore as expected.