QSMS Network Connection Issue
Incident Report for WYVERN
Postmortem

This morning our hosting provider conducted an update of our kubernetes hosting environment to a newer version automatically, they made the update a required update. Unfortunately our ingress controller ( the software inside kubernetes that directs traffic ) was of a version that was incompatible with the updated Kubernetes software. This caused a conflict and network traffic was dropped causing connection issues to the application. Removing, Reinstalling and then updating the DNS to point to the load balancer resolved the issue. Steps have been implemented to mitigate this in the future as part of our sms process.

Posted Oct 19, 2022 - 14:48 UTC

Resolved
This incident has been resolved.
Posted Oct 19, 2022 - 14:43 UTC
Update
We are continuing to monitor, DNS should have had time to flush and our monitoring service / developers are now able to fully access the systems. If you are having an issue please give it another 20 minute s and try again. If you continue to have issues please reach out to us.
Posted Oct 19, 2022 - 14:30 UTC
Monitoring
A fix has been implemented, some dns may take some time to update, Access should be restored as DNS cache on devices clears and picks up the new Entries.
Posted Oct 19, 2022 - 13:44 UTC
Identified
We have identified the issue as a software conflict and are working to upgrade the components to restore
Posted Oct 19, 2022 - 13:09 UTC
Investigating
We are currently investigating this issue.
Posted Oct 19, 2022 - 12:18 UTC
This incident affected: US Based - ACES / QSMS (SMS - Domestic).