We have resolved the issue and the system is functional. We will improve system performance further as the week goes on.
Posted Oct 31, 2022 - 15:12 UTC
Update
We are currently implementing a fix for our performance issue. During this time the system will be up, however you still may notice some performance issues here and there and possible slower than normal responses. If you have an urgent PASS Request and are unable to complete a pass report or update information for a pass report, please reach out to our support team. We will post a message when we have completed this operation and restore the Operator Status widgets.
Please Note: There may be some lag when saving data, It may take a moment or two before it shows in the system during this time.
Posted Oct 30, 2022 - 18:07 UTC
Monitoring
We have created new read only servers and have added a few more to help with the server load. We are disabling the system compliance widget which is a main drag on the performance for the moment. We have one more fix which should be implemented in the next day that will restore performance, Once implemented we will be able to restore the widgets.
Posted Oct 29, 2022 - 01:27 UTC
Identified
We have discovered another issue were replication and other issues with our read only databases are not in sync with the primary and are acting slow. We are working with our host to restart, the system may be inaccessible while the restart happens.
Posted Oct 28, 2022 - 22:44 UTC
Monitoring
We have deployed our fix for what should be the main culprit ( pass reports and loading safety reports ). This should have a good trickle affect into the rest of the system. We will continue to look at other improvements around the system that interact with the safety reports and work to improve efficiency further. Some of the DB requests are hung up in limbo and will take some time to clear out, while this happens the system will be a little slower, but should be functional
Posted Oct 28, 2022 - 18:01 UTC
Identified
We have identified that it is a Database related issue and some queries that have degraded over time. We are working to deploy a fix that should improve the query times.
Posted Oct 28, 2022 - 17:18 UTC
Investigating
Pass Reports are being slow to create and return. We are investigating
Posted Oct 28, 2022 - 16:02 UTC
This incident affected: US Based - ACES / QSMS (Aces Platform).