At approximately 1:54PM today, our monitoring service alerted us to an inability to log into DonorPerfect via our API. We shortly thereafter confirmed that clients were also unable to log in, though clients who were already logged in were able to continue to work in many instances.

Our support team analyzed the situation and realized that a critical table used by our login process was missing data. Because this data is regularly replicated and backed-up multiple times per day, the team was able to restore the missing data quickly and login service was restored at 2:12pm. We confirmed shortly after that all services were back and continued to monitor the situation. No other residual issues were noted.

What Happened: In researching a root cause, our team found that a data replication process was inadvertently configured to run against itself on one of our servers, effectively deleting data held within that table and causing login attempts to fail.

What are we doing: The process has been re-coded and re-deployed to prevent this from happening in the future.