Resolved -
We've seen no additional performance degradation after our mitigation work.
Jan 20, 09:00 EST
Update -
We have completed processing of our primary indexing backlog, session filtering and metrics will be mostly up-to-date. Our secondary processing system is currently processing and includes some metrics, issues and alerting.
Jan 19, 10:41 EST
Monitoring -
We have identified the cause of the issue, applied a fix and have begun processing the backlog of events.
Jan 19, 03:15 EST
Update -
We have restored some indexing to the database but are continuing to fully resolve the situation.
Jan 19, 00:03 EST
Update -
We are continuing to work with our vendor to remedy the situation. Session data collection continues to be unaffected, but secondary indexing that powers search and metrics continues to be delayed.
Jan 18, 20:00 EST
Update -
We are continuing to work with our vendor to resolve the issue.
Jan 18, 15:02 EST
Update -
We are continuing to work with our vendor to resolve the issue with ingesting new data.
Jan 18, 13:00 EST
Identified -
We have identified the issue with our database vendor and are working with them to remedy the situation.
Jan 18, 11:24 EST
Investigating -
We are currently investigating an issue impacting our primary search database. Searching for sessions, loading metric charts, issues and alerting are all impacted. Session data collection has not been impacted.
Jan 18, 10:41 EST