Sep 18, 11:57 UTC Resolved - We are processing real-time data for all data types again. Sep 18, 11:45 UTC Update - Error ingestion is fully operational. We are currently working on recovering real-time processing of spans. Sep 18, 10:58 UTC Identified - We have identified a networking issue with an upstream provider and are working on mitigations. Sep 18, 10:56 UTC Update - We have identified a ne…
Sep 18, 10:58 UTC Resolved - We have identified a networking issue with an upstream provider and are working on mitigations. Sep 18, 10:56 UTC Update - We have identified a networking issue with an upstream provider and are working on mitigations. Sep 18, 10:17 UTC Investigating - We are currently investigating an issue affecting spans ingestion in the EU region
Sep 16, 22:06 UTC Completed - The scheduled maintenance has been completed Sep 16, 22:06 UTC Update - Scheduled maintenance is still in progress. We will provide updates as necessary. Sep 16, 21:15 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Sep 14, 22:10 UTC Scheduled - We will be performing a database migration in the EU region that wil…
Sep 10, 17:10 UTC Resolved - Ingestion is now operating normally. Sep 10, 16:52 UTC Update - We are continuing to monitor delayed ingestion. Sep 10, 16:13 UTC Investigating - We are investigating degraded ingestion in US.
Sep 10, 16:26 UTC Resolved - Issue fully resolved Sep 10, 16:07 UTC Monitoring - Issue were identified and fixes were applied. We continue to monitor situation Sep 10, 15:25 UTC Identified - During migration to new control silo region we started to observe increased number of 500 errors. Migration procedure was rolled back.
Sep 1, 20:29 UTC Resolved - Spans ingestion is now operating normally. Sep 1, 19:31 UTC Update - Latency is almost fully recovered. Sep 1, 18:30 UTC Monitoring - A fix has been implemented, and latency is coming down. Sep 1, 17:50 UTC Update - We are continuing to investigate an issue that affects spans ingestion latency Sep 1, 16:41 UTC Update - We are continuing to investigate an issue that affe…
Sep 1, 11:21 UTC Resolved - This incident has been resolved. Sep 1, 09:03 UTC Update - We're monitoring the implemented fix Sep 1, 08:57 UTC Monitoring - We identified the issue and rolled out a configuration change Sep 1, 08:01 UTC Investigating - We are currently investigating what's going on and why requests timing out
Aug 31, 22:43 UTC Resolved - This incident has been resolved. Aug 31, 21:26 UTC Update - Our systems are recovering from an ingestion backlog, webhooks are currently delayed Aug 31, 21:16 UTC Monitoring - Our systems are recovering from an ingestion backlog Aug 31, 21:12 UTC Investigating - API is experiencing issues Aug 31, 21:02 UTC Monitoring - The fix has been deployed and our ingestion backlo…
Aug 27, 15:41 UTC Resolved - Our latency is now back to normal. Incident resolved Aug 27, 13:58 UTC Update - Our latency is now back to normal. We'll keep monitoring our systems a few more time before declaring the incident as resolved Aug 27, 13:39 UTC Update - We keep monitoring our systems while we recover to our usual latency Aug 27, 12:48 UTC Update - We are continuing to monitor while our sy…
Aug 26, 20:36 UTC Resolved - The issue has been resolved Aug 26, 20:04 UTC Update - We have put a fix into place and are monitoring the situation Aug 26, 18:47 UTC Monitoring - We have put a fix into place and are monitoring the situation Aug 26, 17:53 UTC Identified - We have identified a potential cause of excess traffic and are working on a fix. Aug 26, 16:17 UTC Investigating - We are experien…
Aug 25, 12:38 UTC Resolved - Error alerting is back to normal. Aug 25, 12:27 UTC Identified - We have identified the issue and implemented a fix. Aug 25, 11:09 UTC Update - We are continuing to investigate the issue Aug 25, 09:38 UTC Investigating - We are investigating delays in error alerts in the US region, up to 10 minutes
Aug 25, 03:14 UTC Completed - The scheduled maintenance has been completed. Aug 25, 03:07 UTC Verifying - Verification is currently underway for the maintenance items. Aug 25, 02:59 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Aug 19, 16:37 UTC Scheduled - On August 24 @8pm PST Sentry will be performing maintenance on our central control p…
Aug 18, 22:42 UTC Resolved - mitigated, gocd functional again Aug 18, 22:33 UTC Identified - RCA complete, mitigating now
Aug 18, 22:21 UTC Resolved - The problem has been resolved, all backlogs have cleared and everything is operational again. Aug 18, 22:15 UTC Monitoring - We have implemented a fix and the systems are recovering. We are monitoring to ensure that backlogs are proessed. Aug 18, 21:56 UTC Identified - We have identified the cause and are working to determine a mitigation. This is only affecting US rig…
Aug 13, 22:27 UTC Resolved - All backlogs have been processed. Aug 13, 22:13 UTC Update - We continue to make progress on the spans backlog with an estimated completion time of 15 minutes. Aug 13, 21:22 UTC Update - We're continuing to make progress on the span ingestion backlog in US. Aug 13, 20:32 UTC Update - We've recovered all ingestion latency in DE. In US, errors have processed their backlo…
Aug 12, 22:52 UTC Resolved - Backlogs have been processed Aug 12, 22:43 UTC Identified - We've mitigated the source of ingestion latency and are processing the backlog. Aug 12, 21:58 UTC Investigating - We are currently investigating error ingestion latency in US.
Aug 12, 14:48 UTC Resolved - The backlog has been processed Aug 12, 14:21 UTC Monitoring - We're processing new events and alerts in realtime, and continue to process the tail of the backlog. Aug 12, 13:28 UTC Identified - A fix has been made and the backlog is being processed. Aug 12, 12:36 UTC Update - We've identified the root cause and are actively working on mitigation. Aug 12, 11:45 UTC Inve…
Aug 12, 14:24 UTC Resolved - API performance has returned to expected levels. Aug 12, 13:14 UTC Monitoring - API performance has returned to expected levels. We are continuing to monitor. Aug 12, 12:58 UTC Investigating - We are currently investigating reports of API timeouts for the Sentry API in our US region.
Aug 7, 17:08 UTC Resolved - The ingestion delay is resolved Aug 7, 16:48 UTC Monitoring - Ingestion latency for spans, crons, and logs is back to normal levels. Aug 7, 16:30 UTC Identified - We've put a fix in place and are monitoring. We anticipate catching up to ingestion within 1 hour Aug 7, 16:21 UTC Investigating - We are currently investigating the issue
Aug 6, 11:21 UTC Resolved - We've processed all backlogged events and have fully recovered. Aug 6, 10:50 UTC Monitoring - We're processing new events in real time, and are working through a backlog of events now. Aug 6, 10:25 UTC Update - We've identified the issue causing delays in ingestion and have applied a fix. We'll update again when we return to real time processing. Aug 6, 09:55 UTC Update…
Jul 27, 18:27 UTC Resolved - The backlog has been cleared and ingestion is back to normal. Jul 27, 18:07 UTC Monitoring - We have resolved the incident and are monitoring our ingestion backlog. Jul 27, 16:56 UTC Identified - We have identified the issue and error and profile ingestion is back up in US. Expect delays in ingestion and alerting for errors and profiles. Jul 27, 16:23 UTC Investigating…
Jul 21, 14:14 UTC Resolved - We've burned through our spans backlog. Latency across span consumers are normal. Jul 21, 13:55 UTC Update - We're still burning through our spans backlog Jul 21, 13:04 UTC Monitoring - We've found and fixed the root cause and are burning through our spans backlog. Ingestion should be back to normal in around an hour. Jul 21, 12:20 UTC Update - We are still actively in…
Jul 20, 16:57 UTC Resolved - We have fully burned the backlog of EU error events ingested during the incident. Latency is normal across all event types. Jul 20, 16:09 UTC Update - We are still burning the backlog of EU region error events received during the incident. We are around halfway through the backlog. Jul 20, 14:50 UTC Update - The backlog on attachments in our EU region is fully burned,…
Jul 17, 06:57 UTC Resolved - The webhook backlog is now resolved. Jul 17, 05:49 UTC Update - The webhook backlog is processing. Jul 17, 04:50 UTC Update - The webhook backlog is processing. Jul 17, 03:49 UTC Update - The webhook backlog is processing. Estimated recovery is now 2-2.5 hours. Jul 17, 02:44 UTC Update - The webhook backlog is processing. Jul 17, 01:28 UTC Update - The webhook backlog…
Jul 14, 19:32 UTC Resolved - Feedback ingestion is back to realtime and the backlog has been processed. Jul 14, 18:16 UTC Update - We are continuing to monitor ingestion, and a fix is being deployed to unblock feedback ingestion. Jul 14, 17:40 UTC Monitoring - We are processing errors in realtime, and error backlogs have been processed. We're continuing to monitor the situation before restoring fe…
Jul 13, 14:25 UTC Resolved - Service has been restored. Jul 13, 12:41 UTC Monitoring - Service has been restored and we are actively monitoring the situation Jul 13, 12:24 UTC Identified - We identified the issue and restoring service. Jul 13, 11:51 UTC Investigating - We are investigating ingestion delays.
Jul 13, 02:56 UTC Resolved - Ingestion has caught up and alerts have been restored. Jul 13, 02:01 UTC Monitoring - We have implemented a fix and are monitoring the situation. We expect ingestion to catch up in the next hour, approximately. Jul 13, 01:46 UTC Identified - We have identified the problem and are working to restore ingestion Jul 13, 00:58 UTC Investigating - We are currently investigat…
Jul 11, 03:34 UTC Resolved - We have processed the backlog. Jul 11, 03:28 UTC Monitoring - We have processed the majority of our backlog. We are now recovering the tail of the backlog. Jul 11, 02:37 UTC Update - We have processed the majority of our backlog. We are now recovering an additional 25M error events ranging from 22:20 to 23:20 UTC, which should complete in the next 2 hours Jul 11, 01:46…
Jul 7, 14:36 UTC Resolved - Between July 6th 20:25 UTC and July 7th 12:25 UTC, a wrong configuration change was deployed causing over-sampling of logs, spans and metrics.The configuration change is now reverted and the issue is fixed. Jul 7, 08:49 UTC Investigating - We are currently investigating reports of inconsistent reported number of processed logs in Sentry for the last day
Jul 7, 00:55 UTC Resolved - The backlgo has been consumed and ingestion and alerting has returned to normal operation Jul 6, 23:55 UTC Update - We're continuing to burn down the EU spans, logs, and metrics ingestion backlog. Jul 6, 23:00 UTC Update - We are now processing the backlog in EU. Expect delays in ingestion and alerting for spans, logs, and metrics on the order of hours. We will keep thi…