Oct 2, 01:32 UTC Resolved - the maintenance is complete and everything has stabilized Oct 2, 01:15 UTC Identified - infrastructure maintenance is having a larger impact on availability of PyPI's backends than anticipated. we are working to stabilize.
Oct 1, 15:35 UTC Completed - The scheduled maintenance has been completed. Sep 25, 15:35 UTC In progress - We will leave this maintenance window notice open for visibility. Sep 25, 15:34 UTC Scheduled - The deprecated XML-RPC `browse()` method has been removed and turned into an error response.After being abused for multiple days and triggering long outages for all users, this deprecated method ha…
Oct 1, 11:18 UTC Resolved - All services are restored. Oct 1, 11:13 UTC Monitoring - Multiple python.org services were impacted by expiration of internal TLS certificates caused by use of APIs deprecated by a recent library update. We have resolved the issue and are monitoring as services come back online after renewing their internal certificates.
Sep 29, 01:24 UTC Resolved - This incident has been resolved. Sep 28, 13:48 UTC Monitoring - We have resolved the issue and are monitoring for changes. Sep 28, 13:47 UTC Update - We are continuing to investigate this issue. Sep 28, 12:42 UTC Investigating - We are currently investigating this issue.
Sep 23, 17:05 UTC Resolved - A fix has been implemented and the incident is closed. Sep 23, 14:06 UTC Monitoring - The traffic has subsided and the cluster is stable. A performance fix has been drafted. Sep 23, 11:02 UTC Investigating - We are currently investigating this issue.
Feb 26, 23:26 UTC Resolved - The incident has been resolved. Feb 26, 20:40 UTC Update - The root cause has been identified as an erroneous scaling operation that affected some application workloads. All services have been restored and are coming back online. We are monitoring to confirm full recovery. Feb 26, 20:04 UTC Investigating - We are currently investigating this issue
Feb 19, 23:59 UTC Resolved - This incident has been resolved. Feb 19, 22:56 UTC Monitoring - A fix has been implemented and we are monitoring the results. Feb 19, 22:46 UTC Investigating - We are currently investigating this issue.
Jan 7, 23:00 UTC Resolved - This incident has been resolved. Jan 7, 22:59 UTC Monitoring - We identified a database lock causing contention and stopped all upload backends to release all locks and are monitoring for recovery. Jan 7, 22:42 UTC Investigating - We are investigating why PyPI uploads are failing.
Nov 12, 16:26 UTC Resolved - This incident has been resolved. Nov 12, 15:50 UTC Monitoring - A fix has been implemented and we are monitoring the results. Nov 12, 15:46 UTC Investigating - We are currently observing a slowdown in background requests processing for events on PyPI.orgThis could lead to stale web pages and API responses as the events system is not able to purge the caches.Our team is…
Jul 29, 19:58 UTC Resolved - This incident has been resolved. Jul 29, 14:41 UTC Monitoring - The PyPI search index has degraded and is not returning results.A reindexing job has begun, and results will be available once it completes.
Jun 11, 19:23 UTC Completed - This maintenance is complete! All metrics have returned to baseline. Jun 11, 19:16 UTC Verifying - All database tasks are complete and we are now monitoring for stability post-upgrade. Services should be back online in full. Jun 11, 19:10 UTC Update - The core upgrade is complete, we are awaiting the completion of post-upgrade statistics updates which are taking longe…
May 27, 12:53 UTC Resolved - We have not seen a return of the traffic that caused this incident after removing the temporary block. We will open a new incident if we need to take action again. May 25, 12:16 UTC Update - We have removed the temporary block and are continuing to monitor. May 24, 23:54 UTC Monitoring - Our block is in place and systems are stable, but we are not satisfied with having…
Apr 24, 12:18 UTC Resolved - This incident is resolved. We're going to do some further investigation to understand what led to the lock contention. Apr 24, 12:09 UTC Monitoring - The locking process has been terminated and we're monitoring as things recover. Apr 24, 12:09 UTC Identified - We've identified an issue with database contention arising from locking. Apr 24, 12:02 UTC Investigating - We'…
Apr 8, 11:00 UTC Resolved - This incident has been resolved. Apr 8, 10:31 UTC Update - Errors have subsided, and we are monitoring as things continue to recover. Apr 8, 10:26 UTC Monitoring - We've increased capacity in the worker pools that service /simple and /json and are monitoring as things recover. Apr 8, 10:23 UTC Identified - We are working to resolve an outage impacting our /simple and /j…
Mar 11, 13:39 UTC Resolved - This incident has been resolved. Mar 11, 13:05 UTC Update - We are continuing to monitor for any further issues. Mar 11, 12:50 UTC Monitoring - Our CDN provider has reported that the node in question has been removed from service. We are monitoring for continued reports. Mar 11, 12:47 UTC Identified - This issue appears to be isolated to a single edge node in our CDN p…
Dec 13, 15:09 UTC Resolved - This incident has been resolved. Dec 13, 14:58 UTC Monitoring - We have begun seeing recovery across all services as of 14:44 UTC, we are monitoring for stability and ensuring all services are healthy. Dec 13, 14:45 UTC Update - We are continuing to work on a fix for this issue. Dec 13, 14:23 UTC Update - We are continuing to work on a fix for this issue. Dec 13, 14:00…
Nov 13, 19:32 UTC Completed - The scheduled maintenance has been completed. Nov 13, 19:30 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Nov 13, 19:15 UTC Scheduled - PyPI will be undergoing a brief period of database maintenance to ensure that a set of migrations can complete without contention.During this time all PyPI backends will be tem…
Aug 21, 20:23 UTC Resolved - This incident has been resolved. Aug 21, 19:53 UTC Identified - An error serving /simple requests has caused timeouts for some requests. We have identified the issue and are working on a revert.
Aug 9, 19:26 UTC Resolved - Uploads are now being accepted properly. Aug 9, 19:23 UTC Monitoring - A fix has been implemented and we are monitoring the results. Aug 9, 19:08 UTC Identified - The issue has been identified and a fix is being implemented. Aug 9, 18:22 UTC Investigating - We are currently investigating an issue with PyPI uploads causing uploads to fail.
Jul 31, 03:34 UTC Resolved - This incident has been resolved. Jul 31, 03:21 UTC Investigating - We are currently investigating an issue with the PyPI backend due to a recent deploy.
Jul 12, 16:18 UTC Resolved - This maintenance is complete. Jul 12, 14:28 UTC Identified - We are performing upgrades to PyPI's database storage which may impact performance and availability of the service as they rollout.
Jun 8, 14:52 UTC Resolved - All maintenance is complete and telemetry around database and application performance have stabilized. This incident is complete. Jun 8, 14:42 UTC Update - All upgrade steps are complete and performance is stabilizing. A final maintenance task is taking place in the background which will minimally impact performance until it is complete. Jun 8, 14:26 UTC Update - We are…
Jun 5, 16:05 UTC Resolved - This incident has been resolved. Jun 5, 15:57 UTC Monitoring - A cascading failure due to an expired certificate led to this outage. The certificate has been expired and systems are coming back online. Jun 5, 15:38 UTC Identified - An outage in our infrastructure is impacting availability of us.pycon.org and python.org
Mar 28, 12:56 UTC Resolved - This incident has been resolved. Mar 28, 02:16 UTC Identified - We have temporarily suspended new project creation and new user registration to mitigate an ongoing malware upload campaign.
Mar 12, 20:41 UTC Resolved - This incident has been resolved. Mar 12, 20:19 UTC Monitoring - Workers have been restarted and are clearing backlog of enqueued tasks. Mar 12, 20:00 UTC Identified - A maintenance event requires us to pause our workers, which will delay cache purges and email delivery. We anticipate this will persist for up to 30 minutes.
Feb 12, 21:06 UTC Resolved - This incident has been resolved. Feb 12, 18:07 UTC Monitoring - Our new internal CA has been rolled out across the infrastructure and we are monitoring for remaining service impact. Feb 12, 18:00 UTC Identified - Our internal Certificate Authority certificate has expired causing cascading outages across multiple services. An updated CA has been created and we are worki…
Jan 19, 12:18 UTC Resolved - Maintenance is complete. Jan 19, 12:04 UTC Monitoring - Maintenance and migration is complete. DNS may take some time to propagate. We will monitor as that occurs. Jan 19, 11:31 UTC Identified - bugs.python.org (and bugs.jython.org, and issues.roundup-tracker.org) is currently under maintenance.
Jan 2, 15:59 UTC Resolved - New user registrations are now permitted. Dec 27, 20:11 UTC Monitoring - New user registration on PyPI is temporarily suspended. The volume of malicious users and malicious projects being created on the index in the past week has outpaced our ability to respond to it in a timely fashion, especially with multiple PyPI administrators on leave.
Dec 4, 13:47 UTC Resolved - This incident has been resolved. Nov 24, 23:14 UTC Monitoring - New user registration on PyPI is temporarily suspended. The volume of malicious users and malicious projects being created on the index in the past week has outpaced our ability to respond to it in a timely fashion, especially with multiple PyPI administrators on leave.
Oct 17, 22:44 UTC Resolved - This incident has been resolved. Oct 17, 19:44 UTC Identified - A burst balance for our cloud storage disks that host our databases was exhausted. We are going to migrate to storage that will allow us to respond by adding more IOPS. Oct 17, 19:32 UTC Investigating - We are investigating and responding to widespread issues with databases in our infrastructure showing po…