R
Significant degradation
Status: ResolvedWe have been monitoring for a while and services seem to be back. Thank you for your patience
R
API degraded for A100s
Status: ResolvedWe have restarted the affected component and service should be back to normal. Thank you for your patience
R
Degraded scale-out due to failed setups pulling from huggingface
Status: ResolvedWe're looking stable now, and with caching back on. Thanks again for your patience!Affected components H100 Hardware (Operational) L40S Hardware (Operational) A100 Hardware (Operational)
R
Hitting GPU Capacity for H100s creating large queue times for some models
Status: ResolvedThis issue is resolvedAffected components H100 Hardware (Operational) Official Models (Operational)
R
HuggingFace download issues
Status: ResolvedWe have not seen elevated rates of HuggingFace/model setup errors for a couple hours now, so we believe this incident is cleared
R
H100 GPU shortage resulting in high queue times
Status: ResolvedH100 capacity has returned to normal levelsAffected components H100 Hardware (Operational)
R
High contention on H100 hardware
Status: ResolvedWe are back under max capacity for H100 hardware. Thank you for your patience!Affected components H100 Hardware (Operational)
R
Limited H100 capacity
Status: ResolvedThe H100 capacity is back in good health. Thanks for your patience!Affected components H100 Hardware (Operational)
R
We're seeing long setup times and high contention for models on some L40S and H200 clusters.
Status: ResolvedSystem is back to operating normallyAffected components H100 Hardware (Operational) L40S Hardware (Operational)
R
Degraded performance on flux-2-klein-4b
Status: ResolvedThis issue has been resolved and queue times are back to normalAffected components Official Models (Operational)
R
Prediction and Training status updates delayed
Status: ResolvedMessage flows are healthy.Affected components Playground (Operational) H100 Hardware (Operational) Streaming API (Operational) HTTP API (Operational) L40S Hardware (Operational) A100 Hardware (Operational) T4 Hardware (Operational) CPU Hardware (Operational) Home Page (Operational)
R
Constrained H100 capacity
Status: ResolvedH100 hardware contention has resolved. Thank you for your patience!Affected components H100 Hardware (Operational)
R
Constrained capacity for H100 hardware
Status: ResolvedThere is no more contention for H100 hardware. Thank you for your patience!Affected components H100 Hardware (Operational)
R
Degraded A100 hardware
Status: ResolvedAll A100 capacity is back. Thanks for your patience!Affected components A100 Hardware (Operational)
R
A100 capacity unavailable during storage maintenance
Status: ResolvedThe maintenance is complete and all systems are reporting healthy. Thank you for your patience!Affected components A100 Hardware (Operational)