Additional message when the build is killed by CircleCI
Naoya Matayoshi
User recently encountered this issue when debugging infrastructure failures in our builds. After some investigation, we discovered the root cause was OOM (out of memory) issues. User were able to resolve it by increasing the resource_class, but the troubleshooting process was challenging without proper visibility.
The main challenge user faced was that when a job is terminated due to resource exhaustion, the memory utilization metrics are not available because the job never completes. The infrastructure failure message doesn't provide any details about what resource limit was exceeded, making it difficult to determine the appropriate fix.
It would be incredibly helpful if CircleCI could provide:
- A clear indication when a job is terminated due to resource limits (memory, CPU, etc.)
- Information about which specific limit was exceeded
- If possible, the resource usage at the time of termination
ZD#165427
Canny AI
Merged in a post:
Add reason for infrastructure failure
P
Patrick Guinard
As described in https://discuss.circleci.com/t/product-update-infrastructure-failure-badge-in-circleci-ui/47282/8 from the customer standpoint it is unclear why the server (especially when reliant on app.circleci.com) has failed.
We've tracked a few of these back to there being issues with memory but there are a few other causes such as using a port in the range of 32768 - 60999 (https://circleci.com/docs/executor-intro/#port-ranges) which support suggested may be a problem.
As a customer troubleshooting build failures on a system that's been stopped by an outside event it's basic troubleshooting for us to have a reasonable error message to work with.
Nathan Fish
Merged in a post:
Notification for Infra Fail or Pipeline create error
hiroaki nagata
Wanted to bring up a situation that a customer recently experienced regarding an infrastructure failure. They were unaware of the issue until they accessed the project page. It would be very helpful if we could implement a notification system for infrastructure failures or errors during pipeline creation. This could greatly improve the customer experience by keeping them informed.
P
Patrick Guinard
At the time of writing this post metrics are unavailable for infrastructure failed builds so troubleshooting memory and/or CPU usage is also not doable.
N
Nav Ramabadran
Hi CircleCI friends!
We'll take a look at how we might improve this experience. But if anyone else encounters/suspects this might be an issue, might I point you to our Common Issues page: