Force-cancel workflow on Server UI
We had some very old jobs stuck in "queued" state for months, which were clearly never going to run. These jobs were not present on any nomad nodes, but could be seen via the API and the web UI. When I asked support how to get rid of them, they gave me instructions for how to use kubectl exec to log into a frontend pod and use the REPL to run circle.http.api.admin-commands/force-cancel-build on each job.
That worked, but there should be a button on the frontend web UI to do the same thing.
Jobs on Server can get lost into this kind of zombie state, and it's not practical to expect people to run a REPL directly on the frontend pod to clear them out - and it's also more risky to do so.