Prevent a job from being cancelled when the workflow is cancelled
Sometimes a workflow is cancelled. One example of this is if you have the
Auto-cancel redundant builds
option turned on, which is really useful.
The problem comes about when you have a job in a workflow that would cause problems if it were cancelled halfway through. Sometimes infrastructure actions need to be completely finished in one atomic change or else things can be left in undesirable states.
It can be that a previous workflow is running and one of these jobs is running at the time when you decide to push another commit and now the above setting means the workflow gets cancelled and the job along with it.
In other build systems there is an option for 'locking' stages so they cannot be cancelled until they complete, some sort of option for being able to prevent a job from being cancelled (perhaps without at least requiring a manual cancellation in the UI) would be very useful.
Anyone reading this, there are a few approaches that could help here. One, could be to place your IAC pipeline run (in my case, Terraform) behind a hold step. That way, you know when you're running an apply/destroy and take manual care to not cancel that job. Another, but quite specific to a terraform plan, is to not the lock state when you're running a terraform plan. However, be sure to lock the state when you're running an apply/destroy.