Part 1 of 2

This resonates strongly. We have had to invent our own fan‑out and fan‑in build orchestration across multiple CircleCI projects, and we have hit exactly this limitation.

We run a top‑level installer workflow in a central repository that triggers pipelines in multiple CircleCI projects backed by different GitHub repositories. Some of these run in parallel, others run sequentially behind a fan‑in gate, and together they produce the dependencies needed for the installer. Once all of those pipelines complete, the workflow then self‑triggers a downstream installer build that consumes the merge modules and localisation artifacts they produced.

In effect, a single logical build fans out into multiple child pipelines and then must deterministically re‑aggregate their outputs, confirming that every job in every pipeline succeeded before it can proceed.

Because there is no native request/response or await capability, we have implemented a fixed three‑job pattern per triggered repository: trigger pipeline, wait for pipeline, then fail if the triggered pipeline was unsuccessful (see attached image).

Making this work requires a surprising amount of custom infrastructure (inspired by the CircleCI “circleback” pattern (https://circleci.com/blog/pipeline-orchestration-circleback/)).

We maintain an internal orb to wrap triggering pipelines, waiting on them, failing on downstream workflow failures, and handling various no‑op and glue jobs. We persist triggered pipeline IDs into the workspace under synthesised keys so downstream jobs can retrieve them later. We pass identifiers such as the triggering pipeline ID and the target approval job name into child pipelines via build parameters so they know which parent job to auto‑approve on completion. Each workflow in the downstream pipeline includes a step whose only purpose is to call back into the parent approval job when it is the final workflow to complete, effectively acting as a YAML‑encoded callback. We also query completed downstream pipelines to verify that all workflows succeeded, since a single failure causes the entire orchestration to fail. For self‑triggered pipelines, we defensively neutralise all unrelated parameters to prevent accidental recursive workflow loops.

At this point, we have effectively rebuilt callbacks and orchestration semantics on top of approval jobs.

Part 2 of 2

There are further workarounds layered on top of this. The system relies on fragile naming conventions for job names, parameter casing, and correlation identifiers, where a single character mismatch can silently wedge the workflow because the callback cannot find its target. We also have custom signalling to mark when the last workflow in a downstream pipeline has finished, regardless of whether it succeeded or failed. Approval jobs are repurposed as RPC return channels, which leaves the CircleCI UI showing “waiting for approval” on jobs no human should ever interact with, but in some failure cases those approvals must be manually cancelled.

The custom orb mostly exists to hide trigger, wait, and fail boilerplate, but compiled configs can end up embedding multiple copies of the orb logic, pushing configuration size limits. Changes to the custom orb code have to be carefully considered as the multiplicative insertions further bloat the compiled config that can cause the configuration size limit to be exceeded!

All of this complexity exists solely to work around the lack of first‑class asynchronous orchestration support.

What would actually solve this is first‑class fan‑out and fan‑in support for multi‑project orchestration, including a native “await external completion” step. That step should wait on one or more external pipelines, return a typed JSON payload that indicates both completion and success, expose that payload directly to downstream jobs, and support timeout and retry. A typed contract between parent and child pipelines would make correlation implicit instead of relying on fragile naming conventions.

Happy to clarify further if useful.