Feature Requests

Request for a native CircleCI feature that automatically posts comments on GitHub Pull Requests when CI tests fail, eliminating the need for manual scripting and improving developer workflow. Problem Statement Currently, when tests fail in CircleCI jobs, developers must navigate between GitHub and CircleCI platforms to understand failure details. This context switching creates friction in the development feedback loop and delays issue resolution. Current Pain Points: Test failures are only visible in CircleCI job output No direct connection between PR discussion and test results Manual implementation requires complex scripting with GitHub API Additional maintenance overhead for custom solutions Error handling complexity for API integrations Proposed Solution A first-party CircleCI integration that automatically detects test failures and posts contextual comments directly on associated GitHub Pull Requests. Core Requirements Failure Detection Monitor specific jobs for test failure status Support multiple test frameworks and output formats Configurable failure thresholds and criteria Comment Management Automatically post new comments on test failures Update existing comments for subsequent runs Option to delete/resolve comments when tests pass Configurable Content Test names and failure summaries Links to full CircleCI logs Custom message templates Severity indicators and categorization Authentication & Security Seamless integration with GitHub API authentication Support for project-level GitHub tokens Respect repository permissions and access controls Implementation Options Option 1: Native CircleCI Feature Built-in configuration option in .circleci/config.yml: yamljobs: test: ... existing job config github_pr_comments: enabled: true on_failure: true template: "custom" include_logs: true Option 2: Official CircleCI Orb Reusable orb with standardized commands: yamlorbs: github-pr: circleci/github-pr@1.0.0 jobs: test: steps: - run: npm test - github-pr/comment-on-failure: template: test-failure Benefits Developer Experience Centralized feedback within PR discussions Reduced context switching between platforms Faster identification and resolution of issues Improved visibility for code reviewers Team Productivity Streamlined CI/CD workflow Reduced manual maintenance overhead Consistent failure reporting across projects Better integration with existing GitHub workflows Technical Advantages First-party support with guaranteed reliability Optimized authentication and API usage Built-in error handling and retry logic Consistent with CircleCI's existing GitHub integration Use Cases Primary Use Case Development teams using GitHub PRs with CircleCI want immediate visibility of test failures without leaving the GitHub interface. Secondary Use Cases Code review processes requiring test status context Teams with junior developers who need guided failure information Organizations with compliance requirements for documented test results Projects with multiple test suites requiring consolidated reporting Success Metrics Reduced time from test failure to developer acknowledgment Decreased CircleCI dashboard visits during PR review Improved test failure resolution times Higher adoption of comprehensive testing practices Implementation Priority High Priority - This addresses a common workflow friction point that affects daily developer productivity across many CircleCI users. Alternative Workarounds While custom scripting solutions exist using GitHub API and CircleCI environment variables, they require: Significant setup and maintenance overhead Custom error handling and retry logic Security token management Framework-specific test output parsing A native solution would eliminate these complexities while providing a more reliable and maintainable approach.

To retrieve test results in a particular time frame for a specific job, you need to step through a few endpoints and query job instance by job instance. It would be helpful if a job's test results were aggregated over a particular time frame, such as a month.

This is a feature results to add pagination for flaky test results. At the moment results are capped at 1000 results, but more results would be useful. This is the end point that is used for those test results: https://circleci.com/api/v2/insights/{project-slug}/flaky-tests

Our organization has a lot of test suites, and some of our test suite targets have a lot of tests (thousands). If a test in one of these larger suites errors out, any remaining tests in the suite will report failures, often leading to too many results for the API to return. It would be nice if the API response included a continuation token so we could get the rest of the tests that have been marked as flaky.

We have parallelism set to 28 to run our tests. If a single parallel run fails, we currently have to rerun the entire job. It would be great to just re-run the parallel run that failed. CCI-I-1329

In certain situations, the files uploaded for test results may become corrupted. Currently there isn't always error messages present in the UI and error messages in runner logs are not very descriptive to the root cause of the failure. It would be highly beneficial for the store_test_results step to provide clear error messages when it fails. Currently, the error messages in the runner logs do not effectively indicate that there may be an issue with the file format, which can lead to confusion. Enhancing the error messaging in both the UI and runner logs would greatly improve user experience and facilitate quicker troubleshooting.

The test page currently only displays the failed tests. There should be an option to display all of the passed ones as well so they don't always have to be retrieved from artifacts.

We want to integrate CircleCI CLI binaries into our Docker image. Currently, if the binary download fails, all subsequent test commands fail, which hampers productivity since the test split command downloads the binaries each time. To address this, we propose downloading the CLI binaries into our Docker image. We also request that CircleCI implement a feature to check for the presence of local CLI binaries before downloading. This would prevent unnecessary re-downloads during multiple CircleCI commands in a single session and reduce the risk of failures

When tests are run with parallelism, tasks which do not complete for any reason may not upload test results. When "re-run failed tests only" is run CircleCI retrieves the failed tests, and re-runs those. It does not re-run tests that are missing due to a failed task. This can result in successful tests being reported and tests from failing tasks being ignored, and subsequently potentially causes PRs with missing tests to be merged. Implementing logic to check for failed tasks and running those tests along with tests reported to have failed would prevent accidentally merging PRs where tests are missing.

In order to analyze time-dependent flacky tests, it would be great to have a way to see all failure history with datetimes. In this way it would be possible to download and analyze at what time they fail most often and especially whether this test is flacky through a long period or it just started to fail this time. As an example, I pushed some critical changes and run into a test which failed after my changes. There is no link to previous failures. Insights do not provide any information. Without history I don't know it's me or it is some random stuff. After digging through the code, turned out the bug reproduces with a very small time frame and it's probability is tiny. Would I have the history for this bug, most likely I would found out, that rarely, time to time it failed before. But right now, I can only guess.