Implement Resource Class as an environment variables.
closed
Oran Wilder
closed
This topic is being closed per the solution provided and inactivity since then.
S
Sebastian Lerner
Hi folks, I'm a product manager at CircleCI. While we do not have an out-of-the-box solution for this yet, I wanted to share a workaround for our Docker resource classes that we've piloted internally that seems to work well for this use case.
For context, the Docker resource classes use cgroups under the hood to manage the resources granted to the container (as Kyle points out in his post above). It is possible to use a cgroup-aware mechanism to calculate the number of CPUs. Below is an example for how to do it in Node:
const fs = require('fs/promises');
const os = require('os');
async function cgroup_cpu_count() {
const quota_s = await fs.readFile('/sys/fs/cgroup/cpu/cpu.cfs_quota_us')
const period_s = await fs.readFile('/sys/fs/cgroup/cpu/cpu.cfs_period_us')
const quota = parseInt(quota_s)
const period = parseInt (period_s)
return quota/period
}
async function cpu_count() {
try {
const cgroups = await fs.stat('/proc/self/cgroup')
return cgroup_cpu_count()
} catch {
return os.cpus().length
}
}
cpu_count().then(console.log)
P
Paul Hawxby
Yes, very much needed. Other CI runners do not have this issue.
The only suggestion I would make is don't make this vendor or OS specific so that should other CI runners need the same libraries don't need to account for every flavour of the env var.
Windows already has
NUMBER_OF_PROCESSORS
, just implement that env var for other operating systems. As far as I can tell there's no examples of one for memory, so how about TOTAL_MEMORY
.Eric Mueller
I would love this (and also just a
CIRCLE_RESOURCE_CLASS
variable - I'm tagging metrics with that information). An alternative implementation would be to expose it via some kind of circleci context
command on the cli tool.E
Emil Ong
This feature would be incredibly helpful with the issue that https://www.npmjs.com/package/terser-webpack-plugin with determining core count for parallelism.
j
john.ruble@atomicobject.com
This happened to me today with a React Native (metro) build, which scaled out to 36 workers and starved.
Circle-specific env vars would be helpful, but what we really need os a common convention across CI providers for CPUs and RAM allocated to the container, so that projects like jest-worker can interpret them.
j
john.ruble@atomicobject.com
Background: our test framework automatically scaled out to 36 instances because that's how many hardware threads the standard Linux interfaces (e.g. /proc/cpuinfo) reported. I would expect similar behavior from make -j, and many other tools that customers run in CI.
Short of fixing those interfaces, it would be great if we could get this information from env vars. Eventually this could make it into common frameworks, default circle configs, etc., so that fewer customers run into the issue.