Java Thread Pool Size Formula for CPU Bound and IO Work
Size a Java thread pool with the cores times wait over compute formula, split CPU and IO pools, measure, and consider virtual threads.

The most repeated advice about thread pools is a formula: number of cores plus one for CPU work, and some bigger number for I/O. The formulas are fine as starting points and wrong as answers, because the right size depends on how long each task waits, which no formula knows. This post replaces the folklore with a measurement you can run on your own machine. It benchmarks a CPU-bound workload and an I/O-bound one across pool sizes, derives the sizing rule from the numbers, cross-checks it with Little's law, explains why mixed work needs separate pools, lists the limits outside the pool that cap you anyway and shows what changes with virtual threads. Every timing below was measured by the program shown, on a 10-core machine with JDK 17; your numbers will differ, and the shape of the results is the point.
CPU-bound work
A CPU-bound task keeps a core busy for its whole run: parsing, hashing, compressing, rendering. A core can run one such task at a time, so threads beyond the core count only take turns. Measure it:
import java.util.*;
import java.util.concurrent.*;
public class PoolSize {
static long burn() {
long x = 0;
for (int i = 0; i < 40_000_000; i++) {
x += i % 7;
}
return x;
}
static long run(int threads, int tasks, Callable<Object> job) throws Exception {
ExecutorService pool = Executors.newFixedThreadPool(threads);
long t0 = System.nanoTime();
List<Future<Object>> futures = new ArrayList<>();
for (int i = 0; i < tasks; i++) {
futures.add(pool.submit(job));
}
for (var f : futures) {
f.get();
}
long ms = (System.nanoTime() - t0) / 1_000_000;
pool.shutdown();
return ms;
}
public static void main(String[] args) throws Exception {
int cores = Runtime.getRuntime().availableProcessors();
System.out.println("cores=" + cores);
run(cores, cores, PoolSize::burn); // warm-up
System.out.println("CPU-bound, 64 tasks");
for (int t : new int[]{1, cores / 2, cores, cores * 4, 64}) {
System.out.printf(" threads=%-3d %5d ms%n", t, run(t, 64, PoolSize::burn));
}
}
}cores=10
CPU-bound, 64 tasks
threads=1 2161 ms
threads=5 443 ms
threads=10 315 ms
threads=40 316 ms
threads=64 303 msRead the shape. One thread takes 2,161 ms. Five threads cut it to 443 ms, close to the five-fold ideal (2161 / 5 = 432). Ten threads, one per core, reach 315 ms. After that the line goes flat: 40 and 64 threads finish in about the same time, since the cores were already saturated and extra threads add scheduling and cache pressure without adding capacity. The curve has a knee at the core count. That is the whole rule for CPU-bound work: size the pool to the number of cores, Runtime.getRuntime().availableProcessors(), and adding one is a hedge against a thread that briefly stalls on a page fault. Note that the sweet spot is "cores available to the JVM", which inside a container is the CPU limit rounded up, not the host's core count; see MaxRAMPercentage and container awareness for what the JVM sees.
I/O-bound work
An I/O-bound task spends most of its time waiting: on a database, an HTTP call, a disk. While it waits it uses no CPU, so far more tasks than cores can make progress at once. Replace the work with a 50 ms wait and run 400 tasks:
System.out.println("I/O-bound (50 ms wait), 400 tasks");
for (int t : new int[]{cores, 16, 50, 100, 400}) {
System.out.printf(" threads=%-3d %5d ms%n", t, run(t, 400, () -> { Thread.sleep(50); return null; }));
}I/O-bound (50 ms wait), 400 tasks
threads=10 2139 ms
threads=16 1322 ms
threads=50 435 ms
threads=100 224 ms
threads=400 71 msThe pattern is the opposite: time keeps falling as threads grow, almost in proportion. With 10 threads, 400 tasks of 50 ms is 40 rounds of 50 ms, about 2,000 ms, which matches. At 400 threads every task waits at once and the whole run takes about one wait, 71 ms. A pool of core size would have left the CPU nearly idle while tasks queued behind sleeping ones.
The rule that generalizes both results is:
threads = cores x (1 + wait time / compute time)For the CPU task, wait is 0 and the result is the core count. For the I/O task the wait is 50 ms against a compute time of roughly 0.05 ms, a ratio of 1000, and the formula says 10,000 threads, which the benchmark confirms is "as many as there are tasks". In a real service the ratio is nearer 10 to 50: a request that computes for 5 ms and waits 45 ms on a database has a ratio of 9, so 10 cores x 10 = 100 threads. Get the numbers from your own traces, not from a guess, and recompute when a dependency gets slower.
Little's Law as a check
Little's law gives a second route to the same answer from the outside: the average number of tasks in flight equals the arrival rate times the average time in the system.
concurrency = throughput x latencyA service that must handle 500 requests per second with a mean of 80 ms per request has 500 x 0.08 = 40 requests in flight, so it needs at least 40 threads working at once. If your pool has 20, requests queue and latency climbs until the 20 threads are saturated; if it has 200, the extra 160 never get used. Compare the two methods: the wait-ratio formula tells you what the hardware can use, and Little's law tells you what the load demands. The pool should be the smaller of the two, and if the load demands more than the hardware can use, you need more instances, not more threads. The server-side version of this calculation is in tuning Tomcat and Netty threads.
Separate pools
One shared pool for everything lets a slow dependency starve fast work. If a payment API stalls and every thread in the shared pool is waiting on it, a request that only needs the cache cannot get a thread. The standard defense is a bulkhead: a separate, bounded pool per kind of work.
public final class Pools {
public static final ExecutorService CPU = new ThreadPoolExecutor(
cores(), cores(), 0L, TimeUnit.MILLISECONDS,
new ArrayBlockingQueue<>(1_000),
namedFactory("cpu"),
new ThreadPoolExecutor.CallerRunsPolicy());
public static final ExecutorService PAYMENTS = new ThreadPoolExecutor(
20, 20, 0L, TimeUnit.MILLISECONDS,
new ArrayBlockingQueue<>(200),
namedFactory("payments"),
new ThreadPoolExecutor.AbortPolicy());
private static int cores() {
return Runtime.getRuntime().availableProcessors();
}
private static ThreadFactory namedFactory(String prefix) {
var counter = new java.util.concurrent.atomic.AtomicInteger();
return r -> {
Thread t = new Thread(r, prefix + "-" + counter.incrementAndGet());
t.setDaemon(true);
return t;
};
}
}Three details carry the weight. The queue is bounded (ArrayBlockingQueue), because an unbounded queue turns overload into memory exhaustion and latency nobody notices. A rejection policy is chosen deliberately: CallerRunsPolicy slows the producer down when the CPU pool is full, which is backpressure, while AbortPolicy fails fast so the payments caller can degrade instead of waiting. And threads have names, so a thread dump tells you which pool is stuck; reading that dump is covered in finding deadlocks in a thread dump. The mechanics of ExecutorService itself, including shutdown, are in using ExecutorService and thread pools properly. Note that Executors.newFixedThreadPool uses an unbounded queue, which is why production code builds a ThreadPoolExecutor directly, as above. The parameters are specified in the ThreadPoolExecutor Javadoc.
Limits outside the pool
More threads help only until something else saturates, and it is usually not the thread count.
- The database connection pool. 100 threads sharing 10 connections means 90 of them wait for a connection. The pool size is the real concurrency limit for database work, and the formula for it is in the HikariCP sizing formula.
- Downstream rate limits. A partner API that allows 50 concurrent calls makes the 51st thread useless or harmful.
- Memory. Each platform thread reserves a stack, about 1 MiB by default, outside the heap. 1,000 threads is a gigabyte of address space, committed as the stack is used. In a container with a limit, that matters; see Java heap size in a container with real numbers.
- Locks. Threads contending on one lock run one at a time whatever the pool size.
- CPU quota throttling. A container limited to 2 CPUs with 50 busy threads is throttled by the scheduler, and latency spikes appear in bursts.
Find the real bottleneck by watching which resource is saturated when you raise load: CPU at 100 percent means the compute pool is right-sized or too big; threads parked on HikariPool.getConnection means the database pool; busy threads blocked in a socket read means a slow dependency.
Measure
Treat any number here as a hypothesis. A reliable procedure is a ramped load test where you change one setting between runs: record throughput, the 99th percentile latency, CPU use and the pool's queue depth for each. Export the pool's own metrics (active threads, queue size, completed tasks, rejections) with Micrometer, which instruments an ExecutorService in one line:
ExecutorService instrumented = ExecutorServiceMetrics.monitor(registry, Pools.PAYMENTS, "payments");A queue that is always empty and active threads far below the maximum mean the pool is oversized. A queue that is always full and rejections that climb mean it is undersized or the work is too slow. Both are visible in a chart long before they show in user complaints.
Virtual threads
Virtual threads change the I/O half of this post. Because a blocked virtual thread costs a few hundred bytes and no operating-system thread, you do not size a pool for waiting work: you start a thread per task.
try (ExecutorService pool = Executors.newVirtualThreadPerTaskExecutor()) {
for (int i = 0; i < 400; i++) {
pool.submit(() -> { Thread.sleep(50); return null; });
}
}On JDK 21 and later this finishes the 400 waiting tasks in a little over one wait, like the 400-thread row above, without the memory of 400 platform threads. What does not change is the CPU half: compute-bound work is still limited by cores, and the limits outside the pool, the connection pool and downstream rate limits, now matter more because nothing else caps the concurrency. Put a semaphore around those scarce resources. The thread-per-task model means you stop tuning pool sizes for waiting and start limiting the resources behind it.
Where this does not apply
If the service runs one task at a time, such as a scheduled job, there is nothing to size. If your work is a mix that you cannot separate, start with the pool at cores x 2 and measure, because the formula needs a wait ratio you do not have. If you use a framework that owns the pools (a servlet container, a reactive runtime), tune its settings first, not your own executor. And the textbook for all of this remains Java Concurrency in Practice, whose chapter on sizing thread pools states the same formula.


