Fixes a thread-pool starvation deadlock observed in CI (log excerpts
from build 29344193740, timing out at 6h). setWindEstimation schedules
one per-competitor task onto ThreadPoolUtil.INSTANCE.getDefaultBackgroundTaskThreadPoolExecutor()
which then blocks in maneuverCache.get(competitor, /* waitForLatest */ true).
That get(_, true) is itself waiting on a SmartFutureCache FutureTask
scheduled on the SAME shared executor. In IncrementalMstHmmWindEstimationForTrackedRaceTest
the test flow is
setWindEstimation(...) --> queues N feed-waiter tasks
triggerManeuverCacheRecalculationForAllCompetitors() --> queues M detection tasks
waitForManeuverDetectionToFinish() --> main also blocks in get(_, true)
Once all pool threads are running feed-waiter tasks (or main is
holding one of them), the detection tasks queued behind them can never
start, and the waiters can never complete. Classic pool-starvation
deadlock. The log shows all 3 threads active, 76 tasks queued,
retrying forever.
Fix: give the feed waiters their own small dedicated daemon
ScheduledExecutorService via ThreadPoolUtil.createBackgroundTaskThreadPoolExecutor.
Its pool size defaults to max(2, getReasonableThreadPoolSize()/4) so
we don't hog resources on machines with modest core counts. Because
the waiters and the detection tasks now run on separate pools, the
waiters can block indefinitely without preventing the detection tasks
they wait for from making progress.
Axel Uhl
49e1f4203a