Skip to content

Do not start nested parallel work from inside a thread-server job - #6260

Open
apullin wants to merge 3 commits into
OpenMathLib:developfrom
apullin:nested-threading
Open

apullin wants to merge 3 commits into
OpenMathLib:developfrom
apullin:nested-threading

Conversation

@apullin

@apullin apullin commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Several level-1 kernels split vectors longer than 10,000 elements across num_cpu_avail(1) threads.

Affected level-1 kernels:

  • x86_64: kernel/x86_64/ddot.c, kernel/x86_64/zdot.c (Haswell, Zen, SkylakeX/Cooperlake/Sapphire Rapids, Sandybridge, Bulldozer family)
  • arm64: kernel/arm64/dot.c, kernel/arm64/zdot_thunderx2t99.c, kernel/arm64/copy_thunderx2t99.c (Neoverse N1/N2/V1/V2, SVE, SME, A64FX, Vortex, ThunderX2/3)

Threaded level-2 drivers call those kernels from inside each worker job.

Affected level-2 callers on columns longer than 10,000 elements (DOT):

  • dgbmv, zgbmv (and sgbmv, cgbmv on arm64) with trans = 'T' or 'C'
  • dtbmv, ztbmv (and stbmv, ctbmv on arm64) with trans = 'T' or 'C'
  • dtpmv, ztpmv (and stpmv, ctpmv on arm64) with trans = 'T' or 'C'
  • zhpmv (and chpmv on arm64)

Affected level-2 callers on strided vectors longer than 10,000 elements on arm64 (COPY_K):

  • sger, dger, cgeru, cgerc, zgeru, zgerc
  • strmv, dtrmv, ctrmv, ztrmv
  • stbmv, dtbmv, ctbmv, ztbmv
  • stpmv, dtpmv, ctpmv, ztpmv
  • ssyr, dsyr, csyr, zsyr, cher, zher
  • ssyr2, dsyr2, csyr2, zsyr2, cher2, zher2
  • sspr, dspr, cspr, zspr, chpr, zhpr
  • sspr2, dspr2, cspr2, zspr2, chpr2, zhpr2

In pthreads builds num_cpu_avail() had no nesting check, so each worker re-entered exec_blas(). Once all pool threads were occupied by the outer call, exec_blas_async() spun forever waiting for idle worker slots while holding exec_queue_lock before signalling any assigned sub-jobs.

This PR tracks whether the current thread is executing a thread-server job and makes num_cpu_avail() return 1 when set. Covered execution paths:

  • pool workers in the pthreads and win32 servers
  • exec_threads() for OpenMP and registered threads callbacks
  • the calling thread while it runs its own share in exec_blas()

Tests

utest/test_nested_thread.c covers:

  • dgbmv('T'), sgbmv('T'), zgbmv('C') with column DOT > 10,000 elements
  • dger, dtbmv with strided COPY > 10,000 elements

With the POSIX pthreads server each case runs in a child process under a timeout so a regression fails cleanly instead of hanging CI.

Verification (before → after on develop)

  • x86_64 (Zen, Haswell, Sandybridge, SkylakeX under Intel SDE): dgbmv('T'), zgbmv('C'), zhpmv, dtpmv('T') hang → pass (170/170 openblas_utest, 1545/1545 openblas_utest_ext)
  • arm64 (NEOVERSEN1 under qemu): all 5 nested_thread cases hang → pass (166/166 openblas_utest)
  • USE_OPENMP=1, USE_THREAD=0, and CMake BUILD_SHARED_LIBS=ON builds: all pass