You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Publish fat JAR with platform-specific native libraries to Maven Central #33
datafusion-java provides a JVM binding to DataFusion via JNI. To distribute it through Maven Central, we need a packaging strategy that delivers the compiled Rust native library (.so / .dylib / .dll) alongside the Java classes so that consumers get a working artifact with a single dependency declaration — no separate native install step.
Goal
Publish a single artifact to Maven Central that works out of the box on:
Linux x86_64
Linux aarch64
macOS x86_64
macOS aarch64
Windows (x86_64) support is desirable but out of scope for the initial release. The design should leave room to add it later without restructuring.
Proposed approach: single fat JAR
Bundle all platform-specific native libraries in one published JAR, organized by OS/arch under a known resource path:
At runtime, a loader class detects the current OS/arch, extracts the matching library from the JAR to a temp file, and calls System.load() on the absolute path. A System.loadLibrary() attempt should come first so users can override with a system-installed build.
This mirrors the approach used by Apache DataFusion Comet (referenced only as prior art for fat-JAR packaging — datafusion-java is not otherwise related to Comet or Spark). The alternative — publishing one JAR per platform with Maven classifiers — is also viable but pushes platform selection onto consumers and complicates dependency declarations.
Work items
Add a native loader class that detects OS/arch, extracts from the resource path, and loads via System.load(), with a System.loadLibrary() fallback. Include temp-file locking to handle concurrent JVMs.
Set up cross-compilation for the four target triples (Linux x86_64, Linux aarch64, macOS x86_64, macOS aarch64). Options: a CI matrix that produces per-arch artifacts, or Docker + OSXCross for cross-platform builds from a single host.
Wire the build so compiled libraries land at the correct target/classes/... path before mvn package runs.
Add a GitHub Actions release workflow: matrix builds per platform produce native libs as artifacts; a final job assembles them into the resource tree and runs mvn deploy.
Windows x86_64 support. The loader OS enum should already account for .dll and the win32 path segment so this becomes a build-matrix change. Windows complicates temp-file cleanup (can't delete a loaded DLL) — extract to a versioned path and let the OS handle cleanup.
@pgwhalen do you have scripts already that you want to contribute for this? If not, I can do something based on the Comet scripts, but that only supports Linux and macOS - no Windows support.
I have a github workflow that builds a fat JAR for 5 platforms including windows and publishes it to github packages, but I've only tested it for macOS aarch64 and Linux x86_64, and it uses gradle tasks instead of maven. It could be a source of inspiration but I'm not sure there's any point from doing it differently from Comet.
Activity
andygrove commentedon May 20, 2026
@pgwhalen do you have scripts already that you want to contribute for this? If not, I can do something based on the Comet scripts, but that only supports Linux and macOS - no Windows support.
pgwhalen commentedon May 20, 2026
I have a github workflow that builds a fat JAR for 5 platforms including windows and publishes it to github packages, but I've only tested it for macOS aarch64 and Linux x86_64, and it uses gradle tasks instead of maven. It could be a source of inspiration but I'm not sure there's any point from doing it differently from Comet.
Personally I have no use for windows support yet.