Skip to main content
Back to timeline
NVIDIA Technical BlogSource publication:

Spark UDFs no longer stall the GPU: agents cut rewrites to hours, one job runs 22x faster

Synopsis

NVIDIA released a set of skills for coding agents that rewrite Apache Spark user-defined functions (UDFs) into GPU-accelerable implementations for the cuDF plugin. UDFs previously broke Spark's declarative execution, forcing the accelerator to convert columnar data to row-wise form and fall back to the CPU, paying a data round trip and forgoing GPU compute. The skills split the rewrite into four verification-driven loops for testing, conversion, benchmarking and optimization, and supply a project template plus cuDF programming guidance; a major consumer technology company used the workflow to convert 30+ production UDFs, with one UDF affecting more than 3,000 jobs showing a 22x speedup in that job, cutting rewrite iteration from days or weeks to hours.

Source-provided article image: How to Rewrite Apache Spark UDFs for GPU Acceleration with AI Agents
Figure 1 · NVIDIA 开发者技术博客

Interpretation

UDFs inside Spark jobs can now be rewritten into GPU-accelerable form by coding agents instead of being rewritten by hand. UDFs were previously opaque to accelerated query engines, and rewriting them required careful API selection, semantic validation and performance measurement, making it a laborious process. A major consumer technology company used the workflow to convert 30+ UDFs in production data pipelines into GPU-accelerated implementations, preserving the behavior of the original CPU UDFs while removing fallbacks from otherwise GPU-accelerated workloads.

The rewrite works because row-by-row UDF logic can be replaced by equivalent operations that act on a whole column at once. The most direct route replaces a simple UDF with an equivalent Spark SQL expression, making the function visible in Spark's expression tree so the cuDF plugin substitutes a GPU implementation during query planning. When SQL primitives are insufficient, a UDF can implement the RapidsUDF interface's evaluateColumnar method, receiving a GPU-resident ColumnVector and composing operations from the Java API for cuDF; highly complex logic can invoke a custom CUDA kernel through JNI.

Agents carry the rewrite because each stage defines both the work to perform and the evidence required before moving on. Rewriting demands familiarity with columnar APIs, which is unfamiliar to practitioners with less experience in vectorized programming. The cuDF skills for Apache Spark decompose the rewrite into four feedback loops for testing, conversion, benchmarking and optimization, and provide a project template with test and benchmark harnesses, output verification routines and build configuration, plus advisory tips on cuDF memory management and configuration.

Perspective

The skills target users of Apache Spark with the NVIDIA cuDF plugin, rewriting UDFs into GPU-accelerable implementations across end-to-end development from testing through conversion and benchmarking to optimization. The production case reported is a major consumer technology company converting 30+ UDFs, with one UDF affecting more than 3,000 jobs showing a 22x speedup in that job. Readers who want to start can install the cuDF skills for Apache Spark from GitHub or with the npx skill installer.

The 22x speedup is a benchmark for one UDF in one job, and the text does not say how widely that figure generalizes to other UDFs or workloads. Rewrites still demand careful API selection, semantic validation and performance measurement, and columnar APIs are unfamiliar to practitioners with less vectorized-programming experience, so the skills offer a guarded workflow rather than an automatic guarantee.