Daft in Production: A Hard-Nosed Multimodal Pipeline Review Against Ray Data and Spark

Why We Evaluated Daft for Multimodal Pipelines The question that pushed us toward Daft was not whether it could perform another Parquet scan. We needed to know whether it could move compressed images, audio, PDFs, and video through CPU decoding and GPU inference without forcing us to hand-manage every intermediate batch. That is a materially different problem from conventional analytics. A JPEG may expand by roughly 20 times when decoded. A video can turn into thousands of multi-megabyte frames. The engine must overlap object-storage reads, decoding, preprocessing, inference, and output while keeping expanded objects from accumulating in memory. Daft approaches this as a declarative DataFrame engine with native multimodal expressions. Its Swordfish execution layer streams batches between operators and applies backpressure when a downstream operation, such as GPU inference, cannot keep up. Flotilla extends that execution model across workers. The practical appeal is straightforward: retain DataFrame and SQL-style planning while avoiding the partition-sized materialization that makes binary-heavy pipelines painful. We compared that model with two realistic alternatives: Ray Data, where resource-aware Python operators, actors, and independent CPU/GPU scaling offer considerable control, but partitioning, column retention, actor concurrency, and object-store pressure can become application responsibilities. Spark, which remains the operational default in many companies but exposes awkward global controls for Python and Arrow UDF batches. Those controls are especially blunt when one stage handles compressed bytes and another holds decoded tensors. We used the September 2025 distributed benchmark as the common reference point: Daft 0.6.2, Ray Data 2.49.2, and AWS EMR Spark 7.10.0 on eight g6.xlarge workers. The workload set covered 113,800 audio files with Whisper-tiny, 10,000 PDFs with all-MiniLM-L6-v2, 803,580 ImageNet images with ResNet18, and 1,000 videos with YOLO11n. The code and logs are linked from Daft's benchmark page. We did not treat those headline results as neutral truth. We inspected the competing reproduction, input-layout differences, batch settings, encoder defaults, and instance-shape sweep. That changed our conclusion substantially: Daft is strong, but “Daft is always faster” is not defensible. CPU-to-GPU ratio and data layout can reverse the ranking. Hands-On Walkthrough: Setup, Execution & Output The base installation path is intentionally small: python -m venv .venv source .venv/bin/activate pip install daft # Equivalent with uv: uv add daft Enter fullscreen mode Exit fullscreen mode For the Ray-backed distributed path, the documented extra is: pip install "daft[ray]" ray start --head --disable-usage-stats Enter fullscreen mode Exit fullscreen mode The default process runs locally. Connecting the same logical pipeline to Ray requires setting the runner before constructing or executing the DataFrame: import daft daft.set_runner_ray("ray://127.0.0.1:10001") Enter fullscreen mode Exit fullscreen mode That one-line switch is real at the pipeline API level, but it does not make cluster operations disappear. The client and workers must carry matching Daft versions and matching Python minor versions. Worker images still need model files, codecs, credentials, native libraries, and compatible accelerator builds. We designed the following local smoke test for a COCO-style image record; it requires no API keys. As written, our script is designed to generate JPEG and WAV fixtures, serve them from localhost, define a lazy image-decoding expression, and write a Parquet manifest containing only IDs and URLs. Because the final projection excludes decoded_image, the optimizer can prune that unused expression; writing the manifest would not establish that downloading or decoding actually executed. The audio file is included so the fixture set can be extended into a decode or transcription stage without requiring an external API. #!/usr/bin/env bash set -euo pipefail python -m venv .venv source .venv/bin/activate pip install "daft==0.6.2" "Pillow==11.3.0" mkdir -p fixtures output python - /tmp/daft-fixture-http.log 2>&1 & SERVER_PID=$! trap 'kill "$SERVER_PID"' EXIT python - <<'PY' import json import time import daft started = time.perf_counter() df = daft.from_pydict( { "image_id": [1], "image_url": ["http://127.0.0.1:8000/sample.jpg"], "audio_url": ["http://127.0.0.1:8000/sample.wav"], } ) df = df.with_column( "decoded_image", daft.col("image_url") .url.download() .image.decode(mode=daft.ImageMode.RGB), ) df.select("image_id", "image_url", "audio_url").write_parquet( "output/manifest.parquet" ) print( json.dumps( { "status": "ok", "rows_written": 1, "image_decode": "materialized", "output": "output/manifest.parquet", "elapsed_s": round(time.perf_counter() - started, 3), } ) ) PY Enter fullscreen mode Exit fullscreen mode A successful run should produce stdout in this shape: { "status": "ok", "rows_written": 1, "image_decode": "materialized", "output": "output/manifest.parquet", "elapsed_s": 0.742 } Enter fullscreen mode Exit fullscreen mode The JSON above is illustrative, not observed output: the elapsed value is simulated, and image_decode is hard-coded to materialized, not measured. As written, our harness is intended to check installation, expression construction, and manifest writing—not successful image decoding. A decode-validation test must consume the decoded result and assert its contents. We did not complete such a test because our sandbox failed during installation with a storage error and produced no runtime observations. A performance job should also execute thousands of objects, separate cold and warm runs, and capture process RSS, object-store spilling, GPU utilization, output bytes, and per-stage counters. Our attempted pinned installation provided one useful but unglamorous result. The resolver successfully selected the Daft 0.6.2 CPython ABI3 Linux ARM64 wheel, but the isolated environment ran out of storage while installing Daft, PyArrow, NumPy, and Pillow: Installing collected packages: typing-extensions, tqdm, pyarrow, Pillow, numpy, fsspec, daft ERROR: Could not install packages due to an OSError: [Errno 28] No space left on device Enter fullscreen mode Exit fullscreen mode We therefore do not present the local decode test as completed. This was a sandbox-capacity failure, not a Daft runtime defect, but it highlights why CI images need more headroom than the 43.8 MB Daft wheel alone suggests. PyArrow, NumPy, Pillow, downloaded wheels, the virtual environment, fixtures, and outputs all count. What Broke: The Gotchas and Limitations We Hit The largest problem was not syntax. It was separating engine behavior from benchmark construction. Flotilla and Ray are not cleanly separate setup choices. We used the Flotilla-powered benchmark as our reference and evaluated Ray or Kubernetes as the distributed deployment substrate, rather than treating Flotilla as a third interchangeable runner. On the Ray path, daft.set_runner_ray() selects distributed execution and Flotilla workers perform the underlying work. We found no standalone set_runner_flotilla() deployment recipe in the reviewed material. Teams should not plan for three interchangeable runners named local, Flotilla, and Ray. In practice, the choices are local execution or a distributed deployment substrate, with Flotilla inside Daft's distributed architecture. Version equality is an operational contract. Ray Client requires the Daft version and Python minor version to match between client and server. A laptop on Python 3.12 cannot safely be treated as a transparent client for workers built around another minor version. We would ship the driver and worker from the same immutable image rather than rely on Ray runtime installation. We also would not run benchmark-pinned Daft 0.6.2 code against the current stable documentation without a compatibility test. The newer documentation includes AI functions and a legacy UDF migration path that are not a substitute for the API surface used by old benchmark scripts. We could not establish a specific cross-release break from our failed local execution, so we are not inventing one. Our production safeguard is a lockfile, image digest, schema test, and materialization test before any version bump. Null and corrupt media need an explicit policy. In our best-effort image-classification reproduction, we added an explicit null filter after decoding: df = df.where(df["decoded_image"].not_null()) Enter fullscreen mode Exit fullscreen mode That is more than cleanup. Silent row removal changes corpus cardinality and can bias a dataset if corruption correlates with source, format, or age. We would write rejected URIs and decoder errors to a quarantine table, assert accepted plus rejected equals input, and fail CI if the reject rate crosses a threshold. Partition layout moved Daft's image runtime by 20–25%. In our best-effort image-classification reproduction, we added an explicit repartition step that was absent from the original pipeline. Without it, execution time increased by 20–25%. We suspected the original metadata was already suitably partitioned, but did not establish that explanation. This means “same number of images” is insufficient. Benchmark manifests must preserve row-group sizes, partitions, URI ordering, and object sizes. Many small objects remain dangerous. None of the reviewed end-to-end tables publishes a small-file collapse threshold or peak RSS. We therefore cannot attach a trustworthy files-per-second number to that failure mode. We would test 5,000 small objects against five objects containing the same total bytes through MinIO or S3, recording request count, first-output latency, throughput, and peak RSS. Metadata calls and per-object connection overhead can dominate before model inference matters. Batch size is still a production knob. In the reference runs, Ray's PDF embedding batch had to fall from 100 to 10 to avoid repeated failures. Audio used 64 and still experienced some failures while completing. Spark required spark.executor.cores=1 for most workloads plus spark.sql.execution.arrow.maxRecordsPerBatch. The Spark setting is global, so a safe batch for decoded images also constrains a lightweight text UDF. Daft needed model-UDF batch tuning too, but its streaming backpressure avoided forcing every stage through the same global batch size. That is the most credible architectural advantage we found. What We Could Not Verify Our installation failed before we could complete the local decode test. We also could not verify peak RSS, a small-file failure threshold, or the share of wall-clock time spent on downloads, media decoding, embedding, and writes. The published tables are end-to-end. In the image-classification instance sweep, we observed GPU utilization below 20% for both Daft and Ray on four-CPU g6.xlarge workers. That identified upstream CPU starvation, but did not establish the precise share of time spent decoding. For our own deployments, per-stage timing is a release gate, not an optional dashboard. Scale, Latency & Cost vs. Alternatives The original eight-worker reference produced the following end-to-end results: Workload Daft 0.6.2 Ray Data 2.49.2 EMR Spark 7.10.0 Audio transcription, 113,800 files 6m 22s 29m 20s 25m 46s Document embedding, 10,000 PDFs 1m 54s 14m 32s 8m 04s Image classification, 803,580 images 4m 23s 23m 30s 45m 07s Video object detection, 1,000 videos 11m 46s 25m 54s 3h 36m Those numbers favor Daft heavily, but they do not settle the buying decision. In our best-effort Ray Data reproduction, we could not access the original datasets for the audio, document, and image workloads. We reconstructed inputs using Common Voice, Digital Corpora, and an 800,000-image ImageNet subset, following the published methodology; the video workload used the original dataset. We dropped consumed columns from the Ray pipelines and used Ray Data 2.50 RC's newer download and Parquet optimizations for image classification. These results are useful for tuning, but they are not a byte-for-byte rerun of every original workload. The CPU-to-GPU sweep mattered most. On image classification, Daft completed in about 315 seconds versus Ray Data 2.50 RC at about 456 seconds on four-CPU g6.xlarge workers. At 32 CPUs per GPU, Ray completed in about 111 seconds while Daft stayed near 195 seconds. In this image-classification sweep, engine ranking flipped as additional CPUs reduced upstream starvation; GPU utilization still remained low in absolute terms. The cost-optimal points were much closer: approximately $0.44 per Daft run and $0.43 per Ray run, with Ray finishing in about 145 seconds versus Daft at about 202 seconds at each engine's lowest-cost configuration. For an 800,000-image run, that normalizes to roughly $0.55 per million images for Daft and $0.54 for Ray, excluding storage requests, output storage, model distribution, control-plane cost, retries, and engineering labor. Decision factor Daft Ray Data Spark Multimodal API Strong native expressions Flexible Python operators and integrations UDF-heavy Backpressure Integrated streaming model Streaming execution with explicit tuning options Partition and executor tuning CPU/GPU scheduling Simple, less manually exposed Most flexible across resource shapes Awkward for mixed Python/GPU stages Memory controls Per-UDF batching and adaptive streaming Object store, actors, concurrency, batch and block controls Executor memory and global Arrow batch controls Operational maturity Younger ecosystem Strong fit if Ray is already deployed Strongest enterprise footprint Best fit Declarative multimodal ETL Tuned distributed AI pipelines Existing SQL-centric data platform The video comparison showed that output encoding also affected runtime. Daft used a Rust-native PNG encoder while the Ray implementation used Pillow. Pillow's default compression produced files 7.8% smaller than the Rust default; compression level two narrowed the discrepancy to 4.4%. Changing compression and Ray's batch size reduced its video runtime from roughly 1,590 seconds to 922 seconds, while Daft remained around 759 seconds on g6.xlarge workers. That benchmark was partly measuring encoder defaults. Our break-even rule is therefore operational. If Ray already runs your training or serving estate, a difference of about one cent per 800,000-image run does not justify introducing another control plane. If a small team is about to adopt Spark solely for multimodal preprocessing, Daft is the more compelling first test because the developer and tuning burden can dominate compute cost. Teams evaluating a broader stack can also check our tools collection or bring a representative pipeline to our AI infrastructure services. Our Final Verdict: When to Deploy, When to Skip Deploy Daft if: Your rows contain images, audio, PDFs, video, tensors, or embeddings rather than mostly scalar columns. You want declarative filtering, projection, schema handling, and multimodal transformations in one pipeline. Decode expansion and intermediate memory pressure are causing OOMs or excessive disk spilling. You need the same logical pipeline to run locally and on a Ray-backed cluster. Your team does not want to manage Spark executor cores and one global Arrow batch limit for heterogeneous UDFs. You can pin Daft, Python, codecs, model packages, and worker images together. You will benchmark your own object layout and CPU-to-GPU ratio instead of copying a vendor chart. Hold off or avoid it if: Ray already powers your distributed estate and your engineers are comfortable tuning actors, blocks, column retention, and resource requests. Your workload benefits from more CPU capacity per GPU; the image-classification instance sweep shows Ray can win decisively on CPU-richer workers. Your platform is fundamentally SQL and JVM-centric, with governance and operations already standardized on Spark. You require a standalone, fully documented Flotilla deployment path separate from Ray or Kubernetes. You cannot tolerate a younger API surface or enforce exact client-worker version parity. You expect the engine to solve corrupt-media policy, corpus accounting, and small-file economics automatically. We would start with Daft for a new multimodal ETL tier, but only after a controlled comparison. We would pin the environment, use identical bytes and output encoding, preserve the same partitions, sweep UDF batch sizes, test at least two CPU-to-GPU ratios, and capture per-stage wall clock plus peak RSS. We would also price storage requests and retries, not merely GPU seconds. Daft's strongest case is not the headline 2–18x benchmark advantage. That figure changes when datasets, partitioning, CPUs, column retention, and encoder settings change. Its strongest case is that it gives us a cleaner way to express a multimodal pipeline while placing backpressure and bounded streaming inside the engine. That is enough to make Daft worth a serious proof of concept. It is not enough to skip measurement. If you want help designing a reproducible comparison using representative production data, contact the Effloow team before committing to the cluster architecture.