Metaxy: Record-Level Feature Metadata Management for Multimodal ML Pipelines

Software that processes large datasets often repeats expensive computations when any part of the input data or processing logic changes. Metaxy is about perfecting the art of doing nothing: only compute what changed, save time and money, and accelerate exploration. In machine learning pipelines that handle video, audio, and images, these computations run on Graphics Processing Units (GPUs) that cost 10 to 100 times more per hour than standard processors. A small change to one processing step can trigger unnecessary recomputation of unrelated steps, wasting both time and money. Metaxy is a Python library that tracks which specific data records need reprocessing after a change, rather than rerunning entire datasets. It builds a dependency graph that connects individual data fields across processing steps. When a researcher modifies one step, Metaxy identifies exactly which records are affected and which can be skipped. This selective approach lets downstream systems avoid redundant GPU work when a change does not affect a record, while preserving complete lineage for reproducibility. The library treats the metadata store, the compute engine, and the orchestrator as pluggable abstractions, so that any compatible orchestrator can consume Metaxy’s record-level diffs and schedule only the necessary GPU workloads.

Journal of Open Source Software, 11(122), 10449