mlx-benchmark
A benchmark suite that measures the runtime of Apple MLX operations and layers across Apple Silicon chips and select CUDA GPUs, compared against equivalent PyTorch operations.
mlx-benchmark measures the runtime of MLX operations on GPU and CPU across the M1 through M5 generations of Apple Silicon chips, alongside equivalent PyTorch operations on the mps, cpu, and cuda backends for GPUs including the RTX 4090, Tesla V100, and A100. By default, MLX operations compiled with mx.compile are included, and a flag can exclude them.
The project produces two kinds of benchmark reports: a detailed benchmark showing the runtime of each individual experiment, and an average runtime benchmark that reports the mean across experiments for easier comparison. Contributors can submit results for a missing device or layer, or resubmit results for an already-listed device when using a newer MLX version, by overriding the existing benchmark table in a pull request.
Running it locally on a Mac requires an osx-arm64 conda environment; other operating systems can only run the PyTorch experiments, on CPU or CUDA, by installing the environment without MLX.