Magenta RealTime 2 is AWESOME

Magenta RealTime 2 was released with native support to JAX/MLX. The JAX code allows the code to run on TPU/GPUs, while the MLX version allows the code and the exciting apps Google released to run locally on Mac devices.

We are amplifying the surface one can interact with this amazing model with:

• PyTorch-native (GitHub): a Torch port of Magenta RealTime 2 & friends: Depthformer (generates the audio tokens), SpectroStream neural codec (encodes/decodes audio tokens), and MusicCoCa style encoder (text & audio prompts → style tokens).
• Transformers: adaptation of the Torch implementation to Transformers (using the trust_remote_code mechanism). magenta-community/magenta-realtime-2 - magenta-community/magenta-realtime-2-small
• PyTorch web-apps: we've ported the Jam, Studio and Collider apps, originally made to run natively on Mac, to PyTorch. That also allows this apps to run on Hugging Face Spaces' ZeroGPU mechanism

TODO

• ONNX Runtime Web / transformers.js support — WebGPU browser-based demo.