AIBit-Discover Open Source Projects AIBit-Discover Open Source Projects

Speech Swift: On-Device AI Speech Toolkit for Apple Silicon

Explore Speech Swift, an open-source toolkit for ASR, TTS, speech-to-speech, VAD, and diarization on Apple Silicon using MLX and CoreML.

What Is Speech Swift?

Speech Swift is an open-source AI speech toolkit designed specifically for Apple Silicon (M-series chips and Neural Engine). It brings a comprehensive suite of speech capabilities—automatic speech recognition (ASR), text-to-speech (TTS), speech-to-speech translation, voice activity detection (VAD), speaker diarization, and more—directly to your Mac or iOS device. All processing happens locally, with no cloud dependencies, no API keys, and no data leaving your device.

Why On-Device Speech Matters

Running speech models on-device offers several critical advantages:

  • Privacy: Audio data never leaves your device, eliminating concerns about cloud-based recording or transcription.
  • Latency: Local inference eliminates network round-trips, enabling real-time streaming and interactive voice agents.
  • Offline Capability: Works without an internet connection, ideal for travel, remote areas, or secure environments.
  • Cost: No API usage fees or cloud compute costs.

Key Capabilities and Models

Speech Swift is organized into modular Swift Package Manager (SPM) targets, so you import only what you need. Here's a breakdown of the major capability groups:

Speech-to-Text (ASR)

  • Qwen3-ASR: Speech-to-text supporting 52 languages, with hybrid MLX and CoreML backends. Available in 0.6B and 1.7B parameter sizes.
  • WhisperASR: Native CoreML implementation of OpenAI's Whisper Large-v3 Turbo, optimized for the Neural Engine.
  • Parakeet TDT: NVIDIA's FastConformer with TDT decoder, supporting 25 European languages via CoreML.
  • Omnilingual ASR: Meta's wav2vec2-based model covering 1,672 languages across 32 scripts, with CoreML (300M) and MLX (300M/1B/3B/7B) variants. Achieves 0.28 RTF on iPhone 16 Pro.
  • Streaming Dictation: Real-time dictation with partial results and end-of-utterance detection (Parakeet-EOU-120M), achieving 0.04 RTF on iPhone 16 Pro.
  • Nemotron Streaming: NVIDIA's low-latency streaming ASR with native punctuation and capitalization, available in multilingual (40 language-locales) and English-only variants.

Text-to-Speech (TTS)

  • Qwen3-TTS: High-quality streaming TTS with custom speaker support, 10 languages.
  • CosyVoice TTS: Streaming TTS with voice cloning, multi-speaker dialogue, and emotion tags (9 languages).
  • VoxCPM2: 48 kHz studio-quality TTS with voice cloning and instruction-driven voice design (2B parameters, MLX bf16/int8, 30 languages).
  • IndexTTS2: Native MLX voice cloning from a reference voice (1.5B-class, English and Mandarin).
  • F5-TTS: Zero-shot voice cloning from a short reference clip (DiT flow matching + Vocos, MLX fp16, 24 kHz).
  • Higgs TTS 3: Conversational TTS with zero-shot voice cloning and inline emotion/style/SFX/prosody tags (Boson Higgs TTS 3, Qwen3-4B backbone, MLX bf16, 100+ languages).
  • Kokoro TTS: Lightweight on-device TTS (82M parameters, CoreML/Neural Engine, 54 voices, iOS-ready, 10 languages) with 0.08 RTF on iPhone 16 Pro.
  • VibeVoice TTS: Long-form/multi-speaker TTS for up to 90-minute podcast/audiobook synthesis (Microsoft VibeVoice Realtime-0.5B + 1.5B, MLX, English and Chinese).
  • Magpie TTS: NVIDIA's multilingual TTS (357M parameters, MLX INT8 or CoreML INT8, 9 languages, 5 baked speakers).
  • Supertonic TTS: On-device flow-matching TTS (99M parameters, CoreML/Neural Engine, 31 languages, 10 voices, 44.1 kHz) with 0.15 RTF on iPhone 16 Pro.
  • Chatterbox TTS: Multilingual TTS with zero-shot voice cloning (Resemble AI, MLX fp16, 23 runtime languages).
  • OmniVoice TTS: Non-autoregressive diffusion TTS with zero-shot voice cloning (k2-fsa, Qwen3 backbone, MLX, 600+ languages, Apache-2.0).
  • Indic-Mio: Hindi/Indic TTS with inline emotion markers and optional reference-voice cloning (MLX, 24 kHz).

LLMs and Translation

  • Qwen3Chat: On-device LLM chat with streaming tokens (Qwen3.5-0.8B MLX/CoreML, plus Qwen3 4B and Gemma 4 E2B/E4B MLX backends).
  • FunctionGemma: On-device LLM for structured function/tool calls (Gemma 3 270M, CoreML 8-bit palettized, Neural Engine, ~252 tok/s on M5 Pro, 128 tok/s on iPhone 16 Pro).
  • MADLAD-400: Many-to-many translation across 400+ languages (3B, MLX INT4 + INT8, T5 v1.1, Apache 2.0).

Speech-to-Speech and Voice Agents

  • Hibiki Zero-3B: Streaming speech-to-speech translation (French/Spanish/Portuguese/German to English, MLX INT4 + INT8, Kyutai Moshi/Mimi stack).
  • PersonaPlex: Full-duplex speech-to-speech (7B parameters, audio in → audio out, 18 voice presets).
  • Audio2Face-3D: Speech-driven facial animation for avatars (NVIDIA Audio2Face-3D v2.3 Mark, 301 facial coefficients, MLX).

Enhancement, Separation, and Audio Generation

  • DeepFilterNet3: Real-time noise suppression (2.1M parameters, 48 kHz).
  • Source Separation: Music source separation via HTDemucs (Demucs v4) and Open-Unmix (4 stems: vocals, drums, bass, other).
  • MAGNeT: Text-to-music generation (Meta MAGNeT Small 300M / Medium 1.5B, MLX INT8, 30-second clips at 32 kHz).
  • Stable Audio 3: Text-to-audio/music generation (Stable Audio 3 Medium, MLX INT8/INT4, 44.1 kHz stereo, variable length).
  • FlashSR: Audio super-resolution (FlashSR ICASSP 2025, MLX, 48 kHz mono, 1-step distilled diffusion).

Turn Detection, Diarization, and Speaker Identity

  • Wake-word: On-device keyword spotting (KWS Zipformer 3M, CoreML, 26× real-time, configurable keyword list).
  • VAD: Voice activity detection (Silero streaming, Pyannote offline, FireRedVAD 100+ languages).
  • Speaker Diarization: Who spoke when (Pyannote pipeline, Sortformer end-to-end on Neural Engine).
  • Speaker Embeddings: WeSpeaker ResNet34 (256-dim), CAM++ (192-dim).

Getting Started

Installation via Swift Package Manager

Add the package to your Package.swift:

.package(url: "https://github.com/soniqo/speech-swift", branch: "main")

Then import only the modules you need:

.product(name: "ParakeetStreamingASR", package: "speech-swift"),
.product(name: "SpeechUI", package: "speech-swift"), // optional SwiftUI views

Transcribe Audio in 3 Lines

import ParakeetStreamingASR

let model = try await ParakeetStreamingASRModel.fromPretrained()
let text = try model.transcribeAudio(audioSamples, sampleRate: 16000)

Live Streaming with Partials

for await partial in model.transcribeStream(audio: samples, sampleRate: 16000) {
    print(partial.isFinal ? "FINAL: \(partial.text)" : "... \(partial.text)")
}

SwiftUI Dictation View

import SwiftUI
import ParakeetStreamingASR
import SpeechUI

@MainActor
struct DictateView: View {
    @State private var store = TranscriptionStore()

    var body: some View {
        TranscriptionView(finals: store.finalLines, currentPartial: store.currentPartial)
            .task {
                let model = try? await ParakeetStreamingASRModel.fromPretrained()
                guard let model else { return }
                for await p in model.transcribeStream(audio: samples, sampleRate: 16000) {
                    store.apply(text: p.text, isFinal: p.isFinal)
                }
            }
    }
}

Text-to-Speech

import Qwen3TTS
import AudioCommon

let model = try await Qwen3TTSModel.fromPretrained()
let audio = model.synthesize(text: "Hello world", language: "english")
try WAVWriter.write(samples: audio, sampleRate: 24000, to: outputURL)

Voice Activity Detection

import SpeechVAD

let vad = try await SileroVADModel.fromPretrained()
let segments = vad.detectSpeech(audio: samples, sampleRate: 16000)
for s in segments { print("\(s.startTime)s → \(s.endTime)s") }

Speaker Diarization

import SpeechVAD

let diarizer = try await DiarizationPipeline.fromPretrained()
let segments = diarizer.diarize(audio: samples, sampleRate: 16000)
for s in segments { print("Speaker \(s.speakerId): \(s.startTime)s - \(s.endTime)s") }

Architecture and Design

Speech Swift is split into one SPM target per model, so consumers only pay for what they import. Shared infrastructure lives in AudioCommon (protocols, audio I/O, HuggingFace downloader, SentencePieceModel) and MLXCommon (weight loading, QuantizedLinear helpers, SDPA multi-head attention helper).

Model Weights and Caching

Model weights download from HuggingFace on first use and cache to ~/Library/Caches/qwen3-speech/. You can override this with QWEN3_CACHE_DIR (CLI) or cacheDir: (Swift API). All fromPretrained() entry points also accept offlineMode: true to skip network when weights are already cached.

For users in mainland China (or anywhere huggingface.co is slow/blocked), you can fetch from a mirror by setting HF_ENDPOINT, e.g., export HF_ENDPOINT=https://hf-mirror.com.

Requirements

  • Swift 6+, Xcode 16+ (with Metal Toolchain)
  • macOS 15+ (Sequoia) or iOS 18+
  • Apple Silicon (M1/M2/M3/M4)

The macOS 15 / iOS 18 minimum comes from MLState—Apple's persistent ANE state API used by the CoreML pipelines (Qwen3-ASR, Qwen3-Chat, Qwen3-TTS) to keep KV caches resident on the Neural Engine across token steps.

Performance Benchmarks

Speech Swift includes on-device iPhone 16 Pro CoreML benchmarks (RTF, tokens/s, peak memory) in docs/benchmarks/ios-coreml.md. Key highlights:

  • Parakeet-EOU-120M: 0.04 RTF on iPhone 16 Pro (streaming dictation)
  • Omnilingual ASR (300M): 0.28 RTF on iPhone 16 Pro
  • Kokoro-82M: 0.08 RTF on iPhone 16 Pro (TTS)
  • Supertonic-3 (99M): 0.15 RTF on iPhone 16 Pro (TTS)
  • FunctionGemma (270M): ~252 tok/s on M5 Pro, 128 tok/s on iPhone 16 Pro

Use Cases

  • Voice Agents: Build conversational AI assistants that run entirely on-device with real-time speech-to-speech capabilities.
  • Transcription: High-accuracy, multilingual transcription for meetings, lectures, or media content.
  • Speech Generation: Generate natural-sounding speech for accessibility, content creation, or voice cloning applications.
  • Audio Enhancement: Clean up noisy recordings, separate audio sources, or restore degraded speech.

Community and Resources

  • Full Documentation: soniqo.audio
  • HuggingFace Models: soniqo on HuggingFace
  • Blog: Technical deep-dives on architecture and benchmarks
  • Discord: Community discussion and support

License

Apache 2.0

Conclusion

Speech Swift represents a significant milestone for on-device AI speech processing on Apple Silicon. By combining the power of MLX and CoreML with a modular, developer-friendly Swift package, it enables a new class of privacy-preserving, low-latency speech applications. Whether you're building a voice assistant, transcription tool, or creative audio application, Speech Swift provides the building blocks you need—all running locally on your Mac or iPhone.

Source

soniqo/speech-swift: AI speech toolkit for Apple Silicon — ASR, TTS, speech-to-speech, VAD, and diarization powered by MLX and CoreML