AIBit-Discover Open Source Projects AIBit-Discover Open Source Projects

FluidAudio: Run SOTA Audio AI Models Locally on Apple Devices with CoreML

FluidAudio is a Swift SDK for running state-of-the-art speech-to-text, text-to-speech, speaker diarization, and voice activity detection entirely on-device using Apple's Neural Engine. This post explores its capabilities, architecture, and how to integrate it into your apps.

Why FluidAudio Matters for Apple Developers

For years, integrating advanced audio AI into Apple apps meant one of two things: either sending audio data to a cloud API (with all the latency, privacy, and cost implications) or struggling to run bloated PyTorch models on-device with poor performance. FluidAudio changes that by providing a polished Swift SDK that offloads inference to the Apple Neural Engine (ANE), delivering fast, power-efficient, and fully local audio AI.

The SDK bundles state-of-the-art open-source models (MIT/Apache 2.0 licensed) for:

  • Automatic Speech Recognition (ASR) — batch and streaming transcription
  • Text-to-Speech (TTS) — parallel and streaming synthesis with voice cloning
  • Speaker Diarization — identifying "who spoke when" in conversations
  • Voice Activity Detection (VAD) — detecting speech segments in audio

All models are pre-converted to CoreML and optimized for the ANE, meaning they use minimal CPU and avoid GPU/MPS entirely. This makes FluidAudio ideal for background processing, ambient computing, and always-on workloads.

Getting Started: Installation

FluidAudio integrates via Swift Package Manager. Add it to your Package.swift:

dependencies: [
    .package(url: "https://github.com/FluidInference/FluidAudio.git", from: "0.12.4"),
],

Then add the product to your target:

.product(name: "FluidAudio", package: "FluidAudio")

In Xcode, use File → Add Packages and paste the repository URL. CocoaPods is also supported via pod 'FluidAudio', '~> 0.12.4', though the maintainers recommend cocoapods-spm for better integration.

Core Capabilities in Depth

1. Automatic Speech Recognition (ASR)

FluidAudio's ASR is powered by NVIDIA's Parakeet TDT models, converted to CoreML. Two versions are available:

  • v3: Multilingual, supporting 25 European languages plus Japanese
  • v2: English-only, optimized for highest recall

Performance: On an M4 Pro, the batch transcription achieves a real-time factor of ~190x — meaning one hour of audio is processed in about 19 seconds. Streaming is supported via SlidingWindowAsrManager for real-time use cases.

Quick Start — Batch Transcription:

import FluidAudio

Task {
    let models = try await AsrModels.downloadAndLoad(version: .v3)
    let asrManager = AsrManager(config: .default)
    try await asrManager.loadModels(models)

    // `samples` is [Float] at 16kHz
    let result = try await asrManager.transcribe(samples)
    print("Transcription: \(result.text)")
}

CLI equivalent:

swift run fluidaudiocli transcribe audio.wav
swift run fluidaudiocli transcribe audio.wav --model-version v2  # English-only

2. Speaker Diarization

FluidAudio offers three diarization pipelines, each with different trade-offs:

Offline Pipeline (Pyannote Community-1)

Best for post-processing recorded meetings. Uses power-set segmentation + WeSpeaker embeddings + VBx clustering. Provides high accuracy but is not suitable for real-time use.

let config = OfflineDiarizerConfig()
let manager = OfflineDiarizerManager(config: config)
try await manager.prepareModels()

let samples = try AudioConverter().resampleAudioFile(path: "meeting.wav")
let result = try await manager.process(audio: samples)

for segment in result.segments {
    print("\(segment.speakerId) \(segment.startTimeSeconds)s → \(segment.endTimeSeconds)s")
}

LS-EEND (Streaming)

End-to-end neural diarization with CoreML. Supports up to 10 speakers, 100ms frame updates with 900ms tentative preview. This is the recommended default for online diarization.

Sortformer (Streaming)

NVIDIA's Sortformer model, offering better speaker identity stability than LS-EEND but limited to 4 speakers. Licensed under NVIDIA Open Model License.

When to use which:

  • LS-EEND: Default for live diarization — high speaker capacity, good benchmarks
  • Sortformer: When identity stability matters more than speaker count
  • Pyannote pipeline: When you need modular control over segmentation and clustering stages

3. Voice Activity Detection (VAD)

Powered by Silero VAD, FluidAudio's detector provides both offline segmentation and streaming APIs. The offline mode returns chunk-level probabilities every 256ms hop:

let results = try await manager.process(samples)
for (index, chunk) in results.enumerated() {
    print(String(format: "Chunk %02d: prob=%.3f", index, chunk.probability))
}

For higher-level integration, use segmentSpeech with configurable min speech/silence durations:

var segmentation = VadSegmentationConfig.default
segmentation.minSpeechDuration = 0.25
segmentation.minSilenceDuration = 0.4

let segments = try await manager.segmentSpeech(samples, config: segmentation)

4. Text-to-Speech (TTS)

FluidAudio ships two TTS backends:

Feature PocketTTS Kokoro
Generation Frame-by-frame autoregressive (80ms) Parallel (all frames at once)
Streaming Yes No
Voice cloning Yes (1–30s sample) No
Pronunciation control No Yes (SSML, custom lexicon)
Languages EN, DE, ES, FR, IT, PT EN, ES, FR, HI, JA, PT, ZH, IT

PocketTTS example:

let manager = PocketTtsManager(language: .spanish)
try await manager.initialize()
let audioData = try await manager.synthesize(text: "Hola, mundo.")
try audioData.write(to: URL(fileURLWithPath: "out.wav"))

Kokoro example (ANE-optimized):

let manager = KokoroAneManager()
try await manager.initialize()
let samples = try await manager.synthesize(text: "Hello from FluidAudio.")
// `samples` is 24 kHz mono Float32 PCM

Configuration for Enterprise Deployments

FluidAudio handles model downloads from HuggingFace automatically, but enterprise environments often need customization:

Custom Model Registry

If you have a local mirror or air-gapped environment:

ModelRegistry.baseURL = "https://your-mirror.example.com"

Proxy Configuration

For corporate firewalls:

export https_proxy=http://proxy.company.com:8080
swift run fluidaudiocli transcribe audio.wav

Offline-Only Mode

For privacy-sensitive apps that ship bundled models:

ModelHub.offlineMode = true
let asr = try await AsrModels.load(from: bundledModelURL, configuration: config)

When offline mode is enabled, any network fetch throws DownloadError.networkDisabled instead of attempting a download.

Real-World Impact: The Showcase

The FluidAudio ecosystem has grown rapidly, with dozens of apps using the SDK. Notable examples include:

  • Voice Ink: Local AI for instant, private transcription
  • Spokenly: Mac dictation with real-time and file transcription
  • Slipbox: Privacy-first meeting assistant
  • Talat: AI meeting notes app (featured in TechCrunch)
  • BoltAI: Content creation using Parakeet models
  • Hedy: Real-time, fully on-device meeting coach across iOS, macOS, Android, and Windows

These apps demonstrate the breadth of use cases FluidAudio enables — from simple dictation to complex multi-speaker meeting analysis.

Why the Apple Neural Engine Matters

FluidAudio's decision to target the ANE rather than GPU/MPS is deliberate. The ANE offers:

  • Lower power consumption: Ideal for always-on and background workloads
  • Dedicated hardware: Doesn't compete with GPU for graphics or compute tasks
  • Consistent performance: Predictable inference times across device generations

This makes FluidAudio particularly well-suited for:

  • Meeting transcription running in the background
  • Voice-controlled interfaces on battery-powered devices
  • Real-time captioning and accessibility features

Conclusion

FluidAudio represents a significant step forward for on-device audio AI on Apple platforms. By combining state-of-the-art models with ANE-optimized CoreML inference, it delivers performance that was previously only achievable with cloud APIs — all while keeping data private and local.

Whether you're building a dictation app, a meeting assistant, or a voice-controlled interface, FluidAudio provides the building blocks with minimal code. The SDK's modular design, comprehensive documentation, and active community make it accessible for both indie developers and enterprise teams.

Get started today: Clone the GitHub repo, check the documentation, and join the Discord community for support and updates.

Source

FluidInference/FluidAudio: Frontier CoreML audio models in your apps — text-to-speech, speech-to-text, voice activity detection, and speaker diarization. In Swift, powered by SOTA open source.