Class: OllamaChat::TTS

Inherits:
Object
  • Object
show all
Includes:
Ollama::Handlers::Concern, KramdownANSI, Utils::UTF8Converter, Utils::ValueFormatter
Defined in:
lib/ollama_chat/tts.rb

Overview

Text-to-Speech handler for streaming audio playback.

This class handles converting streamed text responses into spoken audio using a remote TTS server. It manages parallel TTS requests for text blocks while maintaining strict ordering of audio chunks through an OrderedQueue. A coordinator thread ensures chunks are played in the correct sequence, even when multiple TTS requests complete out of order.

Examples:

tts = OllamaChat::TTS.new(chat: chat, voice: 'alloy')
tts.call(response)

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods included from KramdownANSI

#configure_kramdown_ansi_styles, #kramdown_ansi_parse, #kramdown_markdown_remove

Methods included from Utils::ValueFormatter

#format_bytes, #format_tokens

Methods included from Utils::UTF8Converter

#convert_to_utf8

Constructor Details

#initialize(chat:, voice: nil) ⇒ OllamaChat::TTS

Initializes a new TTS handler.

Sets up the audio player, ordered queue for managing audio chunks, and starts a coordinator thread that processes chunks in order.

Parameters:

  • chat (OllamaChat::Chat) —

    the chat instance for logging

  • voice (String, nil) (defaults to: nil) —

    the voice ID to use for TTS (must be in the list returned by self.voices; falls back to nil if invalid or not provided)



50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
 
# File 'lib/ollama_chat/tts.rb', line 50

def initialize(chat:, voice: nil)
  if voice
    if model = @voice_model = chat.config.voice.model?
      self.class.voices(model:).member?(voice) or voice = nil
    else
      self.class.voices.member?(voice) or voice = nil
      @voice_model = 'tts-1'
    end
  end
  @chat              = chat
  @voice             = voice
  @buffer            = +''
  @audio_player      = OllamaChat::Utils::AudioPlayer.new
  @ordered_queue     = OllamaChat::Utils::OrderedQueue.new
  @id_mutex          = Mutex.new
  @next_id           = 0
  @current_thread_id = 1
  @enqueued_count    = 0
  @finished_count    = 0
  @worker_threads    = []
  @coordinator       = Thread.new { run_coordinator }

  super(output: Tins::NULL)
end

Instance Attribute Details

#voice ⇒ String? (readonly)

The voice attribute reader returns the voice associated with the object.

Returns:

  • (String, nil) —

    the voice ID or nil if not set



79
80
81
 
# File 'lib/ollama_chat/tts.rb', line 79

def voice
  @voice
end

#voice_model ⇒ String? (readonly)

The model attribute reader returns the TTS model ID from config.

Returns:

  • (String, nil) —

    the model ID or nil if not configured



84
85
86
 
# File 'lib/ollama_chat/tts.rb', line 84

def voice_model
  @voice_model
end

Class Method Details

.voices(model: nil) ⇒ Array<String>

Returns a sorted list of available TTS voice IDs.

When model is nil the generic /v1/voices endpoint is queried and a hash array ([{"id": "…"}, …]) is expected. When a model name is given, /v1/audio/voices?model=<id> is queried and a string array (["…", …]) is expected (audio.cpp convention). If the server is unreachable or returns an error, an empty array is returned.

Parameters:

  • model (String, nil) (defaults to: nil) —

    the TTS model ID for per-model queries

Returns:

  • (Array<String>) —

    a sorted list of unique voice identifiers



29
30
31
32
33
34
35
36
37
 
# File 'lib/ollama_chat/tts.rb', line 29

def self.voices(model: nil)
  url = model ? OC::OLLAMA::CHAT::TTS_URL +
                  "/v1/audio/voices?model=#{model}"
              : OC::OLLAMA::CHAT::TTS_URL + '/v1/voices'
  voices = JSON.parse(Excon.get(url, expects: 200).body)["voices"]
  (model ? voices : voices.map { |v| v["id"] }).sort
rescue
  []
end

Instance Method Details

#call(response) ⇒ self

Processes a streaming response chunk for TTS conversion.

Buffers incoming text content and extracts complete blocks (separated by blank lines) for immediate TTS processing. When the response is done, finalizes playback by waiting for all pending TTS requests to complete.

Parameters:

  • response (Ollama::Response) —

    the streaming response object containing text content and a done flag

Returns:

  • (self) —

    returns self for chaining



96
97
98
99
100
101
102
103
104
105
106
107
108
109
 
# File 'lib/ollama_chat/tts.rb', line 96

def call(response)
  if content = response.response || response.message&.content

    @buffer << content

    process_pending_blocks # Process and "read" completed blocks immediately! 🏎️💨
  end

  if response.done
    finalize
  end

  self
end

#enqueue_tts(block) ⇒ Thread (private)

Enqueues a block of text for TTS conversion.

Spawns a background thread that fetches audio chunks from the TTS server and pushes them into the ordered queue along with a :finished sentinel. Each enqueue operation increments the pending thread counter.

Parameters:

  • block (String) —

    the text to convert to speech

Returns:

  • (Thread) —

    the spawned TTS fetch thread



141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
 
# File 'lib/ollama_chat/tts.rb', line 141

def enqueue_tts(block)
  thread_id = @id_mutex.synchronize { @next_id += 1 }
  @chat.log(:info, "TTS: Enqueueing block for synthesis", data: { thread_id:, block: })
  @id_mutex.synchronize { @enqueued_count += 1 }

  @worker_threads << Thread.new do
    chunk_id = 0
    data = {
      model:           @voice_model,
      input:           block,
      response_format: 'pcm', # This is currently ignored by audio.cpp 😿
    }
    if stream = @chat.config.voice.stream?&.enabled
      data |= {
        stream:        true,
        stream_format: @chat.config.voice.stream.format,
      }
    else
      data[:stream] = false
    end
    @voice and data[:voice] = @voice

    @ordered_queue.push([thread_id, chunk_id += 1], :started)

    fetch_tts(data) do |chunk|
      @ordered_queue.push([thread_id, chunk_id += 1], chunk)
    end

    @ordered_queue.push([thread_id, chunk_id += 1], :finished)
  end
end

#fetch_tts(data) {|chunk| ... } ⇒ Object (private)

Fetches TTS audio chunks from the remote server.

Makes a POST request to the TTS API with the given data, streaming audio chunks back via the provided block. Handles connection timeouts and error responses gracefully.

Parameters:

  • data (Hash) —

    the TTS request payload including:

    • :model [String] the TTS model to use (default: "tts-1")
    • :input [String] the text to convert
    • :response_format [String] the audio format (default: "pcm")
    • :stream [Boolean] whether to stream (default: true)
    • :voice [String, nil] optional voice ID

Yields:

  • (chunk) —

    yields each audio chunk (raw PCM bytes)

Yield Parameters:

  • chunk (String) —

    the raw audio data

Raises:

  • (Excon::Error) —

    on HTTP failure (logged and rescued)



286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
 
# File 'lib/ollama_chat/tts.rb', line 286

def fetch_tts(data, &block)
  @chat.log(:info, 'TTS: Requesting audio synthesis', data:)
  body = JSON.dump(data)
  url = OC::OLLAMA::CHAT::TTS_URL + '/v1/audio/speech'
  excon = Excon.new(
    url,
    connect_timeout:  60,
    read_timeout:    360,
    write_timeout:   360,
    logger: @chat.debug ? OllamaChat::Utils::ExconLogger.new(@chat) : nil,
  )
  first_chunk = true
  response_block = -> chunk, _remaining, _total do
    if first_chunk
      first_chunk = false
      # audio.cpp wraps non-streaming responses in a 44-byte RIFF/WAV
      # header regardless of the requested response_format; strip it
      # so the player receives raw PCM.
      chunk = chunk[44..] if chunk.start_with?('RIFF')
    end
    block.call(chunk) unless chunk.empty?
  end
  excon.post(
    body:           ,
    headers:        { 'Content-Type' => 'application/json' },
    expects:        200,
    response_block: ,
  )
rescue => e
  data = {}
  if response = e.ask_and_send(:response)
    result = JSON.parse(response.body) rescue nil
    data[:response] = {
      status: response.status,
      result: ,
    }
  end
  @chat.log(:error, e, data:)
end

#finalize ⇒ Object (private)

Finalizes TTS playback.

Flushes any remaining buffered text, waits for all enqueued TTS threads to complete, stops the audio player, and waits for playback to finish. This ensures all audio is played before the handler shuts down.



178
179
180
181
182
183
184
185
186
187
188
189
190
191
 
# File 'lib/ollama_chat/tts.rb', line 178

def finalize
  @audio_player.start

  unless @buffer.blank?
    enqueue_tts(@buffer.dup)
    @buffer.clear
  end

  # Wait for all enqueued TTS threads to finish
  sleep 0.1 while @id_mutex.synchronize { @finished_count < @enqueued_count }

  @audio_player.stop
  sleep 0.1 while @audio_player.playing?
end

#join_workers ⇒ Object (private)

Joins all worker threads spawned by enqueue_tts.

Useful for tests that need to ensure all background fetch threads have completed before teardown.



197
198
199
 
# File 'lib/ollama_chat/tts.rb', line 197

def join_workers
  @worker_threads.each(&:join)
end

#process_pending_blocks ⇒ Object (private)

Note:

Blocks are identified by double-newline separators, ignoring leading numbered lists like "1."

Processes pending blocks from the buffer.

Extracts complete text blocks (terminated by blank lines \n\n) from the buffer and enqueues them for TTS conversion. This allows for near- real-time speech synthesis as text arrives incrementally.



121
122
123
124
125
126
127
128
129
130
 
# File 'lib/ollama_chat/tts.rb', line 121

def process_pending_blocks
  loop do
    match = @buffer.match(/\A(.{160,})\n\n/m) || @buffer.match(/\A(.*?)\n\n/m) or break
    size = match[0].size

    chunk = @buffer.slice!(0, size).full?(:strip) or next
    chunk = kramdown_markdown_remove(chunk) if @chat.markdown.on?
    enqueue_tts(chunk)
  end
end

#run_coordinator ⇒ Object (private)

Note:

This runs in a background thread started during initialization

The coordinator loop that processes audio chunks in order.

Continuously monitors the ordered queue, popping chunks only when they belong to the current thread (maintaining strict ordering within each TTS request). When a :finished sentinel is encountered, switches to the next thread's chunks if available.



209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
 
# File 'lib/ollama_chat/tts.rb', line 209

def run_coordinator
  loop do
    item = @ordered_queue.peek

    if item.nil?
      sleep 0.01
      next
    end

    (thread_id, _chunk_id), _payload = item

    if @chat.debug
      @chat.log(
        :debug,
        'TTS: Coordinator peeking queue',
        data: {
          id: [ thread_id, _chunk_id ],
          payload: _payload.is_a?(Symbol) ? _payload : :audio_data,
        }
      )
    end

    @current_thread_id ||= thread_id

    if thread_id == @current_thread_id
      _id, payload = @ordered_queue.pop

      case payload
      when :started
        @chat.log(:info, 'TTS: Block synthesis scheduled', data: {
          id: _id,
          finished: @finished_count,
          pending: (@enqueued_count - @finished_count),
        })
      when :finished
        @id_mutex.synchronize do
          @finished_count += 1
          @chat.log(:info, 'TTS: Block synthesis completed', data: {
            id: _id,
            finished: @finished_count,
            pending: (@enqueued_count - @finished_count),
          })
        end
        next_item = @ordered_queue.peek
        @current_thread_id = next_item ? next_item[0][0] : nil
      else
        if @chat.debug
          @chat.log(
            :debug, 'TTS: Feeding audio chunk to player',
            data: { id: _id, size: format_bytes(payload.bytesize) }
          )
        end
        @audio_player << payload
      end
    else
      sleep 0.01
    end
  end
end