Module: OllamaChat::SourceFetching

Included in:
Chat
Defined in:
lib/ollama_chat/source_fetching.rb

Overview

The SourceFetching module encapsulates methods for retrieving content from different source types including URLs, file paths, and shell commands. It handles the logic for determining the appropriate fetching method based on the source identifier and processes the retrieved content through specialized parsers depending on the content type. The module also manages image handling, document importing, summarizing, and embedding operations while providing error handling and debugging capabilities.

Examples:

Fetching content from a URL

chat.fetch_source('https://example.com/document.html') do |source_io|
  # Process the fetched content
end

Importing a local file

chat.fetch_source('/path/to/local/file.txt') do |source_io|
  # Process the imported file content
end

Executing a shell command

chat.fetch_source('!ls -la') do |source_io|
  # Process the command output
end

Instance Method Summary collapse

Instance Method Details

#add_image(images, source_io, source) ⇒ Object

Adds an image to the images collection from the given source IO and source identifier.

This method takes an IO object containing image data and associates it with a source, creating an Ollama::Image instance and adding it to the images array.

Parameters:

  • images (Array) —

    The collection of images to which the new image will be added

  • source_io (IO) —

    The input stream containing the image data

  • source (String, #to_s) —

    The identifier or path for the source of the image



110
111
112
113
114
115
 
# File 'lib/ollama_chat/source_fetching.rb', line 110

def add_image(images, source_io, source)
  feedback("Adding #{source_io&.content_type} image #{source.to_s.inspect}.", type: :info)
  log(:info, "Image added", data: { source: source.to_s, content_type: source_io&.content_type })
  image = Ollama::Image.for_io(source_io, path: source.to_s)
  (images << image).uniq!
end

#bulk_embed_sources(sources_tags) ⇒ Array<String>

Embeds multiple sources concurrently using a bounded thread pool.

Parameters:

  • sources_tags (Hash{String => Array}) —

    mapping of source identifier to the tags array to apply to that source

Returns:

  • (Array<String>) —

    results from each successful embed call



305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
 
# File 'lib/ollama_chat/source_fetching.rb', line 305

def bulk_embed_sources(sources_tags)
  prompt = prompt(:embed)
  return sources_tags.map { |s, tags| embed(s, prompt:, tags:) }.compact if
    sources_tags.size <= 1 or config.embedding.concurrency <= 1

  results = []
  mutex   = Mutex.new
  Tins::Limited.new(config.embedding.concurrency, name: 'embed').process do |l|
    sources_tags.each do |s, tags|
      l.execute do
        r = embed(s, prompt:, tags:)
        mutex.synchronize { results << r } if r
      end
    end
    l.stop
  end
  results
end

#embed(source, tags: [], prompt: prompt(:embed)) ⇒ String?

Embeds content from the specified source.

This method fetches content from a given source (command, URL, or file) and processes it for embedding using the embed_source method.

Parameters:

  • source (String) —

    The source identifier which can be a command, URL, or file path

  • tags (Array<String>) (defaults to: []) —

    Tags to apply to the embedded source

  • prompt (#to_s) (defaults to: prompt(:embed)) —

    The prompt template to format the result with (defaults to prompt(:embed))

Returns:

  • (String, nil) —

    The formatted embedding result or summary message, or nil if the operation fails



288
289
290
291
292
293
294
295
296
297
298
 
# File 'lib/ollama_chat/source_fetching.rb', line 288

def embed(source, tags: [], prompt: prompt(:embed))
  prompt = prompt.to_s
  embedding.on? or return
  fetch_source(source) do |source_io|
    content = parse_source(source_io)
    content.present? or return
    source_io.rewind
    embed_source(source_io, source, tags:) or return
  end
  prompt.to_s.named_placeholders_interpolate({ source:, collection: })
end

#embed_source(source_io, source, tags: [], count: nil) ⇒ Array, ...

Embeds content from the given source IO and source identifier.

This method processes document content by splitting it into chunks using various splitting strategies (Character, RecursiveCharacter, Semantic) and adds the chunks to a document store for embedding.

Parameters:

  • source_io (IO) —

    The input stream containing the document content to embed

  • source (String, #to_s) —

    The identifier or path for the source of the content

  • count (Integer, nil) (defaults to: nil) —

    An optional counter for tracking processing order

Returns:

  • (Array, String, nil) —

    The embedded chunks or processed content, or nil if embedding is disabled or fails



211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
 
# File 'lib/ollama_chat/source_fetching.rb', line 211

def embed_source(source_io, source, tags: [], count: nil)
  embedding.on? or return parse_source(source_io)
  unless @documents.source_modified?(source)
    infobar.puts "Source #{source.to_s.inspect} already up-to-date. => Skipping."
    log(:info, "Source up-to-date", data: { source: source.to_s })
    return
  end
  text = parse_source(source_io) or return
  splitter_config = config.embedding.splitter
  inputs = nil
  case splitter_config.name
  when 'Character'
    splitter = Documentrix::Documents::Splitters::Character.new(
      chunk_size: splitter_config.chunk_size,
    )
    inputs = splitter.split(text)
  when 'RecursiveCharacter'
    splitter = Documentrix::Documents::Splitters::RecursiveCharacter.new(
      chunk_size: splitter_config.chunk_size,
    )
    inputs = splitter.split(text)
  when 'Semantic'
    splitter = Documentrix::Documents::Splitters::Semantic.new(
      ollama: embedding_ollama, model: config.embedding.model.name,
      chunk_size: splitter_config.chunk_size,
    )
    inputs = splitter.split(
      text,
      breakpoint: splitter_config.breakpoint.to_sym,
      percentage: splitter_config.percentage?,
      percentile: splitter_config.percentile?,
    )
  end
  inputs or return
  m = "Embedded #{italic { source_io&.content_type }} document "\
    "#{source.to_s.inspect} in collection #{collection.to_s.inspect}."
  log(:info, "Source embedded", data: {
    source: source.to_s,
    content_type: source_io&.content_type&.to_s,
    collection: collection.to_s,
    chunks: inputs.size
  })
  if count
    infobar.puts '%u. %s' % [ count, m ]
  else
    infobar.puts m
  end
  source  = source.to_s
  command = false
  inputs  = inputs.map { |chunk| "Source: #{source}\n#{chunk}" }
  if source.start_with?(?!)
    source = truncate(
      source[1..-1].gsub(/\W+/, ?_),
      length: 10
    )
    command = true
  end
  if !command
    @documents.source_update(inputs, source:, tags:, batch_size: config.embedding.batch_size?)
  else
    @documents.add(inputs, source:, tags:, batch_size: config.embedding.batch_size?)
  end
end

#fetch_source(source, check_exist: false) {|tmp| ... } ⇒ Object

The fetch_source method retrieves content from various source types including commands, URLs, and file paths. It processes the source based on its type and yields a temporary file handle for further processing.

Parameters:

  • source (String, #to_path) —

    the source identifier which can be a command, URL, or file path

Yields:

  • (tmp) 


35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
 
# File 'lib/ollama_chat/source_fetching.rb', line 35

def fetch_source(source, check_exist: false, &block)
  confirm_notification = false
  source = source.ask_and_send_or_self(:to_path).to_s
  case source
  when %r{\A!(.*)}
    command = $1
    OllamaChat::Utils::Fetcher.execute(command) do |tmp|
      log(:info, "Command executed", data: { command:, bytes: format_bytes(tmp.size) })
      block.(tmp)
    end
  when %r{\Ahttps?://\S+}
    url = $&
    if captions = OllamaChat::Invidious.fetch_video_info(url, chat: self)
      block.(captions)
    else
      get_url(url, cache:) do |tmp|
        log(:info, "URL fetched", data: { url:, bytes: format_bytes(tmp.size) })
        block.(tmp)
      end
    end
  when %r{\Afile://([^\s#]+)}
    filename = $1
    filename = URI.decode_www_form_component(filename)
    filename = File.expand_path(filename)
    check_exist && !File.exist?(filename) and return
    fetch_source_as_filename(filename, &block)
  when  %r{\A((?:\.\.|[~.]?)/(?:\\ |\\|[^\\]+)+)}
    filename = $1
    filename = filename.gsub('\ ', ' ')
    filename = File.expand_path(filename)
    check_exist && !File.exist?(filename) and return
    fetch_source_as_filename(filename, &block)
  when %r{\A"((?:\.\.|[~.]?)/(?:\\"|\\|[^"\\]+)+)"}
    filename = $1
    filename = filename.gsub('\"', ?")
    filename = File.expand_path(filename)
    check_exist && !File.exist?(filename) and return
    fetch_source_as_filename(filename, &block)
  else
    raise "invalid source #{source.inspect}"
  end
rescue OllamaChat::HTTPError => e
  msg = "Fetching source #{source.to_s.inspect}: #{e.class} #{e}"
  feedback(msg, type: :warn)
  log(:warn, msg, data: { source: source.to_s, status: e.status })
  confirm_notification = true
  msg = OllamaChat::Utils::Fetcher::ResponseMetadata.failed(msg)
  msg.response_status = e.status
  msg.response_reason = e.message
  block.(msg)
  msg
rescue => e
  msg = "Fetching source #{source.to_s.inspect}: #{e.class} #{e}"
  feedback("#{msg}\n#{e.backtrace * ?\n}", type: :warn)
  log(:error, msg, data: { source: source.to_s, error: e.class })
  confirm_notification = true
  msg = OllamaChat::Utils::Fetcher::ResponseMetadata.failed(msg)
  block.(msg)
  msg
ensure
  if confirm_notification
    confirm?(prompt: "\n⏎  Press any key to continue (%s). ", output: STDERR, timeout: 3)
  end
end

#fetch_source_as_filename(filename) {|file| ... } ⇒ nil, Object (private)

Reads a file and extends it with header extension metadata. It then yields the file to the provided block for processing. If the file does not exist, it outputs an error message to standard error.

Parameters:

  • filename (String) —

    the path to the file to be read

Yields:

  • (file) —

    yields the opened file with header extension

Returns:

  • (nil) —

    returns nil if the file does not exist

  • (Object) —

    returns the result of the block execution if the file exists



336
337
338
339
340
341
 
# File 'lib/ollama_chat/source_fetching.rb', line 336

def fetch_source_as_filename(filename, &block)
  OllamaChat::Utils::Fetcher.read(filename) do |tmp|
    log(:info, "File read", data: { path: filename, bytes: format_bytes(tmp.size) })
    block.(tmp)
  end
end

#import(source, language: nil) ⇒ String?

Imports content from the specified source and processes it.

This method fetches content from a given source (command, URL, or file) and passes the resulting IO object to the import_source method for processing.

Parameters:

  • source (String) —

    The source identifier which can be a command, URL, or file path

Returns:

  • (String, nil) —

    A formatted message indicating the import result and parsed content, # or nil if the operation fails



149
150
151
152
153
154
155
 
# File 'lib/ollama_chat/source_fetching.rb', line 149

def import(source, language: nil)
  fetch_source(source) do |source_io|
    content = import_source(source_io, source, language:) or return
    source_io.rewind
    content
  end
end

#import_source(source_io, source, language: nil) ⇒ String

The import_source method processes and imports content from a given source, displaying information about the document type and returning a formatted string that indicates the import result along with the parsed content.

Parameters:

  • source_io (IO) —

    the input stream containing the document content

  • source (String) —

    the source identifier or path

Returns:

  • (String) —

    a formatted message indicating the import result and the parsed content



126
127
128
129
130
131
132
133
134
135
136
137
 
# File 'lib/ollama_chat/source_fetching.rb', line 126

def import_source(source_io, source, language: nil)
  source        = source.to_s
  document_type = source_io&.content_type.full? { |ct| italic { ct } + ' ' }
  feedback("Importing #{document_type}document #{source.to_s.inspect} now.", type: :info)
  log(:info, "Source imported", data: { source:, content_type: source_io&.content_type })
  source_content = parse_source(source_io, language:)
  <<~EOT
    Imported #{source.inspect}:

    #{source_content}
  EOT
end

#summarize(source, words: nil, instruction: nil) ⇒ String?

Summarizes content from the specified source.

This method fetches content from a given source (command, URL, or file) and generates a summary using the summarize_source method.

Parameters:

  • source (String) —

    The source identifier which can be a command, URL, or file path

  • words (Integer, nil) (defaults to: nil) —

    The target number of words for the summary (defaults to 100)

  • instruction (String, nil) (defaults to: nil) —

    An optional instruction to focus the summary

Returns:

  • (String, nil) —

    The formatted summary message or nil if the operation fails



191
192
193
194
195
196
197
 
# File 'lib/ollama_chat/source_fetching.rb', line 191

def summarize(source, words: nil, instruction: nil)
  fetch_source(source) do |source_io|
    content = summarize_source(source_io, source, words:, instruction:) or return
    source_io.rewind
    content
  end
end

#summarize_source(source_io, source, words: nil, instruction: nil) ⇒ String?

Summarizes content from the given source IO and source identifier.

This method takes an IO object containing document content and generates a summary based on the configured prompt template and word count.

Parameters:

  • source_io (IO) —

    The input stream containing the document content to summarize

  • source (String, #to_s) —

    The identifier or path for the source of the content

  • words (Integer, nil) (defaults to: nil) —

    The target number of words for the summary (defaults to 100)

  • instruction (String, nil) (defaults to: nil) —

    An optional instruction to focus the summary (e.g. "stress breaking changes and deprecations")

Returns:

  • (String, nil) —

    The formatted summary message or nil if content is empty or cannot be processed



168
169
170
171
172
173
174
175
176
177
178
179
 
# File 'lib/ollama_chat/source_fetching.rb', line 168

def summarize_source(source_io, source, words: nil, instruction: nil)
  infobar.puts "Summarizing #{italic { source_io&.content_type }} document #{source.to_s.inspect} now."
  log(:info, "Source summarized", data: { source: source.to_s, content_type: source_io&.content_type, words: })
  words = words.to_i
  words < 1 and words = 100
  instruction    = instruction.full? { ", focusing specifically on: #{_1}" }
  source_content = parse_source(source_io)
  source_content.present? or return
  prompt(:summarize).to_s.named_placeholders_interpolate(
    { source_content:, words:, instruction: }
  )
end