Module: OllamaChat::Parsing

Includes:
Utils::AnalyzeDirectory
Included in:
Chat
Defined in:
lib/ollama_chat/parsing.rb

Overview

A module that provides content parsing functionality for OllamaChat.

The Parsing module encapsulates methods for processing various types of input sources including HTML, XML, CSV, RSS, Atom, PDF, and Postscript documents. It handles content extraction and conversion into standardized text formats suitable for chat interactions. The module supports different document policies for handling imported or embedded content and provides utilities for parsing structured data from multiple source types.

Examples:

Processing different document types

chat.parse_source(html_io)    # Converts HTML to markdown
chat.parse_source(pdf_io)     # Extracts text from PDF files
chat.parse_source(csv_io)     # Formats CSV data into readable strings

Constant Summary collapse

DOCUMENT_POLICY_STATES =

An array of valid document policy states that define how document references in user text are handled.

These states control the behavior of the document policy selector:

  • ignoring: Document references are ignored.
  • embedding: Document references are embedded into the conversation context for RAG.
  • importing: Document references are imported into the conversation.
  • summarizing: Document references are summarized for reference.
%w[ ignoring embedding importing summarizing ]
HAS_AUDIO =

Matches MIME types for audio and video content, including common container formats. Used by #parse_source and the get_url tool to route media streams to #parse_audio for transcription.

%r{\A(audio/|video/|application/(mp4|ogg|opus|webm|x-matroska)\z)}
CONTENT_REGEXP =

Regular expression to scan content for url/file references

%r{
  (https?://\S+)                         # Match HTTP/HTTPS URLs
  |                                      # OR
  (file://(?:[^\s#]+))                   # Match file:// URLs
  |                                      # OR
  "((?:\.\.|[~.]?)/(?:\\"|\\|[^"\\]+)+)" # Quoted file path with escaped " quotes
  |                                      # OR
  ((?:\.\.|[~.]?)/(?:\\\ |\\|[^\\\s]+)+) # File path with escaped spaces
}x

Instance Method Summary collapse

Methods included from Utils::AnalyzeDirectory

#generate_structure, #recurse_generate_structure, #structure_each_entry

Instance Method Details

#epub_read(io) ⇒ String?

Converts an EPUB source to Plaintext via Pandoc and returns the text content.

Parameters:

  • io (IO) —

    the input stream containing the EPUB data

Returns:

  • (String, nil) —

    the extracted text as a plaintext string, or nil if conversion fails



203
204
205
206
207
208
209
210
211
212
213
214
 
# File 'lib/ollama_chat/parsing.rb', line 203

def epub_read(io)
  pandoc = `which pandoc`.chomp
  if pandoc.present?
    Tempfile.create(['epub_in', '.epub']) do |tmp|
      IO.copy_stream(io, tmp)
      `#{pandoc} -f epub -t plain "#{tmp.path}"`
    end
  else
    feedback("Cannot convert EPUB, pandoc not in path.", type: :warn)
    nil
  end
end

#parse_atom(source_io) ⇒ String

The parse_atom method processes an Atom feed from the provided IO source and converts it into a formatted text representation. It extracts the feed title and iterates through each item to build a structured output containing titles, links, and update dates.

The content of each item is converted using reverse_markdown for better readability.

Parameters:

  • source_io (IO) —

    the input stream containing the Atom feed data

Returns:

  • (String) —

    a formatted string representation of the Atom feed with title, items, links, update dates, and content



168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
 
# File 'lib/ollama_chat/parsing.rb', line 168

def parse_atom(source_io)
  feed = RSS::Parser.parse(source_io, false, false)
  title = <<~EOT
    # #{feed.title.content}

  EOT
  feed.items.inject(title) do |text, item|
    text << <<~EOT
      ## [#{item&.title&.content}](#{item&.link&.href})

      updated on #{item&.updated&.content}

      #{reverse_markdown(item&.content&.content)}

    EOT
  end
end

#parse_audio(source_io, language: nil) ⇒ String?

Transcribes audio or video content via the ASR service. Delegates to ASR.transcribe which handles format conversion (e.g. video → WAV via ffmpeg) and the multipart upload to the server.

Parameters:

  • source_io (IO) —

    the input stream containing audio/video data.

  • language (String, nil) (defaults to: nil) —

    optional language hint (e.g. "en")

Returns:

  • (String, nil) —

    the transcribed text, or nil on failure.



122
123
124
 
# File 'lib/ollama_chat/parsing.rb', line 122

def parse_audio(source_io, language: nil)
  OllamaChat::ASR.transcribe(source_io, chat: self, language:)
end

#parse_content(content, images) ⇒ String

Parses a string for URLs, file refs, and image links, then returns the transformed content. Detects http(s) URLs, file:// paths, quoted file paths, and collects any image URLs into the supplied images array.

Parameters:

  • content (String) —

    the raw text to parse

  • images (Array) —

    mutable array that will be cleared then filled with discovered image URLs

Returns:

  • (String) —

    the content after all supported transformations (URLs resolved, file refs expanded, image URLs collected)



295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
 
# File 'lib/ollama_chat/parsing.rb', line 295

def parse_content(content, images)
  images.clear
  contents = [ content ]
  content.scan(CONTENT_REGEXP).each { |url, file_url, quoted_file, file|
    if file && Pathname.new(file).expand_path.directory?
      max_depth = config.parsing.directory_structure_max_depth
      contents << generate_structure(file, max_depth:, include_hidden: true).to_json
      next
    end
    check_exist = false
    case
    when url
      source = url
    when file_url
      check_exist = true
      source      = file_url
    when quoted_file
      file = quoted_file.gsub('\"', ?")
      file =~ %r(\A[~./]) or file.prepend('./')
      check_exist = true
      source      = file
    when file
      file = file.gsub('\ ', ' ')
      file =~ %r(\A[~./]) or file.prepend('./')
      check_exist = true
      source      = file
    end
    fetch_source(source, check_exist:) do |source_io|
      case source_io&.content_type&.media_type
      when 'image'
        add_image(images, source_io, source)
        if source_io&.content_type&.sub_type == 'png'
          source_io.rewind
          if results = parse_png(source_io).full?
            contents.concat results
          end
        end
      when 'audio', 'video'
        document_policy.selected == 'ignoring' and next
        text = parse_audio(source_io)
        text_io = OllamaChat::Utils::Fetcher::ResponseMetadata.as_text(text)
        process_document(text_io, source, contents)
      when 'text', 'application', nil
        process_document(source_io, source, contents)
      else
        feedback(
          "Cannot fetch #{source.to_s.inspect} with content type " \
          "#{source_io&.content_type.inspect}", type: :warn
        )
      end
    end
  }
  contents.select { _1.present? rescue nil }.compact * "\n\n"
end

#parse_png(source_io) ⇒ Array<String>?

Extracts embedded metadata from a PNG image, including character profiles, prompts, and workflows. Character profiles are automatically personalized to replace placeholders with the current user's name.

Parameters:

  • source_io (IO) —

    The input stream containing the PNG binary data.

Returns:

  • (Array<String>, nil) —

    An array of formatted metadata sections (as strings) if any were found, or nil if no supported metadata was extracted.



83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
 
# File 'lib/ollama_chat/parsing.rb', line 83

def parse_png(source_io)
  metadata = OllamaChat::Utils::PNGMetadataExtractor.extract_all(source_io) or return
  results = []

  if data = metadata.delete('chara') and
      (char, char_data = OllamaChat::Utils::PNGMetadataExtractor.decode_character(data))
    then
    name = char_data.dig('data', 'name').full? || Pathname.new(source_io.path).basename.sub_ext('').to_s
    results << "Character Profile:\n\n#{personalize_character_profile(char, name:)}"
  end

  if data = metadata.delete('parameters') and
    params = OllamaChat::Utils::PNGMetadataExtractor.parse_a1111_parameters(data)
  then
    results << "Generation Settings:\n\n#{params.to_json}"
  end

  if data = convert_to_utf8(metadata.delete('prompt'))
    results << "Prompt:\n\n#{data}"
  end

  if data = convert_to_utf8(metadata.delete('workflow'))
    results << "Workflow:\n\n#{data}"
  end

  if data = metadata.full? { _1.transform_values { |v| convert_to_utf8(v) } }
    results << "Metadata:\n\n#{data}"
  end

  results.full?
end

#parse_rss(source_io) ⇒ String

The parse_rss method processes an RSS feed source and converts it into a formatted text representation. It extracts the channel title and iterates through each item in the feed to build a structured output. The method uses the RSS parser to handle the source input and formats the title, link, publication date, and description of each item into a readable text format with markdown-style headers and links.

Parameters:

  • source_io (IO) —

    the input stream containing the RSS feed data

Returns:

  • (String) —

    a formatted string representation of the RSS feed with channel title and item details



138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
 
# File 'lib/ollama_chat/parsing.rb', line 138

def parse_rss(source_io)
  feed = RSS::Parser.parse(source_io, false, false)
  title = <<~EOT
    # #{feed&.channel&.title}

  EOT
  feed.items.inject(title) do |text, item|
    text << <<~EOT
      ## [#{item&.title}](#{item&.link})

      updated on #{item&.pubDate}

      #{reverse_markdown(item&.description)}

    EOT
  end
end

#parse_source(source_io, language: nil) ⇒ String?

The parse_source method processes different types of input sources and converts them into a standardized text representation.

Parameters:

  • source_io (IO) —

    the input source to be parsed

Returns:

  • (String, nil) —

    the parsed content as a string or nil if the content type is not supported



40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
 
# File 'lib/ollama_chat/parsing.rb', line 40

def parse_source(source_io, language: nil)
  case source_io&.content_type
  when 'text/html'
    reverse_markdown(source_io.read)
  when 'text/xml', 'application/xml'
    if source_io.read(8192) =~ %r(^\s*<rss\s)
      source_io.rewind
      return parse_rss(source_io)
    end
    source_io.rewind
    source_io.read
  when 'application/rss+xml'
    parse_rss(source_io)
  when 'application/atom+xml'
    parse_atom(source_io)
  when 'application/postscript'
    ps_read(source_io)
  when 'application/pdf'
    pdf_read(source_io)
  when 'application/epub+zip'
    epub_read(source_io)
  when HAS_AUDIO
    parse_audio(source_io, language:)
  when 'image/png'
    results = parse_png(source_io) and return results.join("\n\n---\n\n")
    feedback("Could not parse metadata from #{source_io&.content_type} document.", type: :warn)
    nil
  when %r(\Aapplication/(json|ld\+json|x-ruby|x-perl|x-gawk|x-python|x-javascript|x-c?sh|x-dosexec|x-shellscript|x-tex|x-latex|x-lyx|x-bibtex)), %r(\Atext/), nil
    source_io.read
  else
    feedback("Cannot parse #{source_io&.content_type} document.", type: :warn)
    return
  end
end

#pdf_read(io) ⇒ String

The pdf_read method extracts text content from a PDF file by reading all pages.

Parameters:

  • io (IO) —

    the input stream containing the PDF data

Returns:

  • (String) —

    the concatenated text content from all pages in the PDF



192
193
194
195
 
# File 'lib/ollama_chat/parsing.rb', line 192

def pdf_read(io)
  reader = PDF::Reader.new(io)
  reader.pages.inject(+'') { |result, page| result << page.text }
end

#personalize_character_profile(char, name:) ⇒ String

Personalizes a character profile by replacing placeholders with actual names.

Parameters:

  • char (String) —

    The raw character JSON string.

  • name (String) —

    The name of the character to replace {char} with.

Returns:

  • (String) —

    The personalized character profile.



265
266
267
268
269
270
 
# File 'lib/ollama_chat/parsing.rb', line 265

def personalize_character_profile(char, name:)
  my_user_name = user_name || 'the user'
  char = char.gsub(/{{user}}/i, my_user_name)
  name.present? and char = char.gsub(/{{char}}/i, name)
  char
end

#process_document(source_io, source, contents) ⇒ Object

Dispatches a parsed source to the handler matching the active #document_policy. This is the shared terminal step for both the audio/video and the text branches of #parse_content, so every policy is applied uniformly regardless of the source media type.

Behaviour per policy:

  • ignoring — no-op, the source is dropped.
  • importing — the source text is appended to contents.
  • embedding — the source is embedded into the RAG context; nothing is appended to contents.
  • summarizing — the source summary is appended to contents.

Parameters:

  • source_io (IO) —

    the parsed/streamable source to process

  • source (String) —

    the original source (URL or path), used for reference and logging by the underlying handlers

  • contents (Array<String>) —

    mutable accumulator that receives the imported or summarized text (mutated in place)



367
368
369
370
371
372
373
374
375
376
377
 
# File 'lib/ollama_chat/parsing.rb', line 367

def process_document(source_io, source, contents)
  case document_policy.selected
  when 'ignoring'
  when 'importing'
    contents << import_source(source_io, source)
  when 'embedding'
    embed_source(source_io, source)
  when 'summarizing'
    contents << summarize_source(source_io, source)
  end
end

#ps_read(io) ⇒ String?

Reads and processes PDF content using Ghostscript for conversion

This method takes an IO object containing PDF data, processes it through Ghostscript's pdfwrite device, and returns the processed PDF content. If Ghostscript is not available in the system path, it outputs an error message.

Parameters:

  • io (IO) —

    An IO object containing PDF data to be processed

Returns:

  • (String, nil) —

    The processed PDF content as a string, or nil if processing fails



225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
 
# File 'lib/ollama_chat/parsing.rb', line 225

def ps_read(io)
  gs = `which gs`.chomp
  if gs.present?
    Tempfile.create do |tmp|
      IO.popen("#{gs} -q -sDEVICE=pdfwrite -sOutputFile=#{tmp.path} -", 'wb') do |gs_io|
        IO.copy_stream(io, gs_io)
        gs_io.close
        File.open(tmp.path, 'rb') do |pdf|
          pdf_read(pdf)
        end
      end
    end
  else
    feedback("Cannot convert #{io&.content_type} with ghostscript, gs not in path.", type: :warn)
  end
end

#reverse_markdown(html) ⇒ String

The reverse_markdown method converts HTML content into Markdown format.

This method processes HTML input and transforms it into equivalent Markdown, using specific conversion options to ensure compatibility and formatting.

Parameters:

  • html (String) —

    the HTML string to be converted

Returns:

  • (String) —

    the resulting Markdown formatted string



251
252
253
254
255
256
257
258
 
# File 'lib/ollama_chat/parsing.rb', line 251

def reverse_markdown(html)
  ReverseMarkdown.convert(
    html,
    unknown_tags: :bypass,
    github_flavored: true,
    tag_border: ''
  )
end