Module: ContextSpook::Utils

Included in:
Generator::Context
Defined in:
lib/context_spook/utils.rb

Overview

The ContextSpook::Utils module provides utility methods for formatting and processing context data.

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.estimate_tokens(text_or_bytesize) ⇒ Integer

The estimate_tokens method provides a crude estimation of token counts from either a raw byte count or a string.

This follows a heuristic where one token is roughly equivalent to 3.5 bytes. A numeric argument is treated as a byte count directly; any other value is converted to a string and measured by its byte size.

Parameters:

  • text_or_bytesize (String, Integer) —

    the content to estimate tokens for, either as a string or a precomputed byte count

Returns:

  • (Integer) —

    the estimated number of tokens



40
41
42
43
44
45
# File 'lib/context_spook/utils.rb', line 40

def estimate_tokens(text_or_bytesize)
  unless text_or_bytesize.is_a?(Numeric)
    text_or_bytesize = text_or_bytesize.to_s.bytesize
  end
  (text_or_bytesize.to_f / 3.5).ceil
end

.format_size(context_size) ⇒ String

The format_size method converts a byte size value into a human-readable string with binary units.

library to display the size with appropriate IEC prefixes (KiB, MiB, etc.) and one decimal place.

Parameters:

  • context_size (Integer) —

    the size in bytes to be formatted

Returns:

  • (String) —

    the formatted size string with binary units



15
16
17
# File 'lib/context_spook/utils.rb', line 15

def format_size(context_size)
  Tins::Unit.format(context_size, format: '%.1f %U', unit: ?B, prefix: :iec_uc)
end

.format_tokens(tokens) ⇒ String

The format_tokens method converts a token count into a human-readable string using SI prefixes (e.g., 1.2 kT).

Parameters:

  • tokens (Integer) —

    the number of tokens to be formatted

Returns:

  • (String) —

    the formatted token string



25
26
27
# File 'lib/context_spook/utils.rb', line 25

def format_tokens(tokens)
  Tins::Unit.format(tokens, unit: ?T, prefix: :si_uc, format: '%.1f %U')
end

Instance Method Details

#estimate_tokens(text_or_bytesize) ⇒ Integer (private)

The estimate_tokens method provides a crude estimation of token counts from either a raw byte count or a string.

This follows a heuristic where one token is roughly equivalent to 3.5 bytes. A numeric argument is treated as a byte count directly; any other value is converted to a string and measured by its byte size.

Parameters:

  • text_or_bytesize (String, Integer) —

    the content to estimate tokens for, either as a string or a precomputed byte count

Returns:

  • (Integer) —

    the estimated number of tokens



40
41
42
43
44
45
# File 'lib/context_spook/utils.rb', line 40

def estimate_tokens(text_or_bytesize)
  unless text_or_bytesize.is_a?(Numeric)
    text_or_bytesize = text_or_bytesize.to_s.bytesize
  end
  (text_or_bytesize.to_f / 3.5).ceil
end

#format_size(context_size) ⇒ String (private)

The format_size method converts a byte size value into a human-readable string with binary units.

library to display the size with appropriate IEC prefixes (KiB, MiB, etc.) and one decimal place.

Parameters:

  • context_size (Integer) —

    the size in bytes to be formatted

Returns:

  • (String) —

    the formatted size string with binary units



15
16
17
# File 'lib/context_spook/utils.rb', line 15

def format_size(context_size)
  Tins::Unit.format(context_size, format: '%.1f %U', unit: ?B, prefix: :iec_uc)
end

#format_tokens(tokens) ⇒ String (private)

The format_tokens method converts a token count into a human-readable string using SI prefixes (e.g., 1.2 kT).

Parameters:

  • tokens (Integer) —

    the number of tokens to be formatted

Returns:

  • (String) —

    the formatted token string



25
26
27
# File 'lib/context_spook/utils.rb', line 25

def format_tokens(tokens)
  Tins::Unit.format(tokens, unit: ?T, prefix: :si_uc, format: '%.1f %U')
end