Module: ContextSpook::Utils
- Included in:
- Generator::Context
- Defined in:
- lib/context_spook/utils.rb
Overview
The ContextSpook::Utils module provides utility methods for formatting and processing context data.
Class Method Summary collapse
-
.estimate_tokens(text_or_bytesize) ⇒ Integer
The estimate_tokens method provides a crude estimation of token counts from either a raw byte count or a string.
-
.format_size(context_size) ⇒ String
The format_size method converts a byte size value into a human-readable string with binary units.
-
.format_tokens(tokens) ⇒ String
The format_tokens method converts a token count into a human-readable string using SI prefixes (e.g., 1.2 kT).
Instance Method Summary collapse
-
#estimate_tokens(text_or_bytesize) ⇒ Integer
private
The estimate_tokens method provides a crude estimation of token counts from either a raw byte count or a string.
-
#format_size(context_size) ⇒ String
private
The format_size method converts a byte size value into a human-readable string with binary units.
-
#format_tokens(tokens) ⇒ String
private
The format_tokens method converts a token count into a human-readable string using SI prefixes (e.g., 1.2 kT).
Class Method Details
.estimate_tokens(text_or_bytesize) ⇒ Integer
The estimate_tokens method provides a crude estimation of token counts from either a raw byte count or a string.
This follows a heuristic where one token is roughly equivalent to 3.5 bytes. A numeric argument is treated as a byte count directly; any other value is converted to a string and measured by its byte size.
40 41 42 43 44 45 |
# File 'lib/context_spook/utils.rb', line 40 def estimate_tokens(text_or_bytesize) unless text_or_bytesize.is_a?(Numeric) text_or_bytesize = text_or_bytesize.to_s.bytesize end (text_or_bytesize.to_f / 3.5).ceil end |
.format_size(context_size) ⇒ String
The format_size method converts a byte size value into a human-readable string with binary units.
library to display the size with appropriate IEC prefixes (KiB, MiB, etc.) and one decimal place.
15 16 17 |
# File 'lib/context_spook/utils.rb', line 15 def format_size(context_size) Tins::Unit.format(context_size, format: '%.1f %U', unit: ?B, prefix: :iec_uc) end |
.format_tokens(tokens) ⇒ String
The format_tokens method converts a token count into a human-readable string using SI prefixes (e.g., 1.2 kT).
25 26 27 |
# File 'lib/context_spook/utils.rb', line 25 def format_tokens(tokens) Tins::Unit.format(tokens, unit: ?T, prefix: :si_uc, format: '%.1f %U') end |
Instance Method Details
#estimate_tokens(text_or_bytesize) ⇒ Integer (private)
The estimate_tokens method provides a crude estimation of token counts from either a raw byte count or a string.
This follows a heuristic where one token is roughly equivalent to 3.5 bytes. A numeric argument is treated as a byte count directly; any other value is converted to a string and measured by its byte size.
40 41 42 43 44 45 |
# File 'lib/context_spook/utils.rb', line 40 def estimate_tokens(text_or_bytesize) unless text_or_bytesize.is_a?(Numeric) text_or_bytesize = text_or_bytesize.to_s.bytesize end (text_or_bytesize.to_f / 3.5).ceil end |
#format_size(context_size) ⇒ String (private)
The format_size method converts a byte size value into a human-readable string with binary units.
library to display the size with appropriate IEC prefixes (KiB, MiB, etc.) and one decimal place.
15 16 17 |
# File 'lib/context_spook/utils.rb', line 15 def format_size(context_size) Tins::Unit.format(context_size, format: '%.1f %U', unit: ?B, prefix: :iec_uc) end |
#format_tokens(tokens) ⇒ String (private)
The format_tokens method converts a token count into a human-readable string using SI prefixes (e.g., 1.2 kT).
25 26 27 |
# File 'lib/context_spook/utils.rb', line 25 def format_tokens(tokens) Tins::Unit.format(tokens, unit: ?T, prefix: :si_uc, format: '%.1f %U') end |