Documentation
¶
Overview ¶
Package tokens provides segmentation for multiline YAML tokens and utilities for syntax highlighting.
Token Segments ¶
YAML lexers produce tokens that may span multiple lines (block scalars, multiline strings, folded content).
When rendering YAML line-by-line for display or editing, each line needs its portion of such tokens while still knowing which original token it came from. This package bridges that gap.
A Segment pairs an original source token with a "part" token representing one line's portion.
For a three-line block scalar, you get three Segments that share the same source pointer but have different parts.
This shared pointer enables efficient deduplication: call Segments.SourceTokens to recover the original tokens without duplicates.
Building Line-Based Views ¶
The [line] package uses Segments to represent a single line's worth of tokens and Segments2 for multiple lines.
Position-based queries like Segments2.TokenRangesAt find all ranges a token occupies across lines, which is useful for highlighting all parts of a token (e.g. for errors).
Pointer Safety ¶
Source pointers are shared internally for equality checks and deduplication, but Segment.Source and Segment.Part return clones to prevent accidental modification. Use Segment.SourceEquals for pointer comparisons.
Syntax Highlighting ¶
TypeStyle maps token types to style.Style values for syntax highlighting. It handles context-sensitive styling: a string followed by a colon is styled as a mapping key, not a plain string.
Multi-Document YAML ¶
SplitDocuments splits a token stream at document headers ("---"), returning an iterator over separate token streams for each YAML document.
Index ¶
- func SplitDocuments(tks token.Tokens) iter.Seq2[int, token.Tokens]
- func TypeStyle(tk *token.Token) style.Style
- func ValueOffset(tk *token.Token) int
- type Segment
- type Segments
- func (s Segments) Append(source, part *token.Token) Segments
- func (s Segments) Clone() Segments
- func (s Segments) Merge(others ...Segments) Segments
- func (s Segments) NextColumn() int
- func (s Segments) PartTokens() token.Tokens
- func (s Segments) SourceTokenAt(col int) *token.Token
- func (s Segments) SourceTokens() token.Tokens
- type Segments2
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func SplitDocuments ¶
SplitDocuments splits a token stream into multiple token streams, one for each YAML document found (separated by '---' tokens).
The returned slices each contain tokens for a single document, preserving original token order and positions.
Each document header token ('---') is included at the start of its document.
func TypeStyle ¶
TypeStyle returns the style.Style for the given *token.Token's token.Type.
It handles context-sensitive styling: a string followed by a colon is styled as a mapping key, and tokens preceded by anchors or aliases inherit that styling.
func ValueOffset ¶
ValueOffset calculates the byte offset where Value starts within the first non-empty line of the *token.Token's Origin.
This offset is used for string slicing operations.
Types ¶
type Segment ¶
type Segment struct {
// contains filtered or unexported fields
}
Segment pairs an original source *token.Token with a segmented part token.
Multiple Segment values may share the same Segment.Source pointer while having distinct Segment.Part values.
Create instances with NewSegment.
func NewSegment ¶
NewSegment creates a new Segment.
func (Segment) Contains ¶
Contains reports whether the given *token.Token matches the source or part pointer of this Segment.
func (Segment) Part ¶
Part returns a clone of the Segment's part *token.Token.
func (Segment) PartEquals ¶
PartEquals reports whether the given *token.Token matches the part pointer.
func (Segment) Source ¶
Source returns a clone of the Segment's source *token.Token.
func (Segment) SourceEquals ¶
SourceEquals reports whether the given *token.Token matches the source pointer.
type Segments ¶
type Segments []Segment
Segments is a sequence of Segment values, typically representing a single line's tokens.
When constructed from a complete token stream, unique Segment.Source pointers represent the original tokens (deduplicated via Segments.SourceTokens).
For multiline tokens, multiple consecutive Segment values share the same Segment.Source pointer.
func (Segments) Clone ¶
Clone returns a copy of the Segments with cloned [Segment.Part]s but shared Segment.Source pointers.
Sources are intentionally shared since they are immutable references to original tokens.
func (Segments) Merge ¶
Merge combines this Segments with others, preserving source pointer identity. Returns a new Segments containing all Segment values in order.
func (Segments) NextColumn ¶
NextColumn returns the next available column (0-indexed).
Note that token.Position is 1-indexed, in which case this value + 1 can be used.
Returns 0 if empty or no Segment has a valid position.
func (Segments) PartTokens ¶
PartTokens returns clones of all Segment.Part [*token.Token]s in order.
func (Segments) SourceTokenAt ¶
SourceTokenAt returns a clone of the source *token.Token at the given 0-indexed column.
Returns nil if no token exists at that column.
func (Segments) SourceTokens ¶
SourceTokens returns clones of unique source tokens in order.
This is the inverse of segmentation: Segment values that share a Segment.Source pointer are deduplicated to return a clone of each original *token.Token once.
type Segments2 ¶
type Segments2 []Segments
Segments2 represents a slice of Segments, i.e. a 2-dimensional collection of Segment values. Each element typically represents one line's Segments.
func (Segments2) ContentRangesAt ¶
ContentRangesAt returns position.Ranges for content at the given position, excluding leading and trailing spaces.
Returns nil if there is no content at the given position or if the token is all whitespace.
func (Segments2) TokenRangesAt ¶
TokenRangesAt returns position.Ranges for all Segment values that share the same source token as the Segment at the given 0-indexed idx (typically line) and col.
Positions are calculated based on Segment widths.
Returns nil if the position is out of bounds or no Segment exists there.
Source Files