pub fn tokens(text: &str) -> Vec<Token>Expand description
Every coloured run in text, in source order, non-overlapping.
The lexer skips comments — a comment is not a token, and making it one would put the layout
algorithm’s “a comment-only line has no indentation” rule at risk (beck_syntax::doc says so
for ##). So comments are recovered from the gaps between the lexer’s own spans: whatever
the lexer did not claim is whitespace up to a #, and a comment from there to the end of the
line. That keeps one scanner rather than two — a second lexer written for highlighting is how
an editor ends up disagreeing with the compiler about where a string ends.
Diagnostics are discarded: a file with an unlexable character still produces a token stream
(that is what lex promises), and highlighting is wanted most exactly when the file is broken.