Module lexer 

Tokens, and the layout algorithm that turns indentation into INDENT/DEDENT.

docs/02-syntax.md §2.8: “logos for tokens; a hand-written layout algorithm producing explicit INDENT/DEDENT/NEWLINE tokens (Python’s approach), with brackets suppressing layout so multi-line calls work.”

Two rules that are easy to get subtly wrong and are therefore stated here:

• Brackets suppress layout. Inside (/[/{ no newline is significant, so a call may span lines without the parser ever seeing it.
• Blank and comment-only lines have no indentation. They emit nothing at all, so a comment at column 0 inside an indented block does not close the block.

Structs

Token

Enums

Raw

Tok

A token after layout: the raw tokens plus the three synthetic ones.

Functions

lex

Lex and lay out one file.