=begin pod

=head1 Markdown::Lex

A Markdown lexer for text that is B<still being written>. It turns a document
into a flat list of typed blocks and styled spans, and hands them to whoever is
drawing the screen: no HTML, no renderer, no dependencies, no opinions about
colour.

It exists because a terminal chat client has to draw a model's answer while the
tokens are still arriving, which means re-lexing the same growing string a few
hundred times a second and never once getting an exception or a paragraph that
flickers out of existence.

=head2 Synopsis

=begin code :lang<raku>
use Markdown::Lex;

for parse("# Title\n\nHello **world** and `code`.\n") -> $block {
    given $block {
        when Markdown::Lex::Heading {
            say "H{$block.level}: ", $block.inlines».text.join;
        }
        when Markdown::Lex::Para {
            for $block.inlines -> $span {
                print $span.bold ?? bold($span.text) !! $span.text;
            }
        }
        when Markdown::Lex::CodeFence {
            say "[{$block.lang // 'text'}]", $block.closed ?? '' !! ' (still writing)';
            .say for $block.lines;
        }
    }
}
=end code

=head2 The two promises

B<Total.> C<parse> never throws. Not on C<Str:U>, not on the empty string, not
on a document that is one unterminated code fence, not on ten thousand
asterisks in a row. Whatever goes in, a C<List> of blocks comes out.

B<Prefix-stable.> Every prefix of a document is itself a valid document, so a
consumer can do this and never think about it again:

=begin code :lang<raku>
my $accumulated = '';

react whenever $token-supply -> $chunk {
    $accumulated ~= $chunk;
    render(parse($accumulated));
}
=end code

Half-arrived constructs degrade to literal text rather than vanishing: C<**bo>
is a plain span until the closing asterisks turn up. The single exception is the
code fence, which becomes a C<CodeFence> with C<:!closed> the moment it opens —
because a renderer wants to draw the box while the code streams into it.

=head2 The blocks

=begin table

Class                       | Attributes
============================|=====================================================
Markdown::Lex::Heading      | C<Int:D $.level> (1..6), C<@.inlines>
Markdown::Lex::Para         | C<@.inlines>
Markdown::Lex::CodeFence    | C<Str $.lang>, C<@.lines>, C<Bool:D $.closed>
Markdown::Lex::Bullet       | C<Int:D $.depth> (0-based), C<Str:D $.marker>, C<@.inlines>
Markdown::Lex::Quote        | C<@.inlines>
Markdown::Lex::Rule         | (none)
Markdown::Lex::Table        | C<@.alignments>, C<@.header>, C<@.rows>, C<.columns>

=end table

All seven are immutable, all seven are a C<Markdown::Lex::Block>, and all seven
have a C<.gist> worth printing. C<use Markdown::Lex;> exports one symbol —
C<parse> — and the classes are reachable fully qualified, which is what a
C<given>/C<when> wants anyway.

A table's cells are C<Markdown::Lex::Cell>, which holds one cell's C<@.inlines>
and nothing else — the same flat spans a paragraph has, so a renderer draws a
cell with the code it already has:

=begin code :lang<raku>
my $table = parse("| Name | Age |\n|:-----|----:|\n| Bob | 4 |").head;

$table.columns;                          # 2
$table.alignments;                       # ("left", "right")
$table.rows[0][0].inlines».text.join;    # "Bob"
=end code

A table needs its delimiter row before it is a table: until that second line
arrives the header row is an ordinary paragraph, which is what keeps a table
that is still being typed from flickering in and out of existence.

=head2 The spans

C<@.inlines> is a flat list of C<Markdown::Lex::Span>, never a tree:

=begin code :lang<raku>
class Markdown::Lex::Span {
    has Str:D  $.text is required;
    has Bool:D $.bold   = False;
    has Bool:D $.italic = False;
    has Bool:D $.code   = False;
    has Str    $.link;              # undefined unless this span is a link
}
=end code

Nesting is expressed by combining flags, because that is exactly what a terminal
can draw:

=begin code :lang<raku>
parse("**bold with *both* inside**").head.inlines;
# Span("bold with " :b) Span("both" :bi) Span(" inside" :b)
=end code

So there is no nesting depth limit and no recursion. Adjacent runs with
identical flags are merged and an empty span is never emitted, which means a
renderer can loop over C<@.inlines> with no defensive checks at all.

=head2 What it lexes

ATX headings, fenced code (backticks or tildes, with an info word), bullets
(C<->/C<*>/C<+> and ordinals, with a depth taken from the indentation), block
quotes, thematic breaks, GFM pipe tables, and paragraphs that merge their soft
line breaks into spaces. Inline: code spans, C<**bold**>, C<*italic*>,
C<_italic_>, C<__bold__>, C<***both***> and C<[text](destination)>.

Emphasis follows CommonMark's delimiter-run rules closely enough that the cases
which bite in practice come out right — C<snake_case_name> is not italic,
C<5 * 3 * 2> is arithmetic, C<*a **b** c*> nests, and a leftover delimiter is
literal text rather than a swallowed one.

=head2 One option

=begin code :lang<raku>
parse("one\ntwo").head.inlines;                # Span("one two")
parse("one\ntwo", :hard-breaks).head.inlines;  # Span("one") Break Span("two")
=end code

C<:hard-breaks> is for drawing text a human typed, where the newlines are where
they are on purpose: each one becomes a zero-text C<Markdown::Lex::Break> span
instead of a space. It is off by default and changes nothing else. Under it each
line is lexed on its own, so emphasis does not reach across a newline — a
divergence from GFM that the module's Pod argues for at length.

=head2 What it does not lex

Setext headings, indented code blocks, images, raw HTML, autolinks, footnotes,
reference links, hard line breaks written as two trailing spaces, and backslash
escapes bar C<\|> in a table row. Nested block structure is out too:
C<Bullet.depth> is the only nesting a consumer gets.

Every one of them degrades to literal text, which is this module's only failure
mode. Nothing is ever dropped — with one documented exception, GFM's rule that a
table row wider than its own header loses the extra cells.

The full list of divergences, with reasons, is in the module's own Pod:
C<raku --doc lib/Markdown/Lex.rakumod>, or just open the file — it opens with
them.

=head2 Testing

=begin code :lang<shell>
prove6 -Ilib t/
=end code

No dependencies, so nothing to install first. The suite covers each block and
inline construct, the CommonMark cases that are easy to get wrong, CRLF and lone
carriage returns, Unicode from ZWJ emoji to combining accents to right-to-left
text, and — the ones that matter most — a loop over B<every prefix> of a torture
document asserting that the lexer neither throws nor drops a single word, plus
timing guards on five-thousand-character delimiter storms to catch a quadratic
regression.

=head2 Author

Matt Doughty

=head2 License

Artistic-2.0

=end pod
