[![Actions Status](https://github.com/m-doughty/Markdown-Lex/actions/workflows/test.yml/badge.svg)](https://github.com/m-doughty/Markdown-Lex/actions)

Markdown::Lex
=============

A Markdown lexer for text that is **still being written**. It turns a document into a flat list of typed blocks and styled spans, and hands them to whoever is drawing the screen: no HTML, no renderer, no dependencies, no opinions about colour.

It exists because a terminal chat client has to draw a model's answer while the tokens are still arriving, which means re-lexing the same growing string a few hundred times a second and never once getting an exception or a paragraph that flickers out of existence.

Synopsis
--------

```raku
use Markdown::Lex;

for parse("# Title\n\nHello **world** and `code`.\n") -> $block {
    given $block {
        when Markdown::Lex::Heading {
            say "H{$block.level}: ", $block.inlines».text.join;
        }
        when Markdown::Lex::Para {
            for $block.inlines -> $span {
                print $span.bold ?? bold($span.text) !! $span.text;
            }
        }
        when Markdown::Lex::CodeFence {
            say "[{$block.lang // 'text'}]", $block.closed ?? '' !! ' (still writing)';
            .say for $block.lines;
        }
    }
}
```

The two promises
----------------

**Total.** `parse` never throws. Not on `Str:U`, not on the empty string, not on a document that is one unterminated code fence, not on ten thousand asterisks in a row. Whatever goes in, a `List` of blocks comes out.

**Prefix-stable.** Every prefix of a document is itself a valid document, so a consumer can do this and never think about it again:

```raku
my $accumulated = '';

react whenever $token-supply -> $chunk {
    $accumulated ~= $chunk;
    render(parse($accumulated));
}
```

Half-arrived constructs degrade to literal text rather than vanishing: `**bo` is a plain span until the closing asterisks turn up. The single exception is the code fence, which becomes a `CodeFence` with `:!closed` the moment it opens — because a renderer wants to draw the box while the code streams into it.

The blocks
----------

<table class="pod-table">
<thead><tr>
<th>Class</th> <th>Attributes</th>
</tr></thead>
<tbody>
<tr> <td>Markdown::Lex::Heading</td> <td>C&lt;Int:D $.level&gt; (1..6), C&lt;@.inlines&gt;</td> </tr> <tr> <td>Markdown::Lex::Para</td> <td>C&lt;@.inlines&gt;</td> </tr> <tr> <td>Markdown::Lex::CodeFence</td> <td>C&lt;Str $.lang&gt;, C&lt;@.lines&gt;, C&lt;Bool:D $.closed&gt;</td> </tr> <tr> <td>Markdown::Lex::Bullet</td> <td>C&lt;Int:D $.depth&gt; (0-based), C&lt;Str:D $.marker&gt;, C&lt;@.inlines&gt;</td> </tr> <tr> <td>Markdown::Lex::Quote</td> <td>C&lt;@.inlines&gt;</td> </tr> <tr> <td>Markdown::Lex::Rule</td> <td>(none)</td> </tr> <tr> <td>Markdown::Lex::Table</td> <td>C&lt;@.alignments&gt;, C&lt;@.header&gt;, C&lt;@.rows&gt;, C&lt;.columns&gt;</td> </tr>
</tbody>
</table>

All seven are immutable, all seven are a `Markdown::Lex::Block`, and all seven have a `.gist` worth printing. `use Markdown::Lex;` exports one symbol — `parse` — and the classes are reachable fully qualified, which is what a `given`/`when` wants anyway.

A table's cells are `Markdown::Lex::Cell`, which holds one cell's `@.inlines` and nothing else — the same flat spans a paragraph has, so a renderer draws a cell with the code it already has:

```raku
my $table = parse("| Name | Age |\n|:-----|----:|\n| Bob | 4 |").head;

$table.columns;                          # 2
$table.alignments;                       # ("left", "right")
$table.rows[0][0].inlines».text.join;    # "Bob"
```

A table needs its delimiter row before it is a table: until that second line arrives the header row is an ordinary paragraph, which is what keeps a table that is still being typed from flickering in and out of existence.

The spans
---------

`@.inlines` is a flat list of `Markdown::Lex::Span`, never a tree:

```raku
class Markdown::Lex::Span {
    has Str:D  $.text is required;
    has Bool:D $.bold   = False;
    has Bool:D $.italic = False;
    has Bool:D $.code   = False;
    has Str    $.link;              # undefined unless this span is a link
}
```

Nesting is expressed by combining flags, because that is exactly what a terminal can draw:

```raku
parse("**bold with *both* inside**").head.inlines;
# Span("bold with " :b) Span("both" :bi) Span(" inside" :b)
```

So there is no nesting depth limit and no recursion. Adjacent runs with identical flags are merged and an empty span is never emitted, which means a renderer can loop over `@.inlines` with no defensive checks at all.

What it lexes
-------------

ATX headings, fenced code (backticks or tildes, with an info word), bullets (`-`/`*`/`+` and ordinals, with a depth taken from the indentation), block quotes, thematic breaks, GFM pipe tables, and paragraphs that merge their soft line breaks into spaces. Inline: code spans, `**bold**`, `*italic*`, `_italic_`, `__bold__`, `***both***` and `[text](destination)`.

Emphasis follows CommonMark's delimiter-run rules closely enough that the cases which bite in practice come out right — `snake_case_name` is not italic, `5 * 3 * 2` is arithmetic, `*a **b** c*` nests, and a leftover delimiter is literal text rather than a swallowed one.

One option
----------

```raku
parse("one\ntwo").head.inlines;                # Span("one two")
parse("one\ntwo", :hard-breaks).head.inlines;  # Span("one") Break Span("two")
```

`:hard-breaks` is for drawing text a human typed, where the newlines are where they are on purpose: each one becomes a zero-text `Markdown::Lex::Break` span instead of a space. It is off by default and changes nothing else. Under it each line is lexed on its own, so emphasis does not reach across a newline — a divergence from GFM that the module's Pod argues for at length.

What it does not lex
--------------------

Setext headings, indented code blocks, images, raw HTML, autolinks, footnotes, reference links, hard line breaks written as two trailing spaces, and backslash escapes bar `\|` in a table row. Nested block structure is out too: `Bullet.depth` is the only nesting a consumer gets.

Every one of them degrades to literal text, which is this module's only failure mode. Nothing is ever dropped — with one documented exception, GFM's rule that a table row wider than its own header loses the extra cells.

The full list of divergences, with reasons, is in the module's own Pod: `raku --doc lib/Markdown/Lex.rakumod`, or just open the file — it opens with them.

Testing
-------

```shell
prove6 -Ilib t/
```

No dependencies, so nothing to install first. The suite covers each block and inline construct, the CommonMark cases that are easy to get wrong, CRLF and lone carriage returns, Unicode from ZWJ emoji to combining accents to right-to-left text, and — the ones that matter most — a loop over **every prefix** of a torture document asserting that the lexer neither throws nor drops a single word, plus timing guards on five-thousand-character delimiter storms to catch a quadratic regression.

Author
------

Matt Doughty

License
-------

Artistic-2.0

