=begin rakudoc :type<fundamental>
=TITLE Minimal documentation of RakuDoc::Render Module
=SUBTITLE Overview of rendering engine.
=for AUTHOR :hidden
Richard Hainsworth aka finanalyst
=for LICENSE :hidden
Artistic-2.0
=for VERSION :hidden
v0.4.0

=head Overview

The module contains the RakuDoc::Processor class.

The class method 'render' is the principal one for rendering RakuDoc v2.
The Raku compiler expects to call render with an AST and to return a Str.
It can be called with the option C<:pre-finalised> to get the processed data,
see the example in L<Debugging|#Debugging>.

The aim is for a B<RakuDoc::Processor> object (an B<RPO>) to be as generic as
possible, allowing for other classes to instantiate it, and to attach
templates that will generate more specific formats, such as HTML,
MarkDown, or EPub pages.

An B<RPO> relies on L<Templates|Templates>, L<PromiseStrings|PromiseStrings>,
and L<ScopedData|ScopedData>.

It is also necessary to have a good understanding of RakuDoc v2.

Some L<environment variables|#Environment variables> provide for some
output control.

The following describes choices that are left by RakuDoc v2 to the renderer.

=head Templates

An B<RPO> contains a default set of Text templates.

The aim of the text templates is that they are the lowest level templates
needed to render any RakuDoc source.

When a renderer is needed to output a new format, eg., HTML, the renderer
instantiates an B<RPO> and attaches new templates to it using the
C<add-template( Pair $p, :source )> method or the C<add-templates( Hash $h, :source )> method.
Setting the C<source> option to some value will help determine which module introduced
the templates.

The design of the Templates object means that new additions to the object
push the previous definition onto a linked list, and so they will always
be available by default.

The set of template keys needed to create a renderer is L<tabulated here|default-text-templates>.

All blocks make be prefixed by C<num>. The renderer handles the enumeration before handling the block.
The I<num> handling sets the config variable C<numeration>, which is the enumeration string as defined
by the specification.

Consequently, all block templates need to handle the numeration value, which is '' for non-I<num> blocks.

All aspects of the output can be defined using the templates.

=head Process

The generic render method transforms the AST of the input file using a handle method for each
type of Block and Markup code. A template exists for each block and markup code.

After all the blocks have been processed, the Table of Contents, Index, and Footnotes are
rendered, with templates for each item and for the list of items.

Any forward references and numerations are then resolved.

The intermediate structure of the processed file can then be accessed before it is finally
rendered into a string.

The final rendering is done by the 'final' template.

Once a string has been obtained, it can be post-processed using the post-process method.
For example, text output can be transformed by wrapping lines.

=head Choices left by RakuDoc v2 to renderer

There are a few choices left by the specification to the renderer. This section contains a
description of those currently known (please raise issues if others are discovered), except for
rendering options regarding customisability that are covered separately below.

=item not all schemas listed for C<L<>> and C<P<>> are implemented, namely C<man:>, C<rakudoc:>,
C<isbn:>
=item the C<toc:> schema is implemented for C<P<>> only in relation to I<level>. Limiting the ToC
to tables or other named blocks is not yet implemented.
=begin item
The C<index:> schema is implemented in relation to I<level>. The index structure is compiled as follows:
each item of an index contains a label (the key of a hash structure), an array of references,
and a sub-index of children with the same structure.

The reference array may be empty if the sub-index has elements, and vice versa.

The level of an index is the depth of the label. But if a label only has elements in a sub-index and
no references of its own, AND the index is constrained to some level, then the label may appear
without a link to the body of the text. The renderer provides an ordered list of items. It is for the
template to determine whether to filter 'empty' labels.
=end item

=item I<numitem> and I<numdefn> counters are reset if the next C<=block> is not the same type.
This behaviour is extended to I<para> and I<nested>.

=head Warnings

In order to offer the maximum flexibility to a RakuDoc author, if there
is an error condition after the RakuDoc source has been compiled,
a set of warnings are generated.

By default these are rendered using the 'warnings' template, and the rendered
version is appended to the output file.

=head Customisability

RakuDoc allows for customisability. This renderer handles customisability
via the template mechanism and L<Plugins|#Plugins>.

=head2 Custom blocks

When custom block is detected according to the RakuDoc v2 spelling rules,
an B<RPO> will check whether a template exists with that name.

If a template has been attached to an B<RPO>, then the rendered content
of the block (including rendered embedded RakuDoc instructions) will be
provided to the template as C<contents> and the verbatim contents with
no renderering will be provided as C<raw>.

So a new block can be created by attaching a template with the blockname
to the B<RPO>.

=head2 Custom markup codes

RakuDoc v2 allows for custom markup codes in two ways:
=item the C<< M< DISPLAY | FUNCTION ; list of comma delimited strings > >>
=item a single Unicode (not ASCII ) character with the I<upper> property.

=head3 Markup M

When a C<< M< DISPLAY | FUNCTION ; LIST > >> is encountered, the renderer
will look for a template named in the R<FUNCTION> position. If no
such template exists,  the text will be rendered verbatim and a warning
logged.

=head3 Markup UNICODE[upper]

When some custom markup is encountered, the renderer will
=item check that the character, eg Ɵ (Greek Capital Theta), has
the unicode property I<Upper>;

=item look for a template called (eg) B<markup-Ɵ>. For clarity,
the Unicode character chosen for the markup code is prefixed
by B<markup-> in order to name the template.

Thus in order to create a new markup code, create an appropriately
named template and add it to the B<RPO>.

=head Plugins

Typically an output, such as HTML, will require the content of the text
to be organised in some way, but it will also need additional files
such CSS or JS content. The templates can handle the content organisation,
but obtaining JS and CSS libraries, or converting SASS to CSS involves
other code to be run.

These can be organised using plugins.

A plugin can attach assets (such as Javascript content) to a hash in its
I<data> workspace. Other plugins can operate on the data in these workspaces.
An example is the C<RakuDoc::Plugin::HTML::SCSS> plugin.

To enable plugins to work, the C<add-plugins( @plugin-list )> method of the
RakuDoc-Processor (rdp) object is called by the renderer. The C<@plugin-list>
must contain the full name of each plugin.

Each plugin must have a C<enable> method that takes an B<rdp>. Typically, the
B<enable> method will attach its config attribute to the workspace, and attach
its templates.

=head2 gather-flatten( $key ) / gather-flatten( @keys )

Typically many of the plugins will define SCSS, Javascript, or links. Since
the relative order in which (eg) css style sheets are attached matter, each
attribute needs to be defined as a sequence of 'Str',<order> tuples.

This method takes all the B<$key> attributes, sorts them by order, then forms
a single sequence that is then used in the templates, eg. to form stylesheets
in the head part of an HTML file.

=head Global functions

Each output format requires a different escaping strategy both for content
and file names.

This is handled by subclassing Processor object and over-riding C<escape>
and C<mangle> methods for each output. See for example how C<RakuDoc::To::HTML>
manages this.

Since these functions are also needed inside templates, they can be called
as follows:

=item $tmpl.globals.escape.($s), where C<$s> is a string. B<Note> the mandatory C<.> before the I< ( ) >
=item $tmpl.globals.mangle.($s)

=head Global data accessible in templates

More information is available in L<Templates|Templates>.

All templates can attach items to the Table of Contents, Index,
Footnotes, and Warnings structures of the rendered source using
the helper methods.

Currently, templates can I<add> but not B<substitute> or B<remove> items from
structures. For example, within a template with a custom block, the ToC structure
has been changed when the block is specified, so adding to the ToC within the
custom block's templates will add information to the ToC structure immediately
under the heading for the block. It is necessary, therefore to add a
I<:caption> to the custom block in order to over-ride the default ToC behaviour.

The following helper methods are available:

=item add-to-toc( %h ), where %h is a hash with the following keys:
=item2 :caption(Str)
=item2 :target( Str )
=item2 :level( Int )

=item add-to-index( %h ), where %h is a hash with the following keys:
=item2 :contents(Str)
=item2 :target(Str)
=item2 :place( Str }

=item add-to-footnotes( %h ), where %h is a hash with the following keys:
=item2 :retTarget(Str)
=item2 :fnTarget(Str)
=item2 :fnNumber(Str}

=item add-to-warnings( $warning )

In addition, within a template, it is possible to attach data
to globals, and to retrieve the data in another template.

For an example of this, see test B<xt/030-customisation-data.rakutest>.
(test number may be changed, but otherwise the filename will be the same).

=head Debugging

When developing new templates or renderers based on an B<RPO>, several
debug options can be attached to the B<RPO>, eg.
=begin code
    my RakuDoc::Processor $rdp .=new( :test );
    $rdp.debug( AstBlock, Templates);
    $ast = Q:to/QAST/.AST;
    =begin rakudoc
    =head This is a header

    Some text

    =end rakudoc
    QAST

    $rv = $rdp.render( $ast, :pre-finalised );
    'myOutput'.IO.spurt: $rdp.finalise
=end code
Commentary on the code
=item C< new( :test )> this attaches the test templates to the B<RPO>
=item the list inside the debug method may contain several entities as described below
=item creation of an AST from a string, or it can be slurped in from a file.
=item C< :pre-finalised > returns the RakuDoc::Processed object
=item if called with C<:pre-finalised>, calling C<finalise> returns a
Str that can then be stored as a file.

=head2 Debug options

The following options are available:
=item None removes all previous debugging
=item All adds all debugging options
=item AstBlock diagnostic whenever a new Block is called
=item BlockType information about which C<RakuAST::Doc::Block> is being processed
=item Scoping produces scope level diagnostics when a new level is started/ended
=item Templates indicates which templates are called and the params they are called with
=item MarkUp like BlockType but gives the MarkUp letter

It is also possible to get the result of one template (so as to reduce
the amount of output information). This is done eg for the 'table' template:
    $rdp.verbose( 'table' );
    $rv =  $rdp.render( $ast, :pre-finalised );

The Test and Pretty options described in L<the Templates documentation|Templates> can
be set on an B<RPO>, eg.
    $rdp.test( True );
    $rdp.pretty( True );

Bear in mind that the B<pretty> flag overrides the B<test> flag, and both override the
B<debug> and B<verbose> flags.

=head Environment variables

By setting the environment variable POSTPROCESSING=1 the text output will be naively wrapped.

For example,
=for code :lang<text>
POSTPROCESSING=1 bin/RenderTextify --pretty rakudociem-ipsum

If the environment variable WIDTH is also set, the text output will be
wrapped to the value.

WIDTH by default is set at 80 chars. To set at 70, use:
=for code :lang<text>
POSTPROCESSING=1 WIDTH=70 bin/RenderTextify rakudociem-ipsum

=head Paragraph id

Since sometimes it is necessary to target a specific paragraph, some
paragraphs get an automated id based on the last Seven hex chars of
the SHA1 encoding of its contents.

Seven chars should be adequate so long as the number of paragraphs
in a single document is below 16,000.

=nested Based on the expectation that we
would see collision in a repository with 2^(2N) objects when using object
names shortened to first N bits
L<Stackoverflow question|https://stackoverflow.com/questions/18134627/how-much-of-a-git-sha-is-generally-considered-necessary-to-uniquely-identify-a>

Long texts or books will probably need more to avoid a conflict. This can be done by setting C<paragraph-id-length>
in the %structure-data to the required number of hex digits.

=head Writing a new Render

Typically, it should only be necessary to change the templates of an existing renderer
to alter the way output is formated. But for a new type of format, a new renderer will
be needed.

It is best to look at the code for the HTML renderer to see how to write a renderer. The
following are only very brief notes.

A new renderer for a new output format will need:
=numitem A processor class, which is sub-classed from C<RakuDoc::Processor>. The C<escape>
method will need to be over-ridden.

=begin numitem
A Renderer class which must be called C<RakuDoc::To::NEWFORMAT>. This is required so that
the code
=for code :lang<text>
raku --rakudoc=NEWFORMAT input-file.rakudoc > output-filename.newformat

will work.
=end numitem
=numitem The Renderer class must contain the method C<render( $ast )>, which is needed by
the raku compiler, as noted above.

=numitem The class should also define and attach templates. Note that if any needed templates
are not included, then the default 'text' templates will be used.

=place semantic:AUTHOR :caption<Credits>

=place semantic:VERSION :!toc

=place semantic:LICENSE
=end rakudoc