=begin pod

=head1 NAME

JSON::Collector - collect and process JSON

=head1 SYNOPSIS

=begin code :lang<raku>
use JSON::Collector;

my $collector = JSON::Collector.new(
  :todo($todo-dir.IO),
  :done($done-dir.IO)
);
$collector.store($json);

for $collector.unprocessed -> \data {
    # process JSON data
    data.mark-as-processed($type);  # or data.discard
}
=end code

=head1 DESCRIPTION

The C<JSON::Collector> distribution provides logic to collect JSON data
into a temporary storage location awaiting processing.  Then, when the
data is processed, it can optionally be stored into an archive location.

This is typically intended to handle JSON as generated by push services
such as Github, SourceHut or Codeberg webhooks.

Although this is typically used for JSON related data, it can also be
used by any other data source: the storage will however be JSON based.

=head1 JSON::Collector

The C<JSON::Collector> class contains the logic for storing and handling
processed data.

=head2 new

=begin code :lang<raku>
my $collector = JSON::Collector.new(
  :todo("todo".IO),
  :done("done".IO),
  :pretty, :spaces(2), :sorted-keys
);
=end code

The C<new> method takes the following named arguments:

=head3 todo

The directory in which items that are added with the C<store> method are
being placed.  Defaults to "todo" in the current directory.  The
directory B<must> exist and be writable.

=head3 done

Where the data that has been processed should be stored.  It can have
3 types of destination.

=item an C<IO::Path> of an existing directory

=item a C<Callable> that will take care of storing processed data

=item C<Nil> indicating processed data should be discarded

Defaults to "done" in the current directory.  Any C<IO::Path>
(implicitely) specified B<must> be an existing directory and be writable.

=head3 pretty / spacing / sorted-keys

These named arguments apply to the way any data added by calling
the C<store> method with a data structure (as opposed to a string)
will be converted to JSON.  The defaults are:

=item pretty = True
=item spacing = 2
=item sorted-keys = True

=head2 store

=begin code :lang<raku>
$collector.store($string); # or $collector.store(data);
=end code

The C<store> method stores the given data in the C<todo> directory.
If the given data is a string, then it is assumed to be valid JSON
and the string will be stored as-is.

In any other case, the given data will be converted to JSON with
the C<pretty>, C<spacing> and C<sorted-keys> settings specified in
the creation of the collector (but can be overridden by specifying
these same named arguments in the call to C<store>);

=begin code :lang<raku>
$collector.store(data, :!pretty);
=end code

The C<store> method either returns an C<IO::Path> object of the
file that was created to store the data, or C<Nil> if something
went wrong.

=head2 unprocessed

=begin code :lang<raku>
for $collector.unprocessed -> \data {
    # process JSON data
    data.mark-as-processed($type);  # or data.discard
}
=end code

The C<unprocessed> method produces a C<Seq> of C<JSON::Collector::Item>
objects of data that has not yet been processed yet.

=head1 JSON::Collector::Item

The C<JSON::Collector::Item> class encapsulates on item that was previously
stored.  It acts as closes as possible to the original data stored, with
some additional methods that can be invoked.

=head2 collector

The C<JSON::Collector> object that created this C<JSON::Collector::Item>
object.

=head2 IO

The C<IO::Path> object of the data.  After having been successfully
processed (with C<mark-as-processed>), then it contains the 

=head2 data

The actual data structure that was stored.  It's generally not needed to
call this method, as the object itself will act as the data as much as
possible.

=head2 mark-as-processed

=begin code :lang<raku>
for $collector.unprocessed -> \item {
    # process JSON data
    item.mark-as-processed($type);
}
=end code

The C<mark-as-processed> method will mark the data that was previously
stored as processed.

Depending on the value specified with C<done> in the C<JSON::Collector>
object, it will do one of three things:

=head3 if done is C<Nil>

Then this call is equivalent to calling the C<discard> method, as in
removing the item from the "todo" directory.

=head3 if done is a C<Callable>

Then the C<Callable> will be called with the C<JSON::Collector::Item>
object as the only positional argument.  If the return value is C<True>,
then the object will be removed from the "todo" directory, otherwise
it will be kept there.

=head3 if done is an C<IO::Path>

The optional type indication (which defaults to the empty string) will
be used as the name of a subdirectory in the "done" directory in which
the data will ultimately be stored.  Defaults to the empty string,
which means no subdirectory.

Then the JSON file in the "todo" directory (or "todo/$type") will be
moved to a sub-directory YYYY/YYYY-MM-DD of that directory, using the
current date.

=head2 discard

=begin code :lang<raku>
for $collector.unprocessed -> \item {
    # process JSON data
    item.discard;
}
=end code

Discards the item from the "todo" directory.

=head1 THEORY OF OPERATION

Whenever some data is stored (and it's not JSON already) it is converted
to JSON.  Then the JSON is spurted to a file with as basename the
current nanosecond value, with the extension "tmp".  Once the write is
completed, the file is (atomically, presumably) renamed to not have any
extension.

This means that many threads or processes can store date asynchronously
in the same collection.  When it comes to processing, it is assumed
that only a B<single> process/thread will be calling the C<unprocessed>
method.  The actual processing of the items I<may> me multi-threaded.

=head1 AUTHOR

Elizabeth Mattijsen <liz@raku.rocks>

Source can be located at: https://codeberg.org/lizmat/actions . Comments and
Pull Requests are welcome.

If you like this module, or what I'm doing more generally, committing to a
L<small sponsorship|https://github.com/sponsors/lizmat/>  would mean a great
deal to me!

=head1 COPYRIGHT AND LICENSE

Copyright 2026 Elizabeth Mattijsen

This library is free software; you can redistribute it and/or modify it under the Artistic License 2.0.

=end pod

# vim: expandtab shiftwidth=4
