=begin pod

=head1 NAME

Ecosystem::Archive::Update - Updating the Raku Ecosystem Archive

=head1 SYNOPSIS

=begin code :lang<raku>
use Ecosystem::Archive::Update;

my $ea = Ecosystem::Archive::Update.new(
  shelves => 'archive',
  jsons   => 'meta',
  sboms   => 'sbom',
);

say "Archive has $ea.meta.elems() identities:";
.say for $ea.meta.keys.sort;
=end code

=head1 DESCRIPTION

The C<Ecosystem::Archive::Update> distribution provides the basic logic
to updating the L<Raku Ecosystem Archive|https://github.com/Raku/REA>,
a place where (almost) every distribution ever available in the Raku
Ecosystem, can be obtained even after it has been removed (specifically
in the case of the old ecosystem master list and the distributions kept
on CPAN).

=head2 NAMED ARGUMENTS

=item :shelves

The name (or an C<IO> object) of a directory in which to place distributions.
This is usually a symlink to the "archive" directory of the actual
L<Raku Ecosystem Archive repository|https://github.com/Raku/REA>.
The default is 'archive', aka the 'archive' subdirectory from the current
directory.

=item :jsons

The name (or an C<IO> object) of a directory in which to store C<META6.json>
files as downloaded. This is usually a symlink to the "meta" directory of
the actual L<Raku Ecosystem Archive repository|https://github.com/Raku/REA>.
The default is 'meta', aka the 'meta' subdirectory from the current directory.

=item :sboms

The name (or an C<IO> object) of a directory in which to store
L<CycloneDX|https://cyclonedx.org>
L<SBOM|https://en.wikipedia.org/wiki/Software_supply_chain> files.
This is usually a symlink to the "sbom" directory of the actual
L<Raku Ecosystem Archive repository|https://github.com/Raku/REA>.
The default is 'sbom', aka the 'sbom' subdirectory from the current
directory.

=item :degree

The number of CPU cores that may be used for parallel processing. Defaults
to the B<half> number of C<Kernel.cpu-cores>.

=item :batch

The number of objects to be processed in parallel per batch. Defaults to
B<64>.

=head1 METHODS

=head2 batch

=begin code :lang<raku>
say "Processing with batches of $ea.batch() objects in parallel";
=end code

The number of objects per batch that will be used in parallel processing.

=head2 clear-notes

=begin code :lang<raku>
my @cleared = $ea.clear-notes;
say "All notes have been cleared";
=end code

Returns the C<notes> of the object as a C<List>, and removes them from the
object.

=head2 degree

=begin code :lang<raku>
say "Using $ea.degree() CPUs";
=end code

The number of CPU cores that will be used in parallel processing.

=head2 investigate-repo

=begin code :lang<raku>
my @found = $ea.investigate-repo($url, "lizmat");
=end code

Performs a C<git clone> on the given URL, scans the repo for changes in the
C<META6.json> file that would change the version, and downloads and saves
a tar-file of the repository (and the associated META information in
C<git-meta>) at that state of the repository.

The second positional parameter indicates the default C<auth> value to be
applied to any JSON information, if no C<auth> value is found or it is
invalid.

Only C<Github> and C<Gitlab> URLs are currently supported.

Returns a list of C<Pair>s of the distributions that were added, with the
identity as the key, and the META information hash as the value.

Updates the C<.meta> information in a thread-safe manner.

=head2 jsons

=begin code :lang<raku>
indir $ea.jsons, {
    my $jsons = (shell 'ls */*', :out).out.lines.elems;
    say "$jsons different distributions";
}
=end code

The C<IO> object of the directory in which the JSON meta files are being
stored.  For instance the C<IRC::Client> distribution:

=begin output
meta
  |- ...
  |- I
     |- ...
     |- IRC::Client
         |- IRC::Client:ver<1.001001>:auth<github:zoffixznet>.json
         |- IRC::Client:ver<1.002001>:auth<github:zoffixznet>.json
         |- ...
         |- IRC::Client:ver<3.007010>:auth<github:zoffixznet>.json
         |- IRC::Client:ver<3.007011>:auth<cpan:ELIZABETH>.json
         |- IRC::Client:ver<3.009990>:auth<cpan:ELIZABETH>.json
     |- ...
  |- ...
=end output

=head2 identities

=begin code :lang<raku>
say "Archive has $ea.identities.elems() identities, they are:";
.say for $ea.identities.keys.sort;
=end code

Returns a hash of all of the META information of all distributions, keyed
by identity (for example "Module::Name:ver<0.1>:auth<foo:bar>:api<1>").
The value is a hash obtained from the distribution's meta data.

=head2 meta-as-json

=begin code :lang<raku>
say $ea.meta-as-json;  # at least 3MB of text
=end code

Returns the JSON of all the currently known meta-information. The
JSON is ordered by identity in the top level array.

=head2 note

=begin code :lang<raku>
$ea.note("something's wrong");
=end code

Add a note to the C<notes> of the object.

=head2 notes

=begin code :lang<raku>
say "Found $ea.notes.elems() notes:";
.say for $ea.notes;
=end code

Returns the C<notes> of the object as a C<List>.

=head2 sboms

=begin code :lang<raku>
indir $ea.sboms, {
    my $sboms = (shell 'ls */*', :out).out.lines.elems;
    say "$sboms distributions with SBOMs";
}
=end code

The C<IO> object of the directory in which the
L<CycloneDX|https://cyclonedx.org>
L<SBOM|https://en.wikipedia.org/wiki/Software_supply_chain> files
are being stored.  For instance the C<IRC::Client> distribution:

=begin output
sbom
  |- ...
  |- I
     |- ...
     |- IRC::Client
         |- IRC::Client:ver<4.0.13>:auth<zef:lizmat>.tar.gz.cdx.json
         |- IRC::Client:ver<4.0.14>:auth<zef:lizmat>.tar.gz.cdx.json
     |- ...
  |- ...
=end output

=head2 shelves

=begin code :lang<raku>
indir $ea.shelves, {
    my $distro-names = (shell 'ls */*/*', :out).out.lines.elems;
    say "$distro-names different distributions in archive";
}
=end code

The C<IO> object of the directory where distributions are being stored
in a subdirectory by the name of the module in the distribution. For
instance the C<silently> distribution:

=begin output
archive
 |- ...
 |- S
    |- ...
    |- silently
        |- silently:ver<0.0.1>:auth<cpan:ELIZABETH>.tar.gz
        |- silently:ver<0.0.2>:auth<cpan:ELIZABETH>.tar.gz
        |- silently:ver<0.0.3>:auth<cpan:ELIZABETH>.tar.gz
        |- silently:ver<0.0.4>:auth<zef:lizmat>.tar.gz
    |- ...
 |- ...
=end output

Note that a subdirectory will contain B<all> distributions of the name,
regardless of version, authority or API value.

=head2 update

=begin code :lang<raku>
my %updated = $ea.update;
=end code

Updates all the meta-information and downloads any new distributions.
Returns a hash with the identities and the meta info of any distributions
that were not seen before. Also updates the C<.identities> information in
a thread-safe manner.

=head1 AUTHOR

Elizabeth Mattijsen <liz@raku.rocks>

Source can be located at: https://codeberg.org/lizmat/Ecosystem-Archive-Update .
Comments and Pull Requests are welcome.

If you like this module, or what I’m doing more generally, committing to a
L<small sponsorship|https://github.com/sponsors/lizmat/> would mean a great
deal to me!

=head1 COPYRIGHT AND LICENSE

Copyright 2021, 2022, 2023, 2024, 2025, 2026 Elizabeth Mattijsen

This library is free software; you can redistribute it and/or modify it under the Artistic License 2.0.

=end pod

# vim: expandtab shiftwidth=4
