=begin pod

=head1 NAME

Needle::Compile - compile a search needle specification

=head1 SYNOPSIS

=begin code :lang<raku>
use Needle::Compile;

my &basic = compile-needle("bar");
say basic("foo bar baz");             # True
say basic("Huey, Dewey, and Louie");  # False

my &capitals-ok = compile-needle("bar", :ignorecase);
say capitals-ok("foo bar baz");             # True
say capitals-ok("FOO BAR BAZ");             # True
say capitals-ok("Huey, Dewey, and Louie");  # False

my &regex-ok = compile-needle("regex" => '\w+');
say regex-ok("foo");  # True
say regex-ok(":;+");  # False

my &regex-matches = compile-needle("regex" => '\w+', :matches);
say regex-matches(":foo:");  # (foo)
say regex-matches(":;+");    # False
=end code

=head1 DESCRIPTION

Needle::Compile exports a subroutine "compile-needle" that takes a number
of arguments that specify a search query needle, and returns a C<Callable>
that can be called with a given haystack to see if there is a match.

It can as such be used as the argument to the C<rak> subroutine provided
by the L<C<rak>|https://raku.land/zef:lizmat/rak> distribution.

=head1 SEARCH QUERY SPECIFICATION

A query can consist of multiple needles of different types.  A type of
needle can be specified in 3 ways:

=head3 implicitely

=begin code :lang<raku>
# accept haystack if "bar" as word is found
my &needle = compile-needle("§bar");
=end code

By using textual markers at the start and end of the given string needle
(see: type is "auto").

=head3 explicitely

=begin code :lang<raku>
# accept haystack if "bar" as word is found
my &needle = compile-needle("words" => "bar");
=end code

By specifying a needle as a C<Pair>, with the key being a string describing
the type.

=head3 mixed in

=begin code :lang<raku>
use Needle::Compile "Type";

# accept haystack if "bar" as word is found
my &needle = compile-needle("bar" but Type("words"));
=end code

If the specified needle supports a C<.type> method, then that method will
be called to determine the type.  This can be done with the C<Type> role
that is optionally exported.

If you specify C<"Type"> in the C<use> statement, a C<Type> role will be
exported that allows you to mixin a type with a given string.  The C<Type>
role will only allow known types in its specification.  If that is too
restrictive for your application, you can define your own C<Type> role.
Which can be as simple as:

=begin code :lang<raku>
my role Type { $.type }
=end code

If you want to be able to dispatch on strings that have a C<but Type>
mixed in, you can also import the C<StrType> class:

=begin code :lang<raku>
use Needle::Compile <Type StrType>;

say "foo" but Type<words> ~~ Str;      # True
say "foo" but Type<words> ~~ StrType;  # True
say "foo"                 ~~ StrType;  # False
=end code

Furthermore, you can call the C<.ACCEPTS> method on the C<StrTtype>
class to check whether a given type string is valid:

=begin code :lang<raku>
use Needle::Compile <Type StrType>;

say StrType.ACCEPTS("contains");    # True
say StrType.ACCEPTS("frobnicate");  # False
=end code

=head2 Modifiers

Many types of matching support C<ignorecase> and C<ignoremark> semantics.
These can be specified explicitely (with the C<:ignorecase> and
C<:ignoremark> named arguments), or implicitely with the C<:smartcase>
and C<:smartmark> named arguments.

The same types also support the <:match> named argument, to return the
string that actually matched, rather than C<True>.

=head3 ignorecase

=begin code :lang<raku>
# accept haystack if "bar" is found, regardless of case
my &needle = compile-needle("bar", :ignorecase);
=end code

Allow characters to match even if they are of mixed case.

=head3 smartcase

=begin code :lang<raku>
# accept haystack if "bar" is found, regardless of case
my &anycase = compile-needle("bar", :smartcase);

# accept haystack if "Bar" is found
my &exactcase = compile-needle("Bar", :smartcase);
=end code

If the needle is a string and does B<not> contain any uppercase characters,
then C<ignorecase> semantics will be assumed.

=head3 ignoremark

=begin code :lang<raku>
# accept haystack if "bar" is found, regardless of any accents
my &anycase = compile-needle("bar", :ignoremark);
=end code

Allow characters to match even if they have accents (or not).

=head3 smartmark

=begin code :lang<raku>
# accept haystack if "bar" is found, regardless of any accents
my &anymark = compile-needle("bar", :smartmark);

# accept haystack if "bår" is found
my &exactmark = compile-needle("bår", :smartmark);
=end code

If the needle is a string and does B<not> contain any characters with accents,
then C<ignoremark> semantics will be assumed.

=head3 matches

=begin code :lang<raku>
my &regex-matches = compile-needle("regex" => '\w+', :matches);
say regex-matches(":foo:");  # (foo)
say regex-matches(":;+");    # False
=end code

Return all the strings that matched as a C<Slip>, rather than C<True>.
Will still return C<False> if no matches were found.

=head2 Types of matchers

=head3 auto

This is the default type of matcher.  It looks at the given string for a
number of markers, and adjust the type of match and the string accordingly.
The following markers are recognized:

=head4 starts with !

=begin code :lang<raku>
# accept haystack if "bar" is NOT found
my &needle = compile-needle('!bar');
=end code

This is a meta-marker.  Assumes the string given (without the C<!>) should
be processed, and its result negated (see: type is "not").

=head4 starts with &

=begin code :lang<raku>
# accept haystack if "foo" and "bar" are found
my &needle = compile-needle('foo', '&bar');
=end code

This is a meta-marker.  Marks the needle produced for the given string
(without the C<&>) as needing an C<&&> infix with its predecessor
be processed, rather than the default C<||> infix (see: type is "and").

=head4 starts with §

=begin code :lang<raku>
# accept haystack if "bar" is found as a word
my &needle = compile-needle('§bar');
=end code

Assumes the string given (without the C<§>) should match with
word-boundary semantics applied (see: type is "words").

=head4 starts with *

=begin code :lang<raku>
# accept haystack if alphabetically before "bar"
my &is-before = compile-needle('* before "bar"');

# return every haystack uppercased
my &uppercased = compile-needle('*.uc');
=end code

Assumes the given string is a valid C<WhateverCode> specification and
attempts to produce that specification accordingly (see: type is "code").

=head4 starts with ^

=begin code :lang<raku>
# accept haystack if it starts with "bar"
my &needle = compile-needle('^bar');
=end code

Assumes the string given (without the C<^>) should match with
C<.starts-with> semantics applied (see: type is "starts-with").

=head4 starts with file:

=begin code :lang<raku>
# accept matching patterns in file "always"
my &needle = compile-needle('file:always');
=end code

Assumes the given string (without the C<file:> prefix) is a valid path
specification, pointing to a file containing the patterns to be used
(see: type is "file").

=head4 starts with jp:

=begin code :lang<raku>
# return result of JSON::Path query "auth"
my &needle = compile-needle('jp:auth');
=end code

Assumes the given string (without the C<jp:> prefix) is a valid
L<C<JSON Path>|https://en.wikipedia.org/wiki/JSONPath> specification
(see: type is "json-path").

=head4 starts with s:

=begin code :lang<raku>
# accept if string contains "foo" *and* "bar"
my &needle = compile-needle('s:foo &bar');
=end code

Splits the given string (without the C<s:> prefix) on whitespace and
interpretes the result as a list of needles  (see: type is "split").

=head4 starts with url:

=begin code :lang<raku>
# accept if any of the needles at given URL matches
my &domain = compile-needle('url:raku.org/robots.txt');  # assumes https://
my &url    = compile-needle('url:https://raku.org/robots.txt');
=end code

Interpretes the given string (without the C<url:> prefix) as a URL
from which to obtain the needles from (see: type is "url").

=head4 ends with $

=begin code :lang<raku>
# accept haystack if it ends with "bar"
my &needle = compile-needle('bar$');
=end code

Assumes the string given (without the C<$>) should match with
C<.ends-with> semantics applied (see: type is "ends-with").

=head4 starts with ^ and ends with $

=begin code :lang<raku>
# accept haystack if it is equal to "bar"
my &needle = compile-needle('^bar$');
=end code

Assumes the string given (without the C<^> and C<$>) should match exactly
(see: type is "equal").

=head4 starts with / and ends with /

=begin code :lang<raku>
# accept haystack if it matches "bar" as a regular expression
my &needle = compile-needle('/bar/');
=end code

Assumes the string given (without the C</>'s) is a regex and attempts
to produce a C<Regex> object and wraps that in a call to C<.contains>
(see: type is "regex").

=head4 starts with { and ends with }

=begin code :lang<raku>
# return the lowercased, whitespace trimmed haystack
my &needle = compile-needle('{.trim.lc}');
=end code

Assumes the string given (without the C<{> and C<}>) is an expression
and attempts to produce the code (see: type is "code").

=head4 none of the above

=begin code :lang<raku>
# accept haystack if it contains "bar"
my &needle = compile-needle("bar");
=end code

Assumes the string given should match with C<.contains> semantics
applied (see: type is "contains").

=head3 and

=begin code :lang<raku>
# accept haystack if "foo" and "bar" are found
my &needle = compile-needle('foo', "and" => 'bar');
=end code

This is a meta-marker.  Marks the needle produced for the given string
as needing an C<&&> infix with its predecessor be processed, rather
than the default C<||> infix.  Has no meaning on the first (or only)
needle in a list of needles.

=head3 code

=begin code :lang<raku>
# return uppercase version of the haystack
my &needle = compile-needle("code" => ".uc");
=end code

Assumes the string is an expression and attempts to produce that code,
with the haystack being presented as the topic (C<$_>).

To facilitate the use of libraries that wish to access that topic, it
is also available as the C<$*_> dynamic variable.

Furthermore, the "code" type also accepts two named arguments:

=head4 :repo

=begin code :lang<raku>
# look for modules in "lib" subdirectory as well
my &needle = compile-needle("code" => '.uc', :repo<lib>);
=end code

Specifies location(s) in which loadable modules should be searched.  It
is the equivalent of Raku's C<-I> command line option, and the C<use lib>
pragma.

=head4 :module

=begin code :lang<raku>
# Load the "Test" module
my &needle = compile-needle("code" => 'is $_, 42', :module<Test>);
=end code

Specifies module(s) that should be loaded.  It is the equivalent of Raku's
C<-M> command line option, and the C<use> statement.

=head3 contains

=begin code :lang<raku>
# accept haystack if it contains "bar"
my &needle = compile-needle("contains" => "bar");
=end code

Assumes the string is a needle in a call to C<.contains>.

=head3 ends-with

=begin code :lang<raku>
# accept haystack if it ends with "bar"
my &needle = compile-needle("ends-with" => "bar");
=end code

Assumes the string is a needle in a call to C<.ends-with>.

=head3 equal

=begin code :lang<raku>
# accept haystack if it is equal to "bar"
my &needle = compile-needle("equal" => "bar");
=end code

Assumes the string is a needle to compare with the haystack using infix
C<eq> semantics.

=head3 file

=begin code :lang<raku>
# return result of matching patterns in file "always"
my &needle = compile-needle("file" => 'always');
=end code

Assumes the given string is a valid path specification, pointing to a
file containing the patterns to be used.

The following types of lines will be ignored:
=item empty lines
=item lines consisting of whitespace only
=item lines starting with "#"

Note that the filename C<"-"> will be interpreted as "read from STDIN".

=head3 json-path

=begin code :lang<raku>
# return result of JSON::Path query "auth"
my &needle = compile-needle("json-path" => 'auth');
=end code

Assumes the given string is a valid
L<C<JSON Path>|https://en.wikipedia.org/wiki/JSONPath> specification.

Must have the L<C<JSON::Path>|https://raku.land/cpan:JNTHN/JSON::Path>
module installed.

The generated C<Callable> will expect an C<Associative> haystack (aka, a
<Hash>) to be passed, and will return a C<Slip> with any results.

=begin code :lang<raku>
# return result of JSON::Path query "auth"
my &needle = compile-needle({ jp('auth').Slip });
=end code

Alternately, you can also the C<jp()> function.  This returns a specialized
C<JP> object.  Note that simply specifying a call to the C<jp> function is
not enough: one must do something with it.  In the above example calling
C<.Slip> will cause the actual querying to happen.

See "Doing JSON path queries in code needles" for more information.

=head3 not

=begin code :lang<raku>
# accept haystack if "bar" is NOT found
my &needle = compile-needle("not" => "bar");
=end code

This is a meta-type: it inverts the result of the result of matching
of the given needle (which can be anything otherwise acceptable).

=head3 regex

=begin code :lang<raku>
# accept haystack if "bar" is found
my &needle = compile-needle("regex" => "bar");
=end code

Assumes the string is a regex specification to be used as a needle in a
call to C<.contains>.

=head3 split

=begin code :lang<raku>
# accept if string contains "foo" *and* "bar"
my &needle = compile-needle("split" => 'foo &bar');
=end code

This is a meta-type: it splits the given string on whitespace and
interpretes the result as a list of needles.

=head3 starts-with

=begin code :lang<raku>
# accept haystack if it starts with "bar"
my &needle = compile-needle("starts-with" => "bar");
=end code

Assumes the string is a needle in a call to C<.starts-with>.

=head3 url

=begin code :lang<raku>
# accept if any of the needles at given URL matches
my &domain = compile-needle("url" => 'raku.org/robots.txt');
my &url    = compile-needle("url" => 'http://example.com/patterns');
=end code

Interpretes the given string as a URL from which to obtain the needles
from.  Assumes C<https://> if no protocol is specified.

Requires the C<curl> program to be installed and runnable.  All protocols
supported by C<curl> can be used in the URL specification.

=head3 words

=begin code :lang<raku>
# accept haystack if "bar" is found as a word
my &needle = compile-needle("words" => "bar");
=end code

Assumes the string is a needle that will match if the needle is found
with word boundaries on either side of the needle.

=head2 Doing JSON path queries in code needles

=begin code :lang<raku>
# return result of JSON::Path query "auth"
my &needle = compile-needle("code" => 'jp("auth").Slip');
=end code

The initial call to the C<jp> function will attempt to load the
C<JSON::Path> module, and fail if it cannot.  If the load is successful,
it will create an object of the (internal) C<JP> class with the giveni
pattern.

If the C<JSON::Path> module was already loaded, it will either produce
the C<JP> object from a cache, or create a new C<JP> object.

The object as such, does nothing.  It needs to have a method called on
it B<within the scope of the code needle> to properly function.

If you're using the (implicit) C<"json-path"> type for your needle, this
is done automatically for you, by calling the C<.Slip> method on it.

The following methods can be called on the C<JP> object:

=table
  method             selected
  ---------------------------
  .value             The first selected value
  .values            All selected values as a Seq
  .paths             The paths of all selected values as a Seq
  .paths-and-values  Interleaved selected paths and values
  .words             All words in selected values as a Slip
  .head              The first N selected values as a Slip
  .tail              The last N selected values as a Slip
  .skip              Skip N selected values, produce rest as a Slip
  .Seq               All selected values as a Seq
  .Bool              True if any values selected, else False
  .List              All selected values as a List
  .Slip              All selected values as a Slip
  .gist              All selected values stringified as a gist
  .Str               All selected values stringified

=begin code :lang<raku>
# return first and third result of JSON::Path query "auth" as a Slip
my &needle = compile-needle({ jp('auth')[0,2] });
=end code

Furthermore, you can use postcircumfix C<[ ]> on the C<JP> object to
select values from the result.

=head1 AST

The C<compile-needle> subroutine also accepts an C<:AST> named argument.
When specified with a true value, it will return the C<RakuAST::Node>
representation of the C<Callable>.

This is intended for debugging and for incorporation into other modules
that synthetically build code using C<RakuAST>.

=begin code :lang<raku>
say compile-needle("bar:, :AST);
=end code
=begin output
RakuAST::PointyBlock.new(
  signature => RakuAST::Signature.new(
    parameters => (
      RakuAST::Parameter.new(
        target => RakuAST::ParameterTarget::Var.new(
          name => "\$_"
        )
      ),
    )
  ),
  body      => RakuAST::Blockoid.new(
    RakuAST::StatementList.new(
      RakuAST::Statement::Expression.new(
        expression => RakuAST::VarDeclaration::Simple.new(
          sigil       => "\$",
          desigilname => RakuAST::Name.from-identifier("/")
        )
      ),
      RakuAST::Statement::Expression.new(
        expression => RakuAST::Term::TopicCall.new(
          RakuAST::Call::Method.new(
            name => RakuAST::Name.from-identifier("contains"),
            args => RakuAST::ArgList.new(
              RakuAST::StrLiteral.new("bar")
            )
          )
        )
      )
    )
  )
)
=end output

=head1 HELPER SUBROUTINES

=head2 implicit2explicit

=begin code :lang<raku>
use Needle::Compile "implicit2explicit";

dd implicit2explicit('foo');    # :contains("foo")
dd implicit2explicit('§bar');   # :words("bar")
dd implicit2explicit('!baz$');  # :not(:ends-with("baz"))
=end code

The C<implicit2explicit> subroutine converts an implicit query specification
into an explicit one (expressed as a C<Pair> with the key as the type, and
the value as the actual string for which to create a needle).

=head1 THEORY OF OPERATION

This module uses the new C<RakuAST> classes as much as possible to create
an executable C<Callable>.  This means that until RakuAST supports the
complete Raku Programming Language features, it B<is> possible that some
code will not actually produce a C<Callable> needle.

There is not a lot of documentation about RakuAST yet, but there are some
blog posts, e.g. L<RakuAST for early adopters|https://dev.to/lizmat/rakuast-for-early-adopters-576n>.

=head1 DEBUGGING

The environment variable C<NEEDLE_COMPILE_DEBUG> can be specified with
2 values:

=item 1 - show deparsed version of the AST that has been created
=item 2 - also show the actual AST that has been created first

=head1 AUTHOR

Elizabeth Mattijsen <liz@raku.rocks>

Source can be located at: https://codeberg.org/lizmat/Needle-Compile .
Comments and Pull Requests are welcome.

If you like this module, or what I’m doing more generally, committing to a
L<small sponsorship|https://github.com/sponsors/lizmat/>  would mean a great
deal to me!

=head1 COPYRIGHT AND LICENSE

Copyright 2024, 2026 Elizabeth Mattijsen

This library is free software; you can redistribute it and/or modify it under the Artistic License 2.0.

=end pod

# vim: expandtab shiftwidth=4
