Availability:
:- use_module(library(semweb/rdf_litindex)).
rdf_tokenize_literal(+Literal,
-Tokens)rdf_litindex:tokenization(Literal, -Tokens). On failure it
calls tokenize_atom/2
from the NLP package and deletes the following: atoms of length 1,
floats, integers that are out of range and the English words and, an, or, of,
on, in, this and the.
Deletion first calls the hook rdf_litindex:exclude_from_index(token,
X). This hook is called as follows:
no_index_token(X) :-
exclude_from_index(token, X), !.
no_index_token(X) :-
...