Package org.coordinatekit.crf.annotator

Class RetokenizeReviewer<F,T extends Comparable<T>>

java.lang.Object
org.coordinatekit.crf.annotator.RetokenizeReviewer<F,T>
Type Parameters:
F - the feature type carried by the tagger and the per-token entries
T - the tag type
@NullMarked public final class RetokenizeReviewer<F,T extends Comparable<T>> extends Object
Reviews an XML training-data file against a new tokenizer and rewrites a corrected copy.

A downstream tokenizer change can leave stored sequences tokenized under an old tokenizer that no longer match what the new one produces. review(Path, Path) walks the input file in document order and, for each TrainingSequence, reconstructs its surface, re-tokenizes it with the configured Tokenizer, and routes it by the AlignmentStatus an AlignmentDetector reports:

  • ALIGNED — the stored tokenization still matches, so the sequence is copied through verbatim, preserving its tags and excluded runs.
  • MISALIGNED — the sequence is freshly re-tokenized and presented through the configured TaggingInterface for a human to re-tag. On ACCEPT the re-tokenized tokens, the chosen tags, and the new excluded runs are written; on SKIP the original (still-misaligned) sequence is kept; on EXIT this and every remaining sequence are copied through unchanged and review stops.
  • UNTOKENIZABLE — the new tokenizer rejected the surface; the original sequence is copied through unchanged and a warning is emitted to the terminal.

The output contains exactly one sequence per input sequence, in document order. Review never drops or reorders sequences; only accepted misaligned sequences differ from their input, so unreviewed items remain a complete, valid dataset. Initial tags are always fresh — model-suggested when a CrfTagger is configured, otherwise every token starts at TagProvider.startingTag(); no old tags carry over.

Review is a fresh, single-pass rewrite: the output path must be absent or empty, and must differ from the input path. There is no content-hash resume — that behavior belongs to Annotator.

Tokenizer/tagger consistency. Alignment detection uses the standalone Tokenizer, while presentation with a tagger uses CrfTagger.tag(String), which re-tokenizes through the tagger's own tokenizer. These two must agree on how a surface splits; the downstream factory that supplies both is responsible for wiring the same tokenizer into each, as the annotate path already assumes.

Instances are constructed via the nested RetokenizeReviewer.Builder.

See Also:

  • Method Details

    • builder

      public static <F, T extends Comparable<T>> RetokenizeReviewer.Builder<F,T> builder()
      Returns a new builder for RetokenizeReviewer.
      Type Parameters:
      F - the feature type carried by the tagger and the per-token entries
      T - the tag type
      Returns:
      a new builder with no values set

    • review

      public void review(Path inputFile, Path outputFile) throws IOException
      Reviews the input file against the configured tokenizer, writing a complete corrected dataset to the output file.

      Each input sequence is routed by its AlignmentStatus: aligned and untokenizable sequences are copied through (the latter with a terminal warning), and misaligned sequences are presented through the TaggingInterface and rewritten on ACCEPT, kept as stored on SKIP, or — on EXIT — copied through along with every remaining sequence before review stops. Every written sequence is flushed immediately. A one-line summary is written to the configured Terminal when the pass completes.

      Parameters:
      inputFile - the XML training-data file to review
      outputFile - the XML training-data file to write the corrected dataset to; must be absent or empty and must differ from inputFile
      Throws:
      ReviewPreconditionException - if inputFile and outputFile are the same path, or outputFile already exists and is non-empty
      IOException - if reading the input or writing the output fails