Byte-Pair Encoding (BPE) subword tokenizer trainer and deterministic text encoder/decoder.
Alpha-Park/genpark-bpe-tokenizer-byte-pair-encoder-skillByte-Pair Encoding (BPE) subword tokenizer trainer and deterministic text encoder/decoder.
Alpha-Park/genpark-levenshtein-damerau-fuzzy-matcher-skillDamerau-Levenshtein edit distance metric with transposition detection and fuzzy string candidate ranker.