Open
Description
Currently there is ByteTrie at https://github.com/epfl-dlab/transformers-CFG/blob/main/transformers_cfg/tokenization/trie.py#L21
and also a TokenTrie at https://github.com/epfl-dlab/transformers-CFG/blob/main/transformers_cfg/tokenization/vocab_struct.py#L13
TokenTrie is the legacy code from previous implementation and it should be completely removed and replaced by ByteTrie.
There is still dependency on it in the parsing, so may need to treat it carefully
Metadata
Metadata
Assignees
Labels
No labels