what way is used for Tokenize the dataset?

[cdgraff]

Doc, say: All models are trained from the 1st 100M tokens

Can share some example how prepare that 100M tokens from the text input?, I'm trying to train support for ES lang.

Thanks

[benob]

The tokenizer is included in the recasepunc.py script. It is a modified version of the BERT tokenizer which keeps punctuation when lower-casing. Example usage:

xzcat cc-100/$lang.txt.xz | python recasepunc.py --lang=$lang preprocess 100000000 > data/$lang-100M.txt