Coverage for maze_dataset/tokenization/common.py: 100%

Shortcuts on this page

r m x toggle line displays

j k next/prev highlighted chunk

0 (zero) top of page

1 (one) first highlighted chunk

[ ] prev/next file

u up to the index

? show/hide this help

2 statements

1"common code for various tokenizers"

4class TokenError(ValueError):

5 """error for tokenization"""

7 pass