View Source on GitHub

maze_dataset.tokenization.common

common code for various tokenizers

View Source

1"common code for various tokenizers"
2
3
4class TokenError(ValueError):
5	"""error for tokenization"""
6
7	pass

class TokenError(builtins.ValueError):

View Source

5class TokenError(ValueError):
6	"""error for tokenization"""
7
8	pass

error for tokenization