L i[HddlmZddlmZddlmZddlmZmZdede fdZ y) )Path)Any)TikTokenConverter)TIKTOKEN_VOCAB_FILETOKENIZER_FILEencoding output_dirct|}|jd|dz tz }|tz }t |j }t |j } ddlm}ddlm }t|tr||}||j|t||j|j j#}|j%|y #t$r tdwxYw) a Converts given `tiktoken` encoding to `PretrainedTokenizerFast` and saves the configuration of converted tokenizer on disk. Args: encoding (`str` or `tiktoken.Encoding`): Tokenizer from `tiktoken` library. If `encoding` is `str`, the tokenizer will be loaded with `tiktoken.get_encoding(encoding)`. output_dir (`str`): Save path for converted tokenizer configuration file. T)exist_oktiktokenr) get_encoding)dump_tiktoken_bpezY`tiktoken` is required to save a `tiktoken` file. Install it with `pip install tiktoken`.) vocab_filepatternadditional_special_tokensN)rmkdirrrstrabsoluter r tiktoken.loadr isinstance_mergeable_ranks ImportError ValueErrorr_pat_str_special_tokens convertedsave) rr save_filetokenizer_filesave_file_absoluteoutput_file_absoluter r tokenizers h/mnt/ssd/data/python-lab/Trading/venv/lib/python3.12/site-packages/transformers/integrations/tiktoken.pyconvert_tiktoken_to_fastr$sj!Jd#Z'*==I.0NY//12~6689 v)3 h $#H-H(335GH"%x/@/@\d\t\tikNN'( vtuuvs &7CC4N) pathlibrtypingr#transformers.convert_slow_tokenizerr$transformers.tokenization_utils_fastrrrr$r#r+s$AT#)s#)#)r*