L i:dZddlZddlZddlZddlZddlZddlmZddlm Z m Z m Z ddl m Z ddlmZddlmZddlmZmZmZmZmZmZddlZdd lmZd d lmZd d lm Z d d l!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:m;Z;mm?Z?d dl@mAZAer#e7rddlBZBe4rddlCZDe-rddlEmZFd8dZGdeHfdZIe6serddlJmKZLe6rddlJmMZMneddGddZMe;jeOZPeQdZReQdZSeTZUeHeTZVeHeQZWeXeTeTfZYeXeHeTeHeTfZZeXeHeQeHeQfZ[eejdeHejeHdfZ]dZ^dZ_dZ`d Zaejd!ZcGd"d#e$ZdGd$d%eZeGd&d'eZfGd(d)eZgGd*d+Zhd,Zid-Zjd.Zke(ekGd/d0ehe&Zld1eHeTd2eTfd3Zme*eljel_neljj8eljjjd4d5d67elj_yy)9a" Base classes common to both the slow and the fast tokenization classes: PreTrainedTokenizerBase (host all the user fronting encoding methods) Special token mixing (host the special tokens logic) and BatchEncoding (wrap the dictionary of output with special method for the Fast tokenizers) N)UserDict)MappingSequenceSized)contextmanager) dataclass)Path) TYPE_CHECKINGAnyCallable NamedTupleOptionalUnion)version) __version__)custom_object_save)CHAT_TEMPLATE_DIRCHAT_TEMPLATE_FILE ExplicitEnumPaddingStrategyPushToHubMixin TensorTypeadd_end_docstrings cached_file copy_func download_urlextract_commit_hashis_flax_available is_jax_tensoris_mlx_availableis_numpy_arrayis_offline_modeis_protobuf_available is_remote_urlis_tf_available is_tf_tensoris_tokenizers_availableis_torch_availableis_torch_deviceis_torch_tensorlist_repo_templatesloggingrequires_backends to_py_obj)render_jinja_template)PROTOBUF_IMPORT_ERRORcbtrddlm}|Stt j |)Nr) DecodeError)r$google.protobuf.messager3 ImportErrorr1format) error_messager3s j/mnt/ssd/data/python-lab/Trading/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.pyimport_protobuf_decode_errorr9Qs(7/66}EFFarrcg}t|dkDrL|D]G}t|dttfr|j t |7|j |I|SNr)len isinstancelisttupleextendflattenappend)r;ressub_arrs r8rCrCZsW C 3x!| $G#a&4-0 77+, 7#  $ Jr:)Encoding) AddedTokenFT)frozeneqc,eZdZdZ ddefdZdZdZy)rHa! AddedToken represents a token to be added to a Tokenizer An AddedToken can have special options defining the way it should behave. The `normalized` will default to `not special` if it is not specified, similarly to the definition in `tokenizers`. Ncontentcn||_||_||_||_||_|||_y| |_yN)rL single_wordlstriprstripspecial normalized)selfrLrOrPrQrRrSs r8__init__zAddedToken.__init__vs<#DL*D  DK DK"DL,6,BjDOG DOr:c|jSrN__dict__rTs r8 __getstate__zAddedToken.__getstate__s == r:c|jSrN)rLrYs r8__str__zAddedToken.__str__s << r:)FFFFN)__name__ __module__ __qualname____doc__strrUrZr\r:r8rHrHls' jn T T ! r:rHgꌠ9Y>)Fg@xD torch.Tensorzspecial_tokens_map.jsonzadded_tokens.jsonztokenizer_config.jsonztokenizer.jsonztokenizer\.(.*)\.jsonc eZdZdZdZdZdZdZy)TruncationStrategyz Possible values for the `truncation` argument in [`PreTrainedTokenizerBase.__call__`]. Useful for tab-completion in an IDE. only_first only_second longest_firstdo_not_truncateN)r]r^r_r` ONLY_FIRST ONLY_SECOND LONGEST_FIRSTDO_NOT_TRUNCATErbr:r8reres JK#M'Or:rec&eZdZUdZeed<eed<y)CharSpanz Character span in the original string. Args: start (`int`): Index of the first character in the original string. end (`int`): Index of the character following the last character in the original string. startendNr]r^r_r`int__annotations__rbr:r8roro J Hr:roc&eZdZUdZeed<eed<y) TokenSpanz Token span in an encoded string (list of tokens). Args: start (`int`): Index of the first token in the span. end (`int`): Index of the token following the last token in the span. rprqNrrrbr:r8rwrwrur:rwc eZdZdZ d,deeeefdeee e e fdedee fde dee f fd Zed ee fd Zed e fd Zd ee efd eee ffdZd efdZdZdZed eee fdZd-de d eefdZd-de d eee fdZd-de d eee fdZd-de d eee fdZd.de dee d e fdZd.de dee d e fdZ d/de dee de d eefdZd.de dee d ee fd Z! d/d!e d"ee de d e fd#Z" d/de dee de d e fd$Z#d/d!e d"ee de d e fd%Z$ d0deeee fde fd&Z%dd'd(eed)fd*e d dfd+Z&xZ'S)1 BatchEncodinga Holds the output of the [`~tokenization_utils_base.PreTrainedTokenizerBase.__call__`], [`~tokenization_utils_base.PreTrainedTokenizerBase.encode_plus`] and [`~tokenization_utils_base.PreTrainedTokenizerBase.batch_encode_plus`] methods (tokens, attention_masks, etc). This class is derived from a python dictionary and can be used as a dictionary. In addition, this class exposes utility methods to map from word/character space to token space. Args: data (`dict`, *optional*): Dictionary of lists/arrays/tensors returned by the `__call__`/`encode_plus`/`batch_encode_plus` methods ('input_ids', 'attention_mask', etc.). encoding (`tokenizers.Encoding` or `Sequence[tokenizers.Encoding]`, *optional*): If the tokenizer is a fast tokenizer which outputs additional information like mapping from word/character space to token space the `tokenizers.Encoding` instance or list of instance (for batches) hold this information. tensor_type (`Union[None, str, TensorType]`, *optional*): You can give a tensor_type here to convert the lists of integers in PyTorch/TensorFlow/Numpy Tensors at initialization. prepend_batch_axis (`bool`, *optional*, defaults to `False`): Whether or not to add a batch axis when converting to tensors (see `tensor_type` above). Note that this parameter has an effect if the parameter `tensor_type` is set, *otherwise has no effect*. n_sequences (`Optional[int]`, *optional*): You can give a tensor_type here to convert the lists of integers in PyTorch/TensorFlow/Numpy Tensors at initialization. NFdataencoding tensor_typeprepend_batch_axis n_sequencesct|||t|tr|g}||_|||r|dj }||_|j||y)Nrr|r})superrUr? EncodingFast _encodingsr~ _n_sequencesconvert_to_tensors)rTrzr{r|r}r~ __class__s r8rUzBatchEncoding.__init__si   Jx$F zH"  8#7H"1+11K' KL^_r:returnc|jS)z `Optional[int]`: The number of sequences used to generate each sample from the batch encoded in this [`BatchEncoding`]. Currently can be one of `None` (unknown), `1` (a single sentence) or `2` (a pair of sentences) )rrYs r8r~zBatchEncoding.n_sequencess   r:c|jduS)z `bool`: Indicate whether this [`BatchEncoding`] was generated from the result of a [`PreTrainedTokenizerFast`] or not. NrrYs r8is_fastzBatchEncoding.is_fasts d**r:itemc t|tr|j|S|j|j|St|tr+|jDcic]}||j||c}St dcc}w)a If the key is a string, returns the value of the dict associated to `key` ('input_ids', 'attention_mask', etc.). If the key is an integer, get the `tokenizers.Encoding` for batch item with index `key`. If the key is a slice, returns the value of the dict associated to `key` ('input_ids', 'attention_mask', etc.) with the constraint of slice. zInvalid key. Only three types of key are available: (1) string, (2) integers for backend Encoding, and (3) slices for data subsetting.)r?rarzrsliceKeyError)rTrkeys r8 __getitem__zBatchEncoding.__getitem__ s dC 99T? " __ (??4( ( e $9=C#C3--C Ce DsBcH |j|S#t$rtwxYwrN)rzrAttributeError)rTrs r8 __getattr__zBatchEncoding.__getattr__!s* !99T? " !  !s!c4|j|jdS)N)rz encodingsrzrrYs r8rZzBatchEncoding.__getstate__'s @@r:c>d|vr |d|_d|vr |d|_yy)Nrzrr)rTstates r8 __setstate__zBatchEncoding.__setstate__*s- U?f DI % #K0DO r:c|jS)z `Optional[list[tokenizers.Encoding]]`: The list all encodings from the tokenization process. Returns `None` if the input was tokenized through Python (i.e., not a fast) tokenizer. rrYs r8rzBatchEncoding.encodings5s r: batch_indexcb|js td|j|jS)a Return the list of tokens (sub-parts of the input strings after word/subword splitting and before conversion to integer indices) at a given batch index (only works for the output of a fast tokenizer). Args: batch_index (`int`, *optional*, defaults to 0): The index to access in the batch. Returns: `list[str]`: The list of tokens at that index. zgtokens() is not available when using non-fast tokenizers (e.g. instance of a `XxxTokenizerFast` class).)r ValueErrortokensrTrs r8rzBatchEncoding.tokens=s4 {+222r:cb|js td|j|jS)a Return a list mapping the tokens to the id of their original sentences: - `None` for special tokens added around or between sequences, - `0` for tokens corresponding to words in the first sequence, - `1` for tokens corresponding to words in the second sequence when a pair of sequences was jointly encoded. Args: batch_index (`int`, *optional*, defaults to 0): The index to access in the batch. Returns: `list[Optional[int]]`: A list indicating the sequence id corresponding to each token. Special tokens added by the tokenizer are mapped to `None` and other tokens are mapped to the index of their corresponding sequence. zmsequence_ids() is not available when using non-fast tokenizers (e.g. instance of a `XxxTokenizerFast` class).)rr sequence_idsrs r8rzBatchEncoding.sequence_idsOs4" {+888r:c|js tdtjdt|j |S)> Return a list mapping the tokens to their actual word in the initial sentence for a fast tokenizer. Args: batch_index (`int`, *optional*, defaults to 0): The index to access in the batch. Returns: `list[Optional[int]]`: A list indicating the word corresponding to each token. Special tokens added by the tokenizer are mapped to `None` and other tokens are mapped to the index of their corresponding word (several tokens will be mapped to the same word index if they are parts of that word). zfwords() is not available when using non-fast tokenizers (e.g. instance of a `XxxTokenizerFast` class).z`BatchEncoding.words()` property is deprecated and should be replaced with the identical, but more self-explanatory `BatchEncoding.word_ids()` property.)rrwarningswarn FutureWarningword_idsrs r8wordszBatchEncoding.wordsgsE    M  }}[))r:cb|js td|j|jS)rziword_ids() is not available when using non-fast tokenizers (e.g. instance of a `XxxTokenizerFast` class).)rrrrs r8rzBatchEncoding.word_idss4 {+444r:batch_or_token_index token_indexc|js td||}nd}|}|dkr|j|z}|dkr|j|z}|j|j |S)a Get the index of the sequence represented by the given token. In the general use case, this method returns `0` for a single sequence or the first sequence of a pair, and `1` for the second sequence of a pair Can be called as: - `self.token_to_sequence(token_index)` if batch size is 1 - `self.token_to_sequence(batch_index, token_index)` if batch size is greater than 1 This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e., words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words. Args: batch_or_token_index (`int`): Index of the sequence in the batch. If the batch only comprises one sequence, this can be the index of the token in the sequence. token_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the token in the sequence. Returns: `int`: Index of the word in the input sequence. zGtoken_to_sequence() is not available when using Python based tokenizersr)rr _batch_size_seq_lentoken_to_sequencerTrrrs r8rzBatchEncoding.token_to_sequencesr4fg g  ".KK.K ?**[8K ?--+5K{+==kJJr:c|js td||}nd}|}|dkr|j|z}|dkr|j|z}|j|j |S)a  Get the index of the word corresponding (i.e. comprising) to an encoded token in a sequence of the batch. Can be called as: - `self.token_to_word(token_index)` if batch size is 1 - `self.token_to_word(batch_index, token_index)` if batch size is greater than 1 This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e., words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words. Args: batch_or_token_index (`int`): Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the token in the sequence. token_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the token in the sequence. Returns: `int`: Index of the word in the input sequence. zCtoken_to_word() is not available when using Python based tokenizersr)rrrr token_to_wordrs r8rzBatchEncoding.token_to_wordsr2bc c  ".KK.K ?**[8K ?--+5K{+99+FFr:batch_or_word_index word_indexsequence_indexc|js td||}nd}|}|dkr|j|z}|dkr|j|z}|j|j ||}|t |SdS)a- Get the encoded token span corresponding to a word in a sequence of the batch. Token spans are returned as a [`~tokenization_utils_base.TokenSpan`] with: - **start** -- Index of the first token. - **end** -- Index of the token following the last token. Can be called as: - `self.word_to_tokens(word_index, sequence_index: int = 0)` if batch size is 1 - `self.word_to_tokens(batch_index, word_index, sequence_index: int = 0)` if batch size is greater or equal to 1 This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e. words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words. Args: batch_or_word_index (`int`): Index of the sequence in the batch. If the batch only comprises one sequence, this can be the index of the word in the sequence. word_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the word in the sequence. sequence_index (`int`, *optional*, defaults to 0): If pair of sequences are encoded in the batch this can be used to specify which sequence in the pair (0 or 1) the provided word index belongs to. Returns: ([`~tokenization_utils_base.TokenSpan`], *optional*): Span of tokens in the encoded sequence. Returns `None` if no tokens correspond to the word. This can happen especially when the token is a special token that has been used to format the tokenization. For example when we add a class token at the very beginning of the tokenization. zDword_to_tokens() is not available when using Python based tokenizersNr)rrrrword_to_tokensrw)rTrrrrspans r8rzBatchEncoding.word_to_tokenssNcd d  !-KK,J ?**[8K >3J{+:::~V#'#3y$==r:c|js td||}nd}|}|j|j|}|t|SdS)a Get the character span corresponding to an encoded token in a sequence of the batch. Character spans are returned as a [`~tokenization_utils_base.CharSpan`] with: - **start** -- Index of the first character in the original string associated to the token. - **end** -- Index of the character following the last character in the original string associated to the token. Can be called as: - `self.token_to_chars(token_index)` if batch size is 1 - `self.token_to_chars(batch_index, token_index)` if batch size is greater or equal to 1 Args: batch_or_token_index (`int`): Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the token in the sequence. token_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the token or tokens in the sequence. Returns: [`~tokenization_utils_base.CharSpan`]: Span of characters in the original string, or None, if the token (e.g. , ) doesn't correspond to any chars in the origin string. zDtoken_to_chars() is not available when using Python based tokenizersNr)rrtoken_to_charsro)rTrrr span_indicess r8rzBatchEncoding.token_to_charssZ8cd d  ".KK.K{3BB;O *6*Bx&LLr:batch_or_char_index char_indexc|js td||}nd}|}|j|j||S)a Get the index of the token in the encoded output comprising a character in the original string for a sequence of the batch. Can be called as: - `self.char_to_token(char_index)` if batch size is 1 - `self.char_to_token(batch_index, char_index)` if batch size is greater or equal to 1 This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e. words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words. Args: batch_or_char_index (`int`): Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the word in the sequence char_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the word in the sequence. sequence_index (`int`, *optional*, defaults to 0): If pair of sequences are encoded in the batch this can be used to specify which sequence in the pair (0 or 1) the provided character index belongs to. Returns: `int`: Index of the token, or None if the char index refers to a whitespace only token and whitespace is trimmed with `trim_offsets=True`. zCchar_to_token() is not available when using Python based tokenizersr)rr char_to_tokenrTrrrrs r8rzBatchEncoding.char_to_token;sIBbc c  !-KK,J{+99*nUUr:c|js td||}nd}|}t|j|j||S)a/ Get the character span in the original string corresponding to given word in a sequence of the batch. Character spans are returned as a CharSpan NamedTuple with: - start: index of the first character in the original string - end: index of the character following the last character in the original string Can be called as: - `self.word_to_chars(word_index)` if batch size is 1 - `self.word_to_chars(batch_index, word_index)` if batch size is greater or equal to 1 Args: batch_or_word_index (`int`): Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the word in the sequence word_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the word in the sequence. sequence_index (`int`, *optional*, defaults to 0): If pair of sequences are encoded in the batch this can be used to specify which sequence in the pair (0 or 1) the provided word index belongs to. Returns: `CharSpan` or `list[CharSpan]`: Span(s) of the associated character or characters in the string. CharSpan are NamedTuple with: - start: index of the first character associated to the token in the original string - end: index of the character following the last character associated to the token in the original string zCword_to_chars() is not available when using Python based tokenizersr)rrro word_to_chars)rTrrrrs r8rzBatchEncoding.word_to_charsesOHbc c  !-KK,J$//+6DDZQ_`bbr:c|js td||}nd}|}|j|j||S)a@ Get the word in the original string corresponding to a character in the original string of a sequence of the batch. Can be called as: - `self.char_to_word(char_index)` if batch size is 1 - `self.char_to_word(batch_index, char_index)` if batch size is greater than 1 This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e. words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words. Args: batch_or_char_index (`int`): Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the character in the original string. char_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the character in the original string. sequence_index (`int`, *optional*, defaults to 0): If pair of sequences are encoded in the batch this can be used to specify which sequence in the pair (0 or 1) the provided character index belongs to. Returns: `int` or `list[int]`: Index or indices of the associated encoded token(s). zBchar_to_word() is not available when using Python based tokenizersr)rr char_to_wordrs r8rzBatchEncoding.char_to_wordsH<ab b  !-KK,J{+88^TTr:c ||St|ts t|}|tjk(r,ts t dddl d fd  j }n|tjk(r,ts t dddl d fd  j }n|tjk(r(ts t dddl m d fd t}nF|tjk(r't!s t d ddlm d fd  fd }n dfd t&}|j)D] \}} |r|g}||s |}|||<"|S#t*$r&}|d k(r t-d|t-d|d|d}~wwxYw)a Convert the inner content to tensors. Args: tensor_type (`str` or [`~utils.TensorType`], *optional*): The type of tensors to use. If `str`, should be one of the values of the enum [`~utils.TensorType`]. If `None`, no modification is done. prepend_batch_axis (`int`, *optional*, defaults to `False`): Whether or not to add the batch dimension during the conversion. NzSUnable to convert output to TensorFlow tensors format, TensorFlow is not installed.rcvtt|dk(r| j}j||SNrdtype)r>rCint32constant)valuertfs r8 as_tensorz3BatchEncoding.convert_to_tensors..as_tensors4wu~&!+ HHE"r{{566r:zMUnable to convert output to PyTorch tensors format, PyTorch is not installed.c6t|trPt|dkDrBt|dtjr%j tj |Stt|dk(r| j}j||Sr) r?r@r>npndarray from_numpyarrayrCint64tensor)rrtorchs r8rz3BatchEncoding.convert_to_tensors..as_tensorsxeT*s5zA~*USTXWYWaWaBb+5++BHHUO<<wu~&!+ !KKE#u||E77r:zEUnable to convert output to JAX tensors format, JAX is not installed.cvtt|dk(r| j}j||Srr>rCrr)rrjnps r8rz3BatchEncoding.convert_to_tensors..as_tensors4wu~&!+ IIE syye44r:zEUnable to convert output to MLX tensors format, MLX is not installed.cttt|dk(r| j}j||Srr)rrmxs r8rz3BatchEncoding.convert_to_tensors..as_tensors4wu~&!+ HHExxUx33r:c0t|jSrN)r?r)objrs r8 is_tensorz3BatchEncoding.convert_to_tensors..is_tensors!#rxx00r:ct|ttfrt|dkDrt|dtttj fr`|Dcgc] }t|}}tt |dkDr1|/|Dcgc]}t j|c}t}tt|dk(r|tj}t j||Scc}wcc}w)Nrrr) r?r@rAr>rrsetasarrayobjectrCr)rrval value_lensrs r8rz3BatchEncoding.convert_to_tensors..as_tensorsutUm4E Q"58dE2::-FG6;!c}}|_|Stjdt|d|Scc}}w)au Send all values to device by calling `v.to(device, non_blocking=non_blocking)` (PyTorch only). Args: device (`str` or `torch.device`): The device to put the tensors on. non_blocking (`bool`): Whether to perform the copy asynchronously. Returns: [`BatchEncoding`]: The same instance after modification. rto)rrz+Attempting to cast a BatchEncoding to type z. This is not supported.) r.r?rar*rsrzrhasattrcallablerloggerwarning)rTrrkvs r8rzBatchEncoding.to4s $ * fc "of&=FTWAX!IIOO-AqWQPTEUZbcdcgcgZh144vL4AnooDI   NNHV Umn o sAC)NNNFN)rrNr=NF)(r]r^r_r`rdictrar rrrrboolrsrUpropertyr~rrrrZrr@rrrrrrrrwrrorrrrrr __classcell__rs@r8ryrys':*.JN48#(%) `tCH~&`5x /E!EFG`4j01 ` ! ` c] `.!Xc]!!+++c3hE#|:K4L,!! A18D$673#3d3i3$99D#4G90**T(3--@*05C5Xc]0C5&%Kc%KQT %Kad%KN$G#$GHSM$G]`$GNab3>#&3>4Z]3> ) 3>j%M3%MXc]%M^fgo^p%MPab(V#&(V4z.SpecialTokensMixin.__init__..sOAz!c:->?O "z2One of the tokens is not a string or an AddedTokenSpecial token - has to be either str or AddedToken but got: )_pad_token_type_idverboserfromkeysSPECIAL_TOKENS_ATTRIBUTES_special_tokens_maprr?r@rAallsetattrrarH TypeErrortype)rTrkwargsrrs r8rUzSpecialTokensMixin.__init__us"# #'==1O1O#P @B  !<= !,,. vJC}d44455%edE];dveWLc=dd;OOOLOD#u-Z'89D#u-#nSE9fgklqgrfs$tuu vr:rcftjd|j|jdS)z The `sanitize_special_tokens` is now deprecated kept for backward compatibility and will be removed in transformers v5. zAThe `sanitize_special_tokens` will be removed in transformers v5.Tspecial_tokens)r warning_once add_tokensall_special_tokens_extendedrYs r8sanitize_special_tokensz*SpecialTokensMixin.sanitize_special_tokenss- _`t??PTUUr:special_tokens_dictc |syg}|jD]\}}||jvs Jd|d|jrtj d|d|d|dk(rt |t tfrtd|DsJd |d |d g}|D]M}t |trt|d d d d }|st||jvr=|j|O|r%t|dkDrt||t |n|jdj!|||z }#t |ttfst#d|d |dt |trt|d d d d }t |tr t|||||vs|j||j%|d }|S)a# Add a dictionary of special tokens (eos, pad, cls, etc.) to the encoder and link them to class attributes. If special tokens are NOT in the vocabulary, they are added to it (indexed starting from the last index of the current vocabulary). When adding new tokens to the vocabulary, you should make sure to also resize the token embedding matrix of the model so that its embedding matrix matches the tokenizer. In order to do that, please use the [`~PreTrainedModel.resize_token_embeddings`] method. Using `add_special_tokens` will ensure your special tokens can be used in several ways: - Special tokens can be skipped when decoding using `skip_special_tokens = True`. - Special tokens are carefully handled by the tokenizer (they are never split), similar to `AddedTokens`. - You can easily refer to special tokens using tokenizer class attributes like `tokenizer.cls_token`. This makes it easy to develop model-agnostic training and fine-tuning scripts. When possible, special tokens are already registered for provided pretrained models (for instance [`BertTokenizer`] `cls_token` is already registered to be `'[CLS]'` and XLM's one is also registered to be `''`). Args: special_tokens_dict (dictionary *str* to *str*, `tokenizers.AddedToken`, or `Sequence[Union[str, AddedToken]]`): Keys should be in the list of predefined special attributes: [`bos_token`, `eos_token`, `unk_token`, `sep_token`, `pad_token`, `cls_token`, `mask_token`, `additional_special_tokens`]. Tokens are only added if they are not already in the vocabulary (tested by checking if the tokenizer assign the index of the `unk_token` to them). replace_additional_special_tokens (`bool`, *optional*,, defaults to `True`): If `True`, the existing list of additional special tokens will be replaced by the list provided in `special_tokens_dict`. Otherwise, `self._special_tokens_map["additional_special_tokens"]` is just extended. In the former case, the tokens will NOT be removed from the tokenizer's full vocabulary - they are only being flagged as non-special tokens. Remember, this only affects which tokens are skipped during decoding, not the `added_tokens_encoder` and `added_tokens_decoder`. This means that the previous `additional_special_tokens` are still added tokens, and will not be split by the model. Returns: `int`: Number of tokens added to the vocabulary. Examples: ```python # Let's see how to add a new classification token to GPT-2 tokenizer = GPT2Tokenizer.from_pretrained("openai-community/gpt2") model = GPT2Model.from_pretrained("openai-community/gpt2") special_tokens_dict = {"cls_token": ""} num_added_toks = tokenizer.add_special_tokens(special_tokens_dict) print("We have added", num_added_toks, "tokens") # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e., the length of the tokenizer. model.resize_token_embeddings(len(tokenizer)) assert tokenizer.cls_token == "" ```rzKey z is not a special tokenz Assigning z to the z key of the tokenizerr c3HK|]}t|ttfywrNrrs r8rz8SpecialTokensMixin.add_special_tokens..s?pef 1sT^N_@`?przTokens z for key z* should all be str or AddedToken instancesFTrQrPrSrRzToken z* should be a str or an AddedToken instancer")rrrrinfor?r@rArrarHr rDr>rrrBrr%)rTr(!replace_additional_special_tokens added_tokensrrto_addtokens r8add_special_tokensz%SpecialTokensMixin.add_special_tokenssx# -335" /JC$888 ]DE\:] ]8|| jxuJJr:ctrNNotImplementedErrorr5s r8r4zSpecialTokensMixin._add_tokens/!!r:c|jS)zH `int`: Id of the padding token type in the vocabulary. )rrYs r8pad_token_type_idz$SpecialTokensMixin.pad_token_type_id2s &&&r:c|}|jdxs|jd}|r|jds|ddn|dd}jjddtfd||fDrv|r9|5|dk7rj |n|Dcgc]}j |c}}|}|dk7r&t |t tfs|td||j|<yt-||ycc}w) N_id_idsrc3@K|]}|jdvywrNrWrnamerTs r8rz1SpecialTokensMixin.__setattr__..?'N =ADDMM"78 8N r z%Cannot set a non-string value as the ) endswithrXgetanyconvert_ids_to_tokensr?rarHrrr __setattr__)rTrrkey_without_idkey_is_special_idrrs` r8rKzSpecialTokensMixin.__setattr__9sLL/G3<<3G -0\\&-AS"Xs3BxN ==  2D 9 E#N FI>EZN K !$"==2259INO#d88=O %11*USR\L]:^chct #H!NOO,1D $ $S ) G U +PsC?c|}|jdxs|jd}|r|jds|ddn|dd}jjddtfd||fDrjd}|sY||&jrt j d|dy||}|d k7r t|S|Dcgc] }t|c}St|}|j|SdS|jvr$tjjd |t9|Scc}w) Nr=r>r?r@rc3@K|]}|jdvywrBrWrCs r8rz1SpecialTokensMixin.__getattr__..WrErFzUsing z, but it is not set yet.r z has no attribute )rGrXrHrIrrerrorragetattrconvert_tokens_to_idsrrr]rr) rTrrLrMrrtokattr_as_tokensrs ` r8rzSpecialTokensMixin.__getattr__QsZLL/G3<<3G -0\\&-AS"Xs3BxN ==  2D 9 E#N FI>EZN K #'--0E"F $&s+3|| vcU2J%KL+C0%(,G%Gs5zhbgMh[^cRUhMhh!(~!>ESE_t11.Aieii dmm # DNN$;$;#<'`, `''`, etc.). Convert potential tokens of `tokenizers.AddedToken` type to string. )rrQrTset_attrattr attr_values r8special_tokens_mapz%SpecialTokensMixin.special_tokens_mapks=22 ,D t,J!+ ,r:cZi}|jD]}|j|}|s|||<|S)a `dict[str, Union[str, tokenizers.AddedToken, list[Union[str, tokenizers.AddedToken]]]]`: A dictionary mapping special token class attributes (`cls_token`, `unk_token`, etc.) to their values (`''`, `''`, etc.). Don't convert tokens of `tokenizers.AddedToken` type to string so they can be used to control more finely how special tokens are tokenized. )rrrVs r8special_tokens_map_extendedz.SpecialTokensMixin.special_tokens_map_extendedzsA22 ,D11$7J!+ ,r:cRg}t}|jjD]x}t|tt fr|Dcgc]}t ||vs|}}nt ||vr|gng}|jtt ||j|z|Scc}w)a `list[Union[str, tokenizers.AddedToken]]`: All the special tokens (`''`, `''`, etc.), the order has nothing to do with the index of each tokens. If you want to know the correct indices, check `self.added_tokens_encoder`. We can't create an order anymore as the keys are `AddedTokens` and not `Strings`. Don't convert tokens of `tokenizers.AddedToken` type to string so they can be used to control more finely how special tokens are tokenized. ) rr\valuesr?r@rAraupdatemaprB)rT all_tokensseenrr0 tokens_to_adds r8r&z.SpecialTokensMixin.all_special_tokens_extendeds u55<<> -E%$/49 T5SZt=S T T+.u:T+Ar KKC/ 0   m ,  - !Us B$B$cT|jDcgc] }t|}}|Scc}w)z `list[str]`: A list of the unique special tokens (`''`, `''`, ..., etc.). Convert tokens of `tokenizers.AddedToken` type to string. )r&ra)rTsall_tokss r8all_special_tokensz%SpecialTokensMixin.all_special_tokenss,%)$D$DEqCFEEFs%c@|j}|j|}|S)zy `list[int]`: List the ids of the special tokens(`''`, `''`, etc.) mapped to class attributes. )rgrR)rTrfall_idss r8all_special_idsz"SpecialTokensMixin.all_special_idss$ **,,X6r:c |jt|jz|_|jD]D\}}t |t t fr||j|<,td|dt|y)a Adds new special tokens to the "SPECIAL_TOKENS_ATTRIBUTES" list which will be part of "self.special_tokens" and saved as a special token in tokenizer's config. This allows us to dynamically add new model-type specific tokens after initializing the tokenizer. For example: if the model tokenizers is multimodal, we can support special image or audio tokens. rrN) rr@keysrr?rarHrrr)rTr#rrs r8"_set_model_specific_special_tokensz5SpecialTokensMixin._set_model_specific_special_tokenss*.)G)G$~ObObOdJe)e&(..0 rJC%#z!2305((-.5bcghmcnbo pqq  rr:F)T)r]r^r_r`rrUrsr'rrarrHrr1rr%r@r4rr;rKrrZr\r&rgrjrmrrs@r8rrNs6 !v2VV+/f!#uS*huSR\_G]>^-^'_"_`f fRlq/KZ%Z:P1Q QR/Kdh/K /Kb"eDItJ7G,G&H"Z^"kn"'3'' ,0,4 DeCcN.C)C$D   T#uS*dSXY\^hYhSiNj=j7k2k-l  T%Z2H-I(DIc rc rr:ra  add_special_tokens (`bool`, *optional*, defaults to `True`): Whether or not to add special tokens when encoding the sequences. This will use the underlying `PretrainedTokenizerBase.build_inputs_with_special_tokens` function, which defines which tokens are automatically added to the input ids. This is useful if you want to add `bos` or `eos` tokens automatically. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `False`): Activates and controls padding. Accepts the following values: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence is provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different lengths). truncation (`bool`, `str` or [`~tokenization_utils_base.TruncationStrategy`], *optional*, defaults to `False`): Activates and controls truncation. Accepts the following values: - `True` or `'longest_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will truncate token by token, removing a token from the longest sequence in the pair if a pair of sequences (or a batch of pairs) is provided. - `'only_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the first sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `'only_second'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the second sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `False` or `'do_not_truncate'` (default): No truncation (i.e., can output batch with sequence lengths greater than the model maximum admissible input size). max_length (`int`, *optional*): Controls the maximum length to use by one of the truncation/padding parameters. If left unset or set to `None`, this will use the predefined model maximum length if a maximum length is required by one of the truncation/padding parameters. If the model has no specific maximum input length (like XLNet) truncation/padding to a maximum length will be deactivated. stride (`int`, *optional*, defaults to 0): If set to a number along with `max_length`, the overflowing tokens returned when `return_overflowing_tokens=True` will contain some tokens from the end of the truncated sequence returned to provide some overlap between truncated and overflowing sequences. The value of this argument defines the number of overlapping tokens. is_split_into_words (`bool`, *optional*, defaults to `False`): Whether or not the input is already pre-tokenized (e.g., split into words). If set to `True`, the tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace) which it will tokenize. This is useful for NER or token classification. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. Requires `padding` to be activated. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). padding_side (`str`, *optional*): The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return Numpy `np.ndarray` objects. a return_token_type_ids (`bool`, *optional*): Whether to return token type IDs. If left to the default, will return the token type IDs according to the specific tokenizer's default, defined by the `return_outputs` attribute. [What are token type IDs?](../glossary#token-type-ids) return_attention_mask (`bool`, *optional*): Whether to return the attention mask. If left to the default, will return the attention mask according to the specific tokenizer's default, defined by the `return_outputs` attribute. [What are attention masks?](../glossary#attention-mask) return_overflowing_tokens (`bool`, *optional*, defaults to `False`): Whether or not to return overflowing token sequences. If a pair of sequences of input ids (or a batch of pairs) is provided with `truncation_strategy = longest_first` or `True`, an error is raised instead of returning overflowing tokens. return_special_tokens_mask (`bool`, *optional*, defaults to `False`): Whether or not to return special tokens mask information. return_offsets_mapping (`bool`, *optional*, defaults to `False`): Whether or not to return `(char_start, char_end)` for each token. This is only available on fast tokenizers inheriting from [`PreTrainedTokenizerFast`], if using Python's tokenizer, this method will raise `NotImplementedError`. return_length (`bool`, *optional*, defaults to `False`): Whether or not to return the lengths of the encoded inputs. verbose (`bool`, *optional*, defaults to `True`): Whether or not to print more information and warnings. **kwargs: passed to the `self.tokenize()` method Return: [`BatchEncoding`]: A [`BatchEncoding`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. [What are input IDs?](../glossary#input-ids) - **token_type_ids** -- List of token type ids to be fed to a model (when `return_token_type_ids=True` or if *"token_type_ids"* is in `self.model_input_names`). [What are token type IDs?](../glossary#token-type-ids) - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names`). [What are attention masks?](../glossary#attention-mask) - **overflowing_tokens** -- List of overflowing tokens sequences (when a `max_length` is specified and `return_overflowing_tokens=True`). - **num_truncated_tokens** -- Number of tokens truncated (when a `max_length` is specified and `return_overflowing_tokens=True`). - **special_tokens_mask** -- List of 0s and 1s, with 1 specifying added special tokens and 0 specifying regular sequence tokens (when `add_special_tokens=True` and `return_special_tokens_mask=True`). - **length** -- The length of the inputs (when `return_length=True`) a Class attributes (overridden by derived classes) - **vocab_files_names** (`dict[str, str]`) -- A dictionary with, as keys, the `__init__` keyword name of each vocabulary file required by the model, and as associated values, the filename for saving the associated file (string). - **pretrained_vocab_files_map** (`dict[str, dict[str, str]]`) -- A dictionary of dictionaries, with the high-level keys being the `__init__` keyword name of each vocabulary file required by the model, the low-level being the `short-cut-names` of the pretrained models with, as associated values, the `url` to the associated pretrained vocabulary file. - **model_input_names** (`list[str]`) -- A list of inputs expected in the forward pass of the model. - **padding_side** (`str`) -- The default value for the side on which the model should have padding applied. Should be `'right'` or `'left'`. - **truncation_side** (`str`) -- The default value for the side on which the model should have truncation applied. Should be `'right'` or `'left'`. Args: model_max_length (`int`, *optional*): The maximum length (in number of tokens) for the inputs to the transformer model. When the tokenizer is loaded with [`~tokenization_utils_base.PreTrainedTokenizerBase.from_pretrained`], this will be set to the value stored for the associated model in `max_model_input_sizes` (see above). If no value is provided, will default to VERY_LARGE_INTEGER (`int(1e30)`). padding_side (`str`, *optional*): The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. truncation_side (`str`, *optional*): The side on which the model should have truncation applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. chat_template (`str`, *optional*): A Jinja template string that will be used to format lists of chat messages. See https://huggingface.co/docs/transformers/chat_templating for a full description. model_input_names (`list[string]`, *optional*): The list of inputs accepted by the forward pass of the model (like `"token_type_ids"` or `"attention_mask"`). Default value is picked from the class attribute of the same name. bos_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the beginning of a sentence. Will be associated to `self.bos_token` and `self.bos_token_id`. eos_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the end of a sentence. Will be associated to `self.eos_token` and `self.eos_token_id`. unk_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing an out-of-vocabulary token. Will be associated to `self.unk_token` and `self.unk_token_id`. sep_token (`str` or `tokenizers.AddedToken`, *optional*): A special token separating two different sentences in the same input (used by BERT for instance). Will be associated to `self.sep_token` and `self.sep_token_id`. pad_token (`str` or `tokenizers.AddedToken`, *optional*): A special token used to make arrays of tokens the same size for batching purpose. Will then be ignored by attention mechanisms or loss computation. Will be associated to `self.pad_token` and `self.pad_token_id`. cls_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the class of the input (used by BERT for instance). Will be associated to `self.cls_token` and `self.cls_token_id`. mask_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing a masked token (used by masked-language modeling pretraining objectives, like BERT). Will be associated to `self.mask_token` and `self.mask_token_id`. additional_special_tokens (tuple or list of `str` or `tokenizers.AddedToken`, *optional*): A tuple or a list of additional special tokens. Add them here to ensure they are skipped when decoding with `skip_special_tokens` is set to True. If they are not part of the vocabulary, they will be added at the end of the vocabulary. clean_up_tokenization_spaces (`bool`, *optional*, defaults to `True`): Whether or not the model should cleanup the spaces that were added when splitting the input text during the tokenization process. split_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the special tokens should be split during the tokenization process. Passing will affect the internal state of the tokenizer. The default behavior is to not split special tokens. This means that if `` is the `bos_token`, then `tokenizer.tokenize("") = ['`]. Otherwise, if `split_special_tokens=True`, then `tokenizer.tokenize("")` will be give `['<','s', '>']`. c,eZdZUdZiZeeefed<iZeeeeeffed<dZ e eed<gdZ e eed<dZ eed <dZeed <dZfd Zed efd Zed efdZej*d efdZej*d efdZdefdZed eeeffdZd efdZd efdZd eeeffdZ ddee eeefe e eeeffde e eeefde e eeefde edededed eeeefd!ed"e ed#e eee fd$ed%ed&e eee!fd eee ee ee e ee"ffd'Z# dd(eeefd)e e eeefd e efd*Z$dde ede e ed efd+Z%e&ddddd,dd-d.eee'jPfd/e eee'jPfd0ed1ed2e eeefd3ef d4Z)e&ddddddd5d6Z*e+d7Z,e&dd8eee!ffd9Z-d:eee'jPfd;edZ. dd:eee'jPfd?e ed ddFee6e7e e6e e7fdMe ee6e7e e6e e7fdHed eeeefd!eeee9dfd"e edNedTedUe ed e ed#e eee fdVe edWe edXedYedZed[ed\ed^ed e"f(d_Z?e4e5e= ddFee6e7e8fdMe ee6e7e8fdHed eeeefd!eeee9dfd"e edNedTedUe ed e ed#e eee fdVe edWe edXedYedZed[ed\ed e"f&d`Z@ddeje9jddLddddddddddddfdFee6e7e8fdMe ee6e7e8fdHedaedbe9d"e edNedTedUe ed e ed#e eee fdVe edWe edXedYedZed[ed\ed^ed e"f(dcZCe4e5e= dddee e6e eDe e7e eEe e8e eFfdHed eeeefd!eeee9dfd"e edNedTedUe ed e ed#e eee fdVe edWe edXedYedZed[ed\ed^ed e"f&deZGdeje9jddLddddddddddddfddee e6e eDe e7e eEe e8e eFfdHedaedbe9d"e edNedTedUe ed e ed#e eee fdVe edWe edXedYedZed[ed\ed^ed e"f&dfZH ddgee"e e"eee8feee e8fe eee8ffd eeeefd"e edUe ed e edWe ed#e eee fd\ed e"fdhZI ddie edje e ed e efdkZJ ddie edje e ed e efdlZKe4e5e= ddme edne e edHed eeeefd!eeee9dfd"e edNedUe ed e ed#e eee fdVe edWe edXedYedZed[ed\edoed e"f&dpZL ddme edne e edqedbeee9fdNed e/e ee ee eff drZMdejdddfdgeeee8fe"fd"e edaedUe ed e edWe ed efdsZNdte ed efduZO ddvee ee e edwdxdyfdzed{e ed e efd|ZP dd}eee eeQjdxfdzed{e ed efd~ZS dd}eee efdzed{e ed efdZT ddie edje e eded e efdZUe+ded efdZVdme ed"e ed\efdZWdZXdZYeZdZ[e&ddZ\ dde ede e ed"e ede ed ed#e ed!ed e"fdZ]xZ^S)PreTrainedTokenizerBasez Base class for [`PreTrainedTokenizer`] and [`PreTrainedTokenizerFast`]. Handles shared (mostly boiler plate) methods for those two classes. vocab_files_namespretrained_vocab_files_mapN _auto_class) input_idstoken_type_idsattention_maskmodel_input_namesright padding_sidetruncation_sidec d|_|D]K}t||stt||s&t |d|d|j j tj||_ |jdd|_ |jdd|_ |jd|jdd}||nt|_|jd |j|_|jd vrt!d |j|jd |j"|_|j"d vrt!d |j"|jd|j$|_|jdd|_|jdd|_i|_d|_|jdd|_t1|j.t2t4fr&|j.Dcic] }|d|d c}|_t7|pdi||jdi|_|j=|j:ycc}w)Nrbz conflicts with the method z in name_or_pathprocessor_classmodel_max_lengthmax_lenry)rxleftzKPadding side should be selected between 'right' and 'left', current value: rzzNTruncation side should be selected between 'right' and 'left', current value: rwclean_up_tokenization_spacesFsplit_special_tokens chat_templaterDtemplateextra_special_tokensr") init_inputsrrrQrrr]copydeepcopy init_kwargspopr|_processor_classVERY_LARGE_INTEGERrryrrzrwrrdeprecation_warnings_in_target_context_managerrr?r@rArrUrrm)rTr rrrrs r8rUz PreTrainedTokenizerBase.__init__sL lCtS!hwtS/A&B$u,GuDQUQ_Q_QhQhPi%jkk l ==0"JJ~r: & + d((4- 8Z^YkYk!lX(6"2HZ4H"H!lD  "6"$*JJ/Er$J! //t?X?X/Y "msIrc@|j|jdz S)zW `int`: The maximum length of a sentence that can be fed to the model. Fpairrnum_special_tokens_to_addrYs r8max_len_single_sentencez/PreTrainedTokenizerBase.max_len_single_sentences# $$t'E'E5'E'QQQr:c@|j|jdz S)zi `int`: The maximum combined length of a pair of sentences that can be fed to the model. TrrrYs r8max_len_sentences_pairz.PreTrainedTokenizerBase.max_len_sentences_pairs# $$t'E'E4'E'PPPr:c||j|jdz k(rM|jrA|jj ddst j dd|jd<ytd)NFrrzXSetting 'max_len_single_sentence' is now deprecated. This value is automatically set up.TrrrrrHrrrrTrs r8rz/PreTrainedTokenizerBase.max_len_single_sentencesz D))D,J,JPU,J,VV V[_[g[g,,001JERnDHD % %&? @j r:c||j|jdz k(rM|jrA|jj ddst j dd|jd<ytd)NTrrFzWSetting 'max_len_sentences_pair' is now deprecated. This value is automatically set up.rrs r8rz.PreTrainedTokenizerBase.max_len_sentences_pairsv D))D,J,JPT,J,UU UZ^ZfZf,,001I5QmCGD % %&> ?vw wr:r~c||_y)z%Sets processor class as an attribute.N)r)rTr~s r8_set_processor_classz,PreTrainedTokenizerBase._set_processor_classs /r:ctrNr7rYs r8added_tokens_decoderz,PreTrainedTokenizerBase.added_tokens_decoders !##r:cdj|jjDcgc]\}}|d|jdc}}}|jj d|j d|jd|jd|jd|jd |jd |jd |jd |zd zScc}}w)Nz z: ,z(name_or_path='z', vocab_size=z, model_max_length=z , is_fast=z, padding_side='z', truncation_side='z', special_tokens=z, clean_up_tokenization_spaces=z, added_tokens_decoder={ z } ))joinrr__repr__rr]r| vocab_sizerrryrzrZr)rTrradded_tokens_decoder_reps r8rz PreTrainedTokenizerBase.__repr__s#);;TXTmTmTsTsTu/vDAq1#R ~Q0G/v#w ~~&&'t7H7H6IJ??++>t?T?T>UU_`d`l`l_mn"//00DTEYEYDZ[#6677VW[WxWxVyz* *-E  EHP  P 0ws C ctrNr7rYs r8__len__zPreTrainedTokenizerBase.__len__s !##r:ct)a Returns the vocabulary as a dictionary of token to index. `tokenizer.get_vocab()[token]` is equivalent to `tokenizer.convert_tokens_to_ids(token)` when `token` is in the vocab. Returns: `dict[str, int]`: The vocabulary. r7rYs r8 get_vocabz!PreTrainedTokenizerBase.get_vocabs "##r:FT conversationtools documentsradd_generation_promptcontinue_final_messagetokenizepadding truncation max_lengthreturn_tensors return_dictreturn_assistant_tokens_masktokenizer_kwargsc j| r |s td| r | s td|i}|j||}t|ttfr-t|dttfst |ddr|}d}n|g}d}|r|r td| r tdi|j |}td||||| ||d |\}}|s|d}|r||f|| | d| d |}| r| rg}|s| r|d }n|d g}tt|D]}dgt||z}||D]Y\}}|j||}|j||d z }|n+t||r|d zn t||D]}d ||< [|j||s| s|d}||d <| r|j| |S|d S|S)aZ Converts a list of dictionaries with `"role"` and `"content"` keys to a list of token ids. This method is intended for use with chat models, and will read the tokenizer's chat_template attribute to determine the format and control tokens to use when converting. Args: conversation (Union[list[dict[str, str]], list[list[dict[str, str]]]]): A list of dicts with "role" and "content" keys, representing the chat history so far. tools (`list[Union[Dict, Callable]]`, *optional*): A list of tools (callable functions) that will be accessible to the model. If the template does not support function calling, this argument will have no effect. Each tool should be passed as a JSON Schema, giving the name, description and argument types for the tool. See our [tool use guide](https://huggingface.co/docs/transformers/en/chat_extras#passing-tools) for more information. documents (`list[dict[str, str]]`, *optional*): A list of dicts representing documents that will be accessible to the model if it is performing RAG (retrieval-augmented generation). If the template does not support RAG, this argument will have no effect. We recommend that each document should be a dict containing "title" and "text" keys. chat_template (`str`, *optional*): A Jinja template to use for this conversion. It is usually not necessary to pass anything to this argument, as the model's template will be used by default. add_generation_prompt (bool, *optional*): If this is set, a prompt with the token(s) that indicate the start of an assistant message will be appended to the formatted output. This is useful when you want to generate a response from the model. Note that this argument will be passed to the chat template, and so it must be supported in the template for this argument to have any effect. continue_final_message (bool, *optional*): If this is set, the chat will be formatted so that the final message in the chat is open-ended, without any EOS tokens. The model will continue this message rather than starting a new one. This allows you to "prefill" part of the model's response for it. Cannot be used at the same time as `add_generation_prompt`. tokenize (`bool`, defaults to `True`): Whether to tokenize the output. If `False`, the output will be a string. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `False`): Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different lengths). truncation (`bool`, defaults to `False`): Whether to truncate sequences at the maximum length. Has no effect if tokenize is `False`. max_length (`int`, *optional*): Maximum length (in tokens) to use for padding or truncation. Has no effect if tokenize is `False`. If not specified, the tokenizer's `max_length` attribute will be used as a default. return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Has no effect if tokenize is `False`. Acceptable values are: - `'tf'`: Return TensorFlow `tf.Tensor` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. - `'jax'`: Return JAX `jnp.ndarray` objects. return_dict (`bool`, defaults to `False`): Whether to return a dictionary with named outputs. Has no effect if tokenize is `False`. tokenizer_kwargs (`dict[str: Any]`, *optional*): Additional kwargs to pass to the tokenizer. return_assistant_tokens_mask (`bool`, defaults to `False`): Whether to return a mask of the assistant generated tokens. For tokens generated by the assistant, the mask will contain 1. For user and system tokens, the mask will contain 0. This functionality is only available for chat templates that support it via the `{% generation %}` keyword. **kwargs: Additional kwargs to pass to the template renderer. Will be accessible by the chat template. Returns: `Union[list[int], Dict]`: A list of token ids representing the tokenized chat so far, including control tokens. This output is ready to pass to the model, either directly or via methods like `generate()`. If `return_dict` is set, will return a dict of tokenizer outputs instead. zr`return_dict=True` is incompatible with `tokenize=False`, because there is no dict of tokenizer outputs to return.zL`return_assistant_tokens_mask=True` is incompatible with `return_dict=False`rmessagesTFa continue_final_message and add_generation_prompt are not compatible. Use continue_final_message when you want the model to continue the final message, and add_generation_prompt when you want to add a header that will prompt it to start a new assistant message instead.zKcontinue_final_message is not compatible with return_assistant_tokens_mask.) conversationsrrrrrr)rrrr1rrtrassistant_masksr|rb) rget_chat_templater?r@rArrZr0ranger>rrDr)rTrrrrrrrrrrrrrrr r is_batchedtemplate_kwargs rendered_chatgeneration_indicesoutrrti current_maskassistant_start_charassistant_end_char start_token end_tokentoken_ids r8apply_chat_templatez+PreTrainedTokenizerBase.apply_chat_template stp x2  ( kl l  #! ..}eD lT5M 2 |Au 6',q/S]:^(MJ)NMJ !$ c, !noo?T44??,A - '')E#9"7 -  - ) ))!,M %%#(- #C/&(O!^$' $4 %(%5$6 "3y>2 =()sS1->'> HZ[\H];D02D*-*;*;A?S*TK(+(9(9!=ORS=S(TI*2 %,1+PYy1}_bclmnco_p,q;9: X 6; ;(..|< =&n*9!*<-.J ;'' r:messageconversation_historyc \d|vr td|t|dk(r|j|gfddd|S||gz}|j|fddd|}|j|fddd|}tt|t|}t |D]}||||k7s||dcS||dS)a Tokenize a single message. This method is a convenience wrapper around `apply_chat_template` that allows you to tokenize messages one by one. This is useful for things like token-by-token streaming. This method is not guaranteed to be perfect. For some models, it may be impossible to robustly tokenize single messages. For example, if the chat template adds tokens after each message, but also has a prefix that is added to the entire chat, it will be impossible to distinguish a chat-start-token from a message-start-token. In these cases, this method will do its best to find the correct tokenization, but it may not be perfect. **Note:** This method does not support `add_generation_prompt`. If you want to add a generation prompt, you should do it separately after tokenizing the conversation. Args: message (`dict`): A dictionary with "role" and "content" keys, representing the message to tokenize. conversation_history (`list[dict]`, *optional*): A list of dicts with "role" and "content" keys, representing the chat history so far. If you are tokenizing messages one by one, you should pass the previous messages in the conversation here. **kwargs: Additional kwargs to pass to the `apply_chat_template` method. Returns: `list[int]`: A list of token ids representing the tokenized message. rzz`encode_message_with_chat_template` does not support `add_generation_prompt`. Please add the generation prompt separately.NrFT)rr)rr>rminr) rTrrr rr prefix_tokensmin_lenrs r8!encode_message_with_chat_templatez9PreTrainedTokenizerBase.encode_message_with_chat_templates4 #f ,  '3/C+D+I+4++WIlU]alekl l+wi7 ))),me^bmflm000  8= PV c-(#f+6w "AQ6!9,abz! "ghr:c>t|jtr[|j}| ||vr||}|S|>| d|vr|d}|Sd|vr|d}|Stdt |j d|S|%|j|j}|Std|S)a Retrieve the chat template string used for tokenizing chat messages. This template is used internally by the `apply_chat_template` method and can also be used externally to retrieve the model's chat template for better generation tracking. Args: chat_template (`str`, *optional*): A Jinja template or the name of a template to use for this conversion. It is usually not necessary to pass anything to this argument, as the model's template will be used by default. tools (`list[Dict]`, *optional*): A list of tools (callable functions) that will be accessible to the model. If the template does not support function calling, this argument will have no effect. Each tool should be passed as a JSON Schema, giving the name, description and argument types for the tool. See our [chat templating guide](https://huggingface.co/docs/transformers/main/en/chat_templating#automated-function-conversion-for-tool-use) for more information. Returns: `str`: The chat template string. tool_usedefaultzThis model has multiple chat templates with no default specified! Please either pass a chat template or the name of the template you wish to use to the `chat_template` argument. Available template names are .a.Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating)r?rrrsortedrl)rTrr template_dicts r8rz)PreTrainedTokenizerBase.get_chat_templates, d(($ / ..M(]m-K -m < 43&$})D$1*$=M.--/$1)$|}d"|vrtA|d"}ddd||d<|rmtC|tD}|jGr|jId#D]8}|jJjMd$}tDd%|jJ|d&|<:n8tO|||||'D]%} | jMd$} tDd%| d$|d&| <'i}!|jQD]\}}"|"d|!|<||k(rBtjj|"r|"|!|<8t|"sDtS|"| (|!|<U t3||"||| | ||||| d|) |!|<t9|!||}|jQD]C\}}"||!vr |rtjd*|"&tjd*|"d+|!|E|jT|!||g|||||||d,| S#t4$rt6$r t5d|d|d|j dwxYw#1swYxYw#t4$rt6$r t5d|d|d|j dwxYw)-a Instantiate a [`~tokenization_utils_base.PreTrainedTokenizerBase`] (or a derived class) from a predefined tokenizer. Args: pretrained_model_name_or_path (`str` or `os.PathLike`): Can be either: - A string, the *model id* of a predefined tokenizer hosted inside a model repo on huggingface.co. - A path to a *directory* containing vocabulary files required by the tokenizer, for instance saved using the [`~tokenization_utils_base.PreTrainedTokenizerBase.save_pretrained`] method, e.g., `./my_model_directory/`. - (**Deprecated**, not applicable to all derived classes) A path or url to a single saved vocabulary file (if and only if the tokenizer only requires a single vocabulary file like Bert or XLNet), e.g., `./my_model_directory/vocab.txt`. cache_dir (`str` or `os.PathLike`, *optional*): Path to a directory in which a downloaded predefined tokenizer vocabulary files should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force the (re-)download the vocabulary files and override the cached versions if they exist. resume_download: Deprecated and ignored. All downloads are now resumed by default when possible. Will be removed in v5 of Transformers. proxies (`dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. token (`str` or *bool*, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated when running `hf auth login` (stored in `~/.huggingface`). local_files_only (`bool`, *optional*, defaults to `False`): Whether or not to only rely on local files and not to attempt to download any files. revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. subfolder (`str`, *optional*): In case the relevant files are located inside a subfolder of the model repo on huggingface.co (e.g. for facebook/rag-token-base), specify it here. inputs (additional positional arguments, *optional*): Will be passed along to the Tokenizer `__init__` method. trust_remote_code (`bool`, *optional*, defaults to `False`): Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to `True` for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine. kwargs (additional keyword arguments, *optional*): Will be passed to the Tokenizer `__init__` method. Can be used to set special tokens like `bos_token`, `eos_token`, `unk_token`, `sep_token`, `pad_token`, `cls_token`, `mask_token`, `additional_special_tokens`. See parameters in the `__init__` for more details. Passing `token=True` is required when you want to use a private model. Examples: ```python # We can't instantiate directly the base class *PreTrainedTokenizerBase* so let's show our examples on a derived class: BertTokenizer # Download vocabulary from huggingface.co and cache. tokenizer = BertTokenizer.from_pretrained("google-bert/bert-base-uncased") # Download vocabulary from huggingface.co (user-uploaded) and cache. tokenizer = BertTokenizer.from_pretrained("dbmdz/bert-base-german-cased") # If vocabulary files are in a directory (e.g. tokenizer was saved using *save_pretrained('./test/saved_model/')*) tokenizer = BertTokenizer.from_pretrained("./test/saved_model/") # If the tokenizer uses a single vocabulary file, you can point directly to this file tokenizer = BertTokenizer.from_pretrained("./test/saved_model/my_vocab.txt") # You can link tokens to special vocabulary when instantiating tokenizer = BertTokenizer.from_pretrained("google-bert/bert-base-uncased", unk_token="") # You should be sure '' is in the vocabulary when doing that. # Otherwise use tokenizer.add_special_tokens({'unk_token': ''}) instead) assert tokenizer.unk_token == "" ```resume_downloadNproxiesuse_auth_token subfolder_from_pipeline _from_autoF _commit_hash gguf_filerThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.V`token` and `use_auth_token` are both specified. Please set only the argument `token`. tokenizerFast) file_typefrom_auto_classrusing_pipelinez+Offline mode: forcing local_files_only=TrueTrzCalling z.from_pretrained() with the path to a single file or url is not supported for this tokenizer. Use a model identifier or the path to a directory instead.z.from_pretrained() with the path to a single file or url is deprecated and won't be possible anymore in v5. Use a model identifier or the path to a directory instead.r vocab_file)added_tokens_filespecial_tokens_map_filetokenizer_config_filetokenizer_filechat_template_filer) rrrrr0rrr user_agent%_raise_exceptions_for_missing_entriesrzCan't load tokenizer for 'z'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'zI' is the correct path to a directory containing all relevant files for a z tokenizer.utf-8r{fast_tokenizer_filesz*.jinja.jinja/chat_template_)rrrr0)r) rrrrrr0rrrrrz loading file z from cache at r0rrr _is_localr)+rrHrrrrr]r#rr,raospathisdirisfiler%r>rqr@rlADDED_TOKENS_FILESPECIAL_TOKENS_MAP_FILETOKENIZER_CONFIG_FILEFULL_TOKENIZER_FILErrOSErrorrropenjsonloadget_fast_tokenizer_filer ris_dirglobrD removesuffixr,rr_from_pretrained)#clsrrrrr0rrrr rrrr from_pipeliner commit_hashrr vocab_filesinit_configurationis_localsingle_file_idfile_idadditional_files_namesfast_tokenizer_fileresolved_config_filereadertokenizer_config template_dir template_file template_namerresolved_vocab_files file_paths# r8from_pretrainedz'PreTrainedTokenizerBase.from_pretrained)sv!**%6=**Y-$4d;JJ{D1  #3T:  **\59jj6 JJ{+  % MME   l#E#.?_eiliuiu_uv  $+8J' (  %5 KKE F# (+,I(J% 77==!>? 77>>7 8MJg>),4=(1"9-4@T[4\(14?5!"+'5 '(7)9##-!)"+>C%05(1422Fw2OQ\] I$ ^L#."3"3"5 g GY22 mI;78 mI;oFZ[bFcEdef g$s## )     -$/    s#$%89V8WX55R4STCCF<<.P[]xxj !45R4ST11N0OP??B||nKYs$P8(Q-Q:82Q*-Q7:2R,rc| jdd} | jd} |jdddu}| s|so|jc| sa|jjtj||tj|g| ||||dtj| }nd}|j dd}|yt |d5}tj|}dddjd }|j d d|s|j dd|j d d }| s|} nd}|}i}|j d d}|Dcgc]}|jd s|}}|)t |d5}|j|d<ddd|D]Q}|j |d}||jd }t |d5}|j||<dddSt|dk(r d|vr |d|d<n|r||d<|s'd|vr#t|dttfr d|di|d<|ddlm} |j%||||| |}|j&}|Tddlm}t3|dr |j4} nd} |D]}!|!t7|vs|!} n| |j| d\}}"||"}|U|j8j;dd|j;ddk7r&t<j?d|d|j8d|jA| |j dd}#|j dd}$|jCD]\}%}&|%|vs |&||%<|j dd}'|||d <||d!<i}(i})d"|vr}|d"jCD]e\}*}t|tDr tGd i|}t|tFr||(tI|*<||)t7|<NtKd#|jLd$n2|$t |$d5}+tj|+},|,jCD]\}}-|| vr| |rt|-tDrd%|-d&<tGd i|-}-nj|d'k(ret|-trU|j d'gxsg}.|-D]8}t|tDrd%|d&<tGd i|}||.vs(|.jO|:|.}-|-||< ddd|#g}/|jP|jSzD]J}|| |d'k(r |/||Dcgc] }t7|c}z }/.|/jOt7||Lt |#d5}0tj|0}1ddd1jCD].\}2}3|2|/v}4tG|2dd|4 |4(|(|3<|(|3|)t7<0|'vt |'d5}5tj|5}5|5j d)}6ddd6D]5}7|7j d*}*tGd i|7|(|*<|(|*|)t7|(|*<7|(|d"<|jU|d+}|jP|jSzD]8}|)ik7s |||d'k7s|)jt7||||||<: || i|}8|(ik7rHt]t|(jSd0d1|8j^kDrt<jYd2|8S#1swY*xYwcc}w#1swYxYw#1swYxYw#t(t*t,f$rd}YwxYw#1swYmxYwcc}w#1swYxYw#1swYtxYw#tW$rt<jYd,YytZ$r,}9d-t7|9vrt<jYd.Yd}9~9yd}9~9wt($r t)d/wxYw)3N from_slowFrr)r0rrrrrrtokenizer_classrrbrrrutf8rrauto_map AutoTokenizer) AutoConfig)r0rrrr)TOKENIZER_MAPPING_NAMES model_typeNNrr}zThe tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization. The tokenizer class you load from this checkpoint is 'z/'. The class this function is called from is 'z'.rr__slow_tokenizerr|rzFound a zV in the saved `added_tokens_decoder`, should be a dictionary or an AddedToken instanceTrRr r+r.id)savezUnable to load tokenizer model from SPM, loading from TikToken will be attempted instead.(Google protobuf error: Tried to load SPM model with non-SPM vocab file).zsentencepiece_processor.cczUnable to load tokenizer model from SPM, loading from TikToken will be attempted instead.(SentencePiece RuntimeError: Tried to load SPM model with non-SPM vocab file).zoUnable to load vocabulary from file. Please check that the provided vocabulary is accessible and not corrupted.rzuSpecial tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.)0rHslow_tokenizer_classrrrrrr r  startswithread removeprefixr>r?rAr@models.auto.configuration_autor)r"r%rrrmodels.auto.tokenization_autor*rr+rar]replacerrr_rrrHrsrrrDrrlconvert_added_tokensr9r, RuntimeErrormaxr):rr rrr0rrrrrrr r$rhas_tokenizer_fileslow_tokenizerrtokenizer_config_handlerconfig_tokenizer_classsaved_init_inputschat_templatesrrextra_chat_templateschat_template_handleextra_chat_templaterrr)configr*r+patternconfig_tokenizer_class_fastrr args_namer!rradded_tokens_mapidxspecial_tokens_map_handlerZrr r#added_tokens_handleadded_tok_encoder str_tokenindexrRtokenizer_file_handler.serialized_tokensrrs: r8rz(PreTrainedTokenizerBase._from_pretrained?sk "JJ{E2 JJ{+ 1556FMUYY /S5M5M5YbkHc66HH 23- 01   #!1) ==( N"N!5 8 89PRV W ,+g> ABY"ii(?@  A&1__5F%G " OO-t 4% 0$7 + r B / %) ",K1556JDQ/Chs~~VfGghh  )(7; H?S,@,E,E,Gy) H#7 L 0445H$OM$/<<=MNMmf5 L9M0D0I0I0K}- L L  L ~ ! # ^(C+9)+DK ( +9K ([(k*5t}E/> J@W.XK + ! ) C #331'%5&7!- 4*0)?)?&&-S6<0!'!2!2J"&J#:""c*G&HH)0J!" )JaJeJe"LKG*,G.51L. ! -||##FB/3I3Q3QRXZ\3]]66L5MN!!$b2 6"1445H$O"6":":;TVZ"[$8$>$>$@ 3 Iy +)2 I& 3.112BDI  %.&>&@1 U&=VC[%%eT2/3E),$.$7$7E $??JuVZD[8CHceg8h8nln5).L#-eT#:7;E)$4,6,?,?E#(0I#I$=$D$DU$K L %>E+0 C(#11.!,!#88;;K;K;MMIC"3'3"==*{SVGW.Xes5z.XXN*11#k#6F2GH I+g>GBU(, 2E(F%G(9(?(?(AO$Iu'>9G2<!%g+_f3(/4H3N$SZ0 O).7;M?T,0II6K,L)#8#<#<^#LLM*6a%+//5C0:0O=N0O(-G[\_G`$S)=c)B%CDa/C *+..{.G 00;3C3C3EE eC2%+c*:*F55'7';';C C@PZ2  F118/YGGMM*,-  KK\  +s1v5 e ]  s\ \-\\\+#\8!B+] ])]"-]'']45^ \\(+\5 8]]]']14]>"_,%_,-"__,c|SrNrb)rmax_model_lengthinit_max_model_lengths r8!_eventually_correct_t5_max_lengthz9PreTrainedTokenizerBase._eventually_correct_t5_max_lengthB s  r:rc t|tr(d|vr$|ddk(r|jdtdi|St|tr.|r,|j }|rd|d<|S|jd|St|t t fr"|Dcgc]}|j|||c}St|tr6|jDcic]\}}||j|||c}}S|Scc}wcc}}w)N__typerHrRr/add_type_fieldrb) r?rrrHrZr@rAr8r)rrr/rXorrs r8r8z,PreTrainedTokenizerBase.convert_added_tokensJ s c4 X_X,9V GGH $$ $ c: &4""$C ,H J "J dE] +cfg^_C,,QT.,Yg g T "ilirirituaeabdeAs//^/\\u u hus C7C<save_directoryrfilename_prefixsave_jinja_filesctjj||r|dzndtz}tjj||r|dzndtz}g}|rt |j trtt|dd5}|j|j dddtjd||j|d|vr|jd||fS|rIt |j tr.|j jD]\} } | d k(r_t|dd5}|j|j d dddtjd||j|jt!|j#d tjj|| d } t| dd5}|j| dddtjd| |j| d|vr|jd||fSt |j tr7|j jD cgc] \} } | | d  c} } |d<||fS|j |j |d<||fS#1swYxYw#1swY[xYw#1swYxYwcc} } w)a5 Writes chat templates out to the save directory if we're using the new format, and removes them from the tokenizer config if present. If we're using the legacy format, it doesn't write any files, and instead writes the templates to the tokenizer config in the correct format. -r}wrrNzchat template saved in rrTexist_okr)rDr)rrrrrr?rrarwriterr,rDrrrr mkdir)rTrZrr[r\rchat_template_dirsaved_raw_chat_template_filesfrrtemplate_filepathrrs r8save_chat_templatesz+PreTrainedTokenizerBase.save_chat_templates] s WW\\ o_s22Qcc GGLL o_s22Qbb )+%  4+=+=s C(#@ ,A**+ , KK12D1EF G ) 0 01C D"22 $$_54 !>>>3*T-?-?"F,0+=+=+C+C+E L' x I-0#H?A 2 29 =>?KK"9:L9M NO1889KL*+1141@(* 5F=/Y_H`(a%/wG*1)*KK"9:K9L MN1889JK L"22 $$_5 !>>>**D 1Y]XjXjXpXpXr0sPTPQST!1K0s _ - !>>>   +040B0B _ -!>>>? , ,??**1ts* J+J8K2K+J58K K legacy_format push_to_hub.c 0 |jdd}|;tjdt|j d t d||d<t jj|rtjd|dyt j|d |rr|jd d}|jd |jt jjd }|j|fi|}|j|} t jj!||r|d zndt"z} t jj!||r|d zndt$z} t'j(|j*} t-|j*j/} | j1ddg| D]}t3||st5||| |< | j1|j6d| vr*|j8| d<| j1|j8|j dd}|j;|| ||\} }t=|j>dkDr"t'j(|j>| d<|j@D]}| j|d|jC| dd} i}|jDjGD]\}}|jI||<|| d<|jJjL}|jOdrt5|ddr|dd}|| d<t5|dd|jP| d<t5|dd|jR| d<|jTtW||| d!| vr5| jd!| jd"d| jd#dd$| vr| jd$tY| d%d&'5}t[j\| d(dd)d*z}|j_|dddtjad+| |jC|jbdd,}tY| d%d&'5}t[j\|d(dd)d*z}|j_|dddtjad-| | | g|}|je||||.}|r%|jg| |j d/|S#1swYxYw#1swYrxYw)0a Save the full tokenizer state. This method make sure the full tokenizer can then be re-loaded using the [`~tokenization_utils_base.PreTrainedTokenizer.from_pretrained`] class method.. Warning,None This won't save modifications you may have applied to the tokenizer after the instantiation (for instance, modifying `tokenizer.do_lower_case` after creation). Args: save_directory (`str` or `os.PathLike`): The path to a directory where the tokenizer will be saved. legacy_format (`bool`, *optional*): Only applicable for a fast tokenizer. If unset (default), will save the tokenizer in the unified JSON format as well as in legacy format if it exists, i.e. with tokenizer specific vocabulary and a separate added_tokens files. If `False`, will only save the tokenizer in the unified JSON format. This format is incompatible with "slow" tokenizers (not powered by the *tokenizers* library), so the tokenizer will not be able to be loaded in the corresponding "slow" tokenizer. If `True`, will save the tokenizer in legacy format. If the "slow" tokenizer doesn't exits, a value error is raised. filename_prefix (`str`, *optional*): A prefix to add to the names of the files saved by the tokenizer. push_to_hub (`bool`, *optional*, defaults to `False`): Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the repository you want to push to with `repo_id` (will default to the name of `save_directory` in your namespace). kwargs (`dict[str, Any]`, *optional*): Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method. Returns: A tuple of `str`: The files saved. rNrr0rzProvided path (z#) should be a directory, not a fileTr`commit_messagerepo_idr0r^r}rrrr\rr)rXr/rrcan_save_slow_tokenizerFr@r% _auto_mapr'rr~)rDr|rr device_mapr_rrindent sort_keys ensure_ascii ztokenizer config file saved in rWzSpecial tokens file saved in )rZ file_namesrir[)rlr0)4rrrrrHrrrrrrPmakedirssplitsep _create_repo_get_files_timestampsrrrrrrrrlr_rrQrZrrhr>rrqr8rrrZrr]rGrorrsrrr dumpsrbr,r\_save_pretrained_upload_modified_files)rTrZrir[rjr rrlrmfiles_timestampsrrr target_keysrr\rerr.rrr%rfout_str write_dictrw save_filess r8save_pretrainedz'PreTrainedTokenizerBase.save_pretrained s1V $4d;  % MME zz'". l-F7O 77>>. ) LL?>*::]^ _  NT2 #ZZ(8$?NjjN,@,@,Mb,QRG'd'':6:G#99.I "$'',, o_s22Qhh# !# o_s22Qff!  ==)9)9:$**//12 .0NOP 7AtQ&-dA&6 # 7  7 78 !)9 97;7P7P 3 4  # #D$=$= >!::&8$?:>:R:R ,o?O; 77 t 1 $.2mmDWY^0_-cr2O.=*+ 4d + 7+/>> Z ( 4+T 2 >262G2G . /    ' t^?*[ K'+<*>>>s O Os&D>DADDct)aO Save only the vocabulary of the tokenizer (vocabulary + added tokens). This method won't save the configuration and special token mappings of the tokenizer. Use [`~PreTrainedTokenizerFast._save_pretrained`] to save the whole state of the tokenizer. Args: save_directory (`str`): The directory in which to save the vocabulary. filename_prefix (`str`, *optional*): An optional prefix to add to the named of the saved files. Returns: `tuple(str)`: Paths to the files saved. r7)rTrZr[s r8rz'PreTrainedTokenizerBase.save_vocabulary` s "!r:textrr1c t)a Converts a string into a sequence of tokens, replacing unknown tokens with the `unk_token`. Args: text (`str`): The sequence to be encoded. pair (`str`, *optional*): A second sequence to be encoded with the first. add_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to add the special tokens associated with the corresponding model. kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific encode method. See details in [`~PreTrainedTokenizerBase.__call__`] Returns: `list[str]`: The list of tokens. r7)rTrrr1r s r8rz PreTrainedTokenizerBase.tokenizer s $"!r:zI **kwargs: Passed along to the `.tokenize()` method. z Returns: `list[int]`, `torch.Tensor`, `tf.Tensor` or `np.ndarray`: The tokenized ids of the text. r text_pairstridec D|j|f|||||||| d| } | dS)aC Converts a string to a sequence of ids (integer), using the tokenizer and vocabulary. Same as doing `self.convert_tokens_to_ids(self.tokenize(text))`. Args: text (`str`, `list[str]` or `list[int]`): The first sequence to be encoded. This can be a string, a list of strings (tokenized string using the `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids` method). text_pair (`str`, `list[str]` or `list[int]`, *optional*): Optional second sequence to be encoded. This can be a string, a list of strings (tokenized string using the `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids` method). )rr1rrrrryrrt) encode_plus) rTrrr1rrrrryrr encoded_inputss r8encodezPreTrainedTokenizerBase.encode sNL*))   1!!%)    k**r:ctrNr7)rTrs r8rz1PreTrainedTokenizerBase.num_special_tokens_to_add r9r:c f|J|durF|D|r@|jjddstjdd|jd<d}|durh|dur5|r"| | |dus|dk(rt j dt j}n?t|t s t |}n#t|t r|}nt j}|durF|D|durtj}n?t|ts t|}n#t|tr|}ntj}| t jk(rr|jtkDrS|r@|jjddstjd d|jd<t j}n |j}tjk7rr|jtkDrS|r@|jjd dstjd d|jd <tj}n |j}t jk7r&|j |j"d kr t%d tjk7r1|t jk7r||||zd k7rt%d|d|d||||fS)z> Find the correct padding/truncation strategy Fz#Truncation-not-explicitly-activatedatTruncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.Trhriz`max_length` is ignored when `padding`=`True` and there is no truncation strategy. To pad to max length, use `padding='max_length'`.zAsking-to-pad-to-max_lengthzAsking to pad to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no padding.z Asking-to-truncate-to-max_lengthzAsking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.rzAsking to pad but the tokenizer does not have a padding token. Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`.zATruncation and padding are both activated but truncation length (z+) is not a multiple of pad_to_multiple_of (z).)rrHrrrrrLONGESTr? DO_NOT_PADrerlrm MAX_LENGTHr LARGE_INTEGERr  pad_token_idr) rTrrrpad_to_multiple_ofrr padding_strategytruncation_strategys r8"_get_padding_truncation_strategiesz:PreTrainedTokenizerBase._get_padding_truncation_strategies s  !g&6:;M00445Z\abNN)TX))*OP(J % $!-"*jE.AZSdEd P$3#:#: 9#27#; G_5#* .99  U "z'=T!&44$ ,>?&8&D#J(:;&0#"4"D"D   ?#=#==((=8#88<<=Z\ab"NN!UTX112OP'6'A'A$!%!6!6J"&8&H&HH((=8#88<<=_afg"NN![Y]112TU*<*L*L'!%!6!6J 99 9t~~?UY]YjYjmnYne  #5#E#E E O$>$>>".&00A5&&0\1\]o\pprt   !4j&HHr: text_targettext_pair_targetis_split_into_wordsrreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthrc id|d|d|d|d| d| d| d| d | d |d |d |d |d|d|d|jd|jd|}| dvrtjd|j || | t d|1|j s|j|jd||d|}|%|j|jd||d|}|j|S|Sdd<|S)a  Main method to tokenize and prepare for the model one or several sequence(s) or one or several pair(s) of sequences. Args: text (`str`, `list[str]`, `list[list[str]]`, *optional*): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). text_pair (`str`, `list[str]`, `list[list[str]]`, *optional*): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). text_target (`str`, `list[str]`, `list[list[str]]`, *optional*): The sequence or batch of sequences to be encoded as target texts. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). text_pair_target (`str`, `list[str]`, `list[list[str]]`, *optional*): The sequence or batch of sequences to be encoded as target texts. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). r1rrrrrrryrrrrrrrrr)rjaxzTensorFlow and JAX classes are deprecated and will be removed in Transformers v5. We recommend migrating to PyTorch classes or pinning your version of Transformers.z3You need to specify either `text` or `text_target`.)rrrtlabelsrb) rrrr$r_rr_switch_to_input_mode _call_one_switch_to_target_mode)rTrrrrr1rrrrrrryrrrrrrrrr  all_kwargsrtarget_encodingss r8__call__z PreTrainedTokenizerBase.__call__' sd "4 w  *  *  f  "#6   !"4  L  n  $%:  $%:  ()B  )*D  %&<  ] #FJJ/EtG`G`$a! " w# ( ] *   b  &! )rs r8_is_valid_text_inputz?PreTrainedTokenizerBase._call_one.._is_valid_text_input sz!S!Ae}-q6Q;!c*!tUm4qt9>EZ!Q-EE r:ztext input must be of type `str` (single example), `list[str]` (batch or single pretokenized example) or `list[list[str]]` (batch of pretokenized examples).rzdwhen tokenizing batches of text, `text_pair` must be a list or tuple with the same length as `text`.zbatch length of `text`: z- does not match batch length of `text_pair`: rbatch_text_or_text_pairsr1rrrrrrryrrrrrrrrrrrrb) rr?r@rArarr>zipbatch_encode_plusr)rTrrr1rrrrrrryrrrrrrrrrr rrrs r8rz!PreTrainedTokenizerBase._call_one s0 ($D)I   )=i)HI  #D4-8hThjQUVWQX[_afZgFhJ#D4-8J )S)$Tc)n)D .s4yk:I'q*FOEZtCi,@'A`d $)4)))A#5 &  &   %8$6* .'<'<+D,F(> ,!" #$&:' ,$4###$6  &  & %8$6* .'<'<+D,F (>!",#$ %&&:) r:c |jd|||| |d|\}}}}|jdid|d|d|d|d|d|d|d |d | d | d | d | d| d|d|d|d|d|d|jd|j|S)ax Tokenize and prepare for the model a sequence or a pair of sequences. This method is deprecated, `__call__` should be used instead. Args: text (`str`, `list[str]` or (for non-fast tokenizers) `list[int]`): The first sequence to be encoded. This can be a string, a list of strings (tokenized string using the `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids` method). text_pair (`str`, `list[str]` or `list[int]`, *optional*): Optional second sequence to be encoded. This can be a string, a list of strings (tokenized string using the `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids` method). rrrrrrrr1rrrrrrryrrrrrrrrrrb)r _encode_plusrr)rTrrr1rrrrrrryrrrrrrrrr rrs r8rz#PreTrainedTokenizerBase.encode_plus s-VElDDkDkE !!1 E  E A-z6!t      2 .  !4  "   !4  2 & * #8 #8 '@ (B $:! "(# $% &"(,BDD]D]!^)  r:rrc trNr7)rTrrr1rrrrrrryrrrrrrrrrr s r8rz$PreTrainedTokenizerBase._encode_plusJ s ."!r:rc |jd|||||d|\}}}}|jdid|d|d|d|d|d|d|d |d | d | d | d | d| d|d|d|d|d||S)a Tokenize and prepare for the model a list of sequences or a list of pairs of sequences. This method is deprecated, `__call__` should be used instead. Args: batch_text_or_text_pairs (`list[str]`, `list[tuple[str, str]]`, `list[list[str]]`, `list[tuple[list[str], list[str]]]`, and for not-fast tokenizers, also `list[list[int]]`, `list[tuple[list[int], list[int]]]`): Batch of sequences or pair of sequences to be encoded. This can be a list of string/string-sequences/int-sequences or a list of pair of string/string-sequences/int-sequence (see details in `encode_plus`). rrr1rrrrrrryrrrrrrrrrrb)r_batch_encode_plus)rTrr1rrrrrrryrrrrrrrrrr rrs r8rz)PreTrainedTokenizerBase.batch_encode_plusc s^ElDDkDkE !!1 E  E A-z6't&& %= 1 . !4  "    !4  2 & * #8 #8 '@ (B $: (! "# $"6'  r:c trNr7)rTrr1rrrrrrryrrrrrrrrrr s r8rz*PreTrainedTokenizerBase._batch_encode_plus s :"!r:rc 8|jjjdrX|jj dds If the `encoded_inputs` passed are dictionary of numpy arrays, PyTorch tensors or TensorFlow tensors, the result will use the same type unless you provide a different tensor type with `return_tensors`. In the case of PyTorch tensors, you will lose the specific device of your tensors however. Args: encoded_inputs ([`BatchEncoding`], list of [`BatchEncoding`], `dict[str, list[int]]`, `dict[str, list[list[int]]` or `list[dict[str, list[int]]]`): Tokenized inputs. Can represent one input ([`BatchEncoding`] or `dict[str, list[int]]`) or a batch of tokenized inputs (list of [`BatchEncoding`], *dict[str, list[list[int]]]* or *list[dict[str, list[int]]]*) so you can use this method during preprocessing as well as in a PyTorch Dataloader collate function. Instead of `list[int]` you can have tensors (numpy arrays, PyTorch tensors or TensorFlow tensors), see the note above for the return type. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`): Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: - `True` or `'longest'` (default): Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'`: No padding (i.e., can output a batch with sequences of different lengths). max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). padding_side (`str`, *optional*): The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. return_attention_mask (`bool`, *optional*): Whether to return the attention mask. If left to the default, will return the attention mask according to the specific tokenizer's default, defined by the `return_outputs` attribute. [What are attention masks?](../glossary#attention-mask) return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return Numpy `np.ndarray` objects. verbose (`bool`, *optional*, defaults to `True`): Whether or not to print more information and warnings. rzAsking-to-pad-a-fast-tokenizerFzYou're using a z tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.TrzRYou should supply an encoding or a list of encodings to this method that includes z, but you provided rvrptrztype of z unknown: zA. Should be one of a python, numpy, pytorch or tensorflow object.)rrr)rrrryrrc3:K|]}t|k(ywrNr>)rr batch_sizes r8rz.PreTrainedTokenizerBase.pad..h sIA3q6Z'IszLSome items in the output dictionary have a different batch size than others.c32K|]}t|ywrNr)rinputss r8rz.PreTrainedTokenizerBase.pad..m sFVS[Fs)#rr]rGrrHrwarning_advicer?r@rArrwrrlrr>rsr'r+rrrrr/r_padryrr^rrr:rrrD)rTrrrrryrrrrexamplerequired_input first_elementrrr_ batch_outputsrrrroutputsrs @r8padzPreTrainedTokenizerBase.pad s\ >> " " + +F 3,,001QSXY%%%dnn&=&=%>?EE OS))*JK ntUm 4NSTDUW^9_[ijk[lmTWc#OWGCL#OOmNm  ! !! $N :!!%!7!7!: ;;NtTbTgTgTiOjNkm  ((>(>q(AB  !j&GCP^L_cdLd$35/0! ! 'q) mdE] 3& t9>$(GM  -#tU);<M*)7)?^ /)7)?^M2::6)7)?^ }oZ]8K7LMVV -224 7 U&/&6s# 7.2-T-T G.U. *!Z((>(>q(AB *^A->u "N!YY%!1#5)&; 'N!^L L( I1F1F1HII Z I 66 6F~FFJ.99  z" 1A*8*>*>*@A$!Qa1gAFAii%!1#5)&;  G&mmo 1 Um+)+M#&c"))%0 1 1 ]GGo$PmPBs0 N: N NN N token_ids_0 token_ids_1ctt|dddu}tt|dddu}|dg|t|z|zzSdg|t|z|zzdgt||zzzS)a Create the token type IDs corresponding to the sequences passed. [What are token type IDs?](../glossary#token-type-ids) Should be overridden in a subclass if the model has a special way of building those. Args: token_ids_0 (`list[int]`): The first tokenized sequence. token_ids_1 (`list[int]`, *optional*): The second tokenized sequence. Returns: `list[int]`: The token type ids. cls_token_idN sep_token_idrr)rsrQr>)rTrrcls_lensep_lens r8$create_token_type_ids_from_sequencesz? ?sgK 007:;qcSEUX_E_>```r:c||S||zS)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. This implementation does not add special tokens and this method should be overridden in a subclass. Args: token_ids_0 (`list[int]`): The first tokenized sequence. token_ids_1 (`list[int]`, *optional*): The second tokenized sequence. Returns: `list[int]`: The model input with special tokens. rb)rTrrs r8 build_inputs_with_special_tokensz8PreTrainedTokenizerBase.build_inputs_with_special_tokens s   [((r:idspair_idsr}c |jd|||||d|\}}}}|du}t|}|r t|nd}| r |s td| r |tjk(r | td| d|j v} | d|j v} i}||z|r|j |ndz}g}|tjk7r$|r"||kDr|j||||z || \}}}| r ||d <||z |d <|r%|j||}|j||}n,|r||zn|}dgt|z|rdgt|zngz}||d <| r||d<|r*|r|j|||d <ndgt|z|d <|j|d |||tjk7s| r!|j|||j || | }|rt|d |d<t#|| |}|S)a Prepares a sequence of input id, or a pair of sequences of inputs ids so that it can be used by the model. It adds special tokens, truncates sequences if overflowing while taking into account the special tokens and manages a moving window (with user defined stride) for overflowing tokens. Please Note, for *pair_ids* different than `None` and *truncation_strategy = longest_first* or `True`, it is not possible to return overflowing tokens. Such a combination of arguments will raise an error. Args: ids (`list[int]`): Tokenized input ids of the first sequence. Can be obtained from a string by chaining the `tokenize` and `convert_tokens_to_ids` methods. pair_ids (`list[int]`, *optional*): Tokenized input ids of the second sequence. Can be obtained from a string by chaining the `tokenize` and `convert_tokens_to_ids` methods. rNrzAsking to return token_type_ids while setting add_special_tokens to False results in an undefined behavior. Please set add_special_tokens to True or set return_token_type_ids to None.zNot possible to return overflowing tokens for pair of sequences with the `longest_first`. Please select another truncation strategy than `longest_first`, for instance `only_second` or `only_first`.rurvr)rnum_tokens_to_removerrrnum_truncated_tokensrtspecial_tokens_mask)rrrryrlengthrrb)rr>rrerlrwrrmtruncate_sequencesrrget_special_tokens_mask&_eventual_warn_about_too_long_sequencerrrrry)rTrrr1rrrrrryrrrrrrrrr}r rrrlen_ids len_pair_idsr total_lenrsequencerurs r8prepare_for_modelz)PreTrainedTokenizerBase.prepare_for_model sPElDDkDkE !!1 E  E A-z6t#c((,s8}! );5  &#'9'G'GG$>  ! ($48N8N$N ! ($48N8N$N !l*[md.L.LRV.L.Wstu   "4"D"D DXadnXn040G0G!%.%;$7 1H1 -C- %3EN/ 05>5KN1 2 <++F'GN8 $% K] r:rc |dkr||gfSt|ts t|}g}|tjk(s|tjk(r|t ||kDrit t |||z}|j dk(r |d|}||d}n/|j dk(r|| d}|d| }ntd|j dd|dt |d }|tjk(r |d |d z}tj|n|tjk(rtjd tjjd | t |nd} t |} t t| | z |} || z } | | kDr| | dzz} | | dzz }n| dz} | | z| dzz }|j dk(r| dkDr|d| n|}| |dkDr|d| n|}n|j dk(r|| d}|||dnd}ntd|j |tjk(r|t ||kDrft t |||z}|j dk(r || d}|d| }nZ|j dk(r |d|}||d}n@td|j tjd|dt |d|d|||fS)a Truncates a sequence pair in-place following the strategy. Args: ids (`list[int]`): Tokenized input ids of the first sequence. Can be obtained from a string by chaining the `tokenize` and `convert_tokens_to_ids` methods. pair_ids (`list[int]`, *optional*): Tokenized input ids of the second sequence. Can be obtained from a string by chaining the `tokenize` and `convert_tokens_to_ids` methods. num_tokens_to_remove (`int`, *optional*, defaults to 0): Number of tokens to remove using the truncation strategy. truncation_strategy (`str` or [`~tokenization_utils_base.TruncationStrategy`], *optional*, defaults to `'longest_first'`): The strategy to follow for truncation. Can be: - `'longest_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will truncate token by token, removing a token from the longest sequence in the pair if a pair of sequences (or a batch of pairs) is provided. - `'only_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the first sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `'only_second'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the second sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `'do_not_truncate'` (default): No truncation (i.e., can output batch with sequence lengths greater than the model maximum admissible input size). stride (`int`, *optional*, defaults to 0): If set to a positive number, the overflowing tokens returned will contain some tokens from the main sequence returned. The value of this argument defines the number of additional tokens. Returns: `tuple[list[int], list[int], list[int]]`: The truncated `ids`, the truncated `pair_ids` and the list of overflowing tokens. Note: The *longest_first* strategy returns empty list of overflowing tokens if a pair of sequences (or a batch of pairs) is provided. rNrrxzinvalid truncation strategy: z, use 'left' or 'right'.zWe need to remove z; to truncate the input but the first sequence has a length z. z/Please select another truncation strategy than z0, for instance 'longest_first' or 'only_second'.zmBe aware, overflowing tokens are not returned for the setting you have chosen, i.e. sequence pairs with the 'zg' truncation strategy. So the returned list will always be empty even if some tokens have been removed.rqzinvalid truncation strategy:z< to truncate the input but the second sequence has a length z1. Please select another truncation strategy than z/, for instance 'longest_first' or 'only_first'.)r?rerjrlr>rrzrrrPrrabsrk)rTrrrrrr window_len error_msgrr first_remove second_remove ids_to_movepair_ids_to_moves r8rz*PreTrainedTokenizerBase.truncate_sequences9sX 1 $"$ $-/AB"45H"I  "4"?"? ? #5#C#C CHX3x.. S64H+HI ''61),[j)9&234C))W4),j[\):&4 445C$'DTEYEYDZZr%stt))=(>?;;>s8*BH'*<*G*GG!$U.//_%aa Y' $6$D$D D NN22D2R2R2X2X1YZ,,  -5,@3x=aL#hGs<'#9:*>)>?H))V3)1+:)>&'(<(=>H$'CDDXDXCY%Z[[ ()=(>?<= 7.5` (Volta). padding_side: The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. return_attention_mask: (optional) Set to False to avoid returning attention mask (default: set to model specifics) rvrrrxrurrzInvalid padding strategy:) rwrrr>rryr;rr) rTrrrrryrrneeds_to_be_padded differences r8rzPreTrainedTokenizerBase._padskH ! ($48N8N$N !'(>(>q(AB 66 6^,J  !&8&D*WiJimnJn%);;q@DVVJ-1K1KKqPSTbPcgqPq !%5^%K01sS5H/HN+ , #c.&99J+7+C<IZIZLw&(7EFV7W[\Z]`jZj7jN#34#~5&'78DC*!  DKK $7-I      s% token_idsc Dt|}|jd|||d|S)a Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokenization spaces. Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`. Args: token_ids (`Union[int, list[int], np.ndarray, torch.Tensor, tf.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces`. kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `str`: The decoded sentence. )rrrrb)r/_decoderTrrrr s r8rzPreTrainedTokenizerBase.decode6s98i( t||  3)E    r:c trNr7rs r8rzPreTrainedTokenizerBase._decode[s "!r:already_has_special_tokensch|r|Jd|j}|Dcgc] }||vrdnd }}|Scc}w)a Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` or `encode_plus` methods. Args: token_ids_0 (`list[int]`): List of ids of the first sequence. token_ids_1 (`list[int]`, *optional*): List of ids of the second sequence. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. a You cannot use ``already_has_special_tokens=False`` with this tokenizer. Please use a slow (full python) tokenizer to activate this argument. Or set `return_special_tokens_mask=True` when calling the encoding method to get the special tokens mask in any tokenizer. rr)rj)rTrrrrjr0rs r8rz/PreTrainedTokenizerBase.get_special_tokens_maskdsX$*k.A  @ A..Q\]E_$ zI). Running this sequence through the model will result in indexing errorsT)r>rrrHrr)rTrrrs r8rz>PreTrainedTokenizerBase._eventual_warn_about_too_long_sequences  #c(T-B-B"BwSWShShlmSm,,001gino''*3xjD4I4I3JK55 aeD % %&\ ]Tnw"B r:cy)zs Private method to put the tokenizer in input mode (when it has different modes for input/outputs) NrbrYs r8rz-PreTrainedTokenizerBase._switch_to_input_mode r:cy)zt Private method to put the tokenizer in target mode (when it has different modes for input/outputs) NrbrYs r8rz.PreTrainedTokenizerBase._switch_to_target_moderr:c#Ktjd|jd|_dd|_|j yw)z Temporarily sets the tokenizer for encoding the targets. Useful for tokenizer associated to sequence-to-sequence models that need a slightly different processing for the labels. a`as_target_tokenizer` is deprecated and will be removed in v5 of Transformers. You can tokenize your labels by using the argument `text_target` of the regular `__call__` method (either in the same call as your input texts if you use the same keyword arguments, or in a separate call.TNF)rrrrrrYs r8as_target_tokenizerz+PreTrainedTokenizerBase.as_target_tokenizersG   ] ##%*.' */' ""$sA A ct|ts |j}ddlmcm}t ||st|d||_y)ac Register this class with a given auto class. This should only be used for custom tokenizers as the ones in the library are already mapped with `AutoTokenizer`. Args: auto_class (`str` or `type`, *optional*, defaults to `"AutoTokenizer"`): The auto class to register this new tokenizer with. rNz is not a valid auto class.) r?rar]transformers.models.automodelsautorrrs)r auto_class auto_modules r8register_for_auto_classz/PreTrainedTokenizerBase.register_for_auto_classsC*c*#,,J66{J/ |+FGH H$r: src_texts tgt_textsmax_target_lengthc Hd} tj| t|jdd|jdd| |j}||fd||||d|} || S||}|j 5||fd||||d|} ddd d| d <| S#1swYxYw) a Prepare model inputs for translation. For best performance, translate one sentence at a time. Arguments: src_texts (`list[str]`): List of documents to summarize or source language texts. tgt_texts (`list`, *optional*): List of summaries or target language texts. max_length (`int`, *optional*): Controls the maximum length for encoder inputs (documents to summarize or source language texts) If left unset or set to `None`, this will use the predefined model maximum length if a maximum length is required by one of the truncation/padding parameters. If the model has no specific maximum input length (like XLNet) truncation/padding to a maximum length will be deactivated. max_target_length (`int`, *optional*): Controls the maximum length of decoder inputs (target language texts or summaries) If left unset or set to `None`, this will use the max_length value. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `False`): Activates and controls padding. Accepts the following values: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different lengths). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return Numpy `np.ndarray` objects. truncation (`bool`, `str` or [`~tokenization_utils_base.TruncationStrategy`], *optional*, defaults to `True`): Activates and controls truncation. Accepts the following values: - `True` or `'longest_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will truncate token by token, removing a token from the longest sequence in the pair if a pair of sequences (or a batch of pairs) is provided. - `'only_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the first sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `'only_second'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the second sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `False` or `'do_not_truncate'` (default): No truncation (i.e., can output batch with sequence lengths greater than the model maximum admissible input size). **kwargs: Additional keyword arguments passed along to `self.__call__`. Return: [`BatchEncoding`]: A [`BatchEncoding`] with the following fields: - **input_ids** -- List of token ids to be fed to the encoder. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model. - **labels** -- List of token ids for tgt_texts. The full set of keys `[input_ids, attention_mask, labels]`, will only be returned if tgt_texts is passed. Otherwise, input_ids, attention_mask will be the only keys. a `prepare_seq2seq_batch` is deprecated and will be removed in version 5 of HuggingFace Transformers. Use the regular `__call__` method to prepare your inputs and targets. Here is a short example: model_inputs = tokenizer(src_texts, text_target=tgt_texts, ...) If you either need to use different keyword arguments for the source and target texts, you should do two calls like this: model_inputs = tokenizer(src_texts, ...) labels = tokenizer(text_target=tgt_texts, ...) model_inputs["labels"] = labels["input_ids"] See the documentation of your specific tokenizer for more details on the specific arguments to the tokenizer of choice. For a more complete example, see the implementation of `prepare_seq2seq_batch`. src_langNtgt_langT)r1rrrr)r1rrrrrtr)rrrrrr ) rTrrrrrrrr formatted_warning model_inputsrs r8prepare_seq2seq_batchz-PreTrainedTokenizerBase.prepare_seq2seq_batchsN$  '7 :t$ :t$  ..J  #)!!        $ *   % % ' #'-,% F "( !4 X  s 6BB!) NNNFFTFFNNFFNrNr,)FT)NNFr)NTFNNrNNrn)FNNNT)NNNNTFNNrFNNNNNFFFFT)NTFNNrFNNNNNFFFFTF)NTFNNrFNNNNNFFFFT)TFNNrFNNNNNFFFFTF)TNNNNNT)NTFNNrNNNNNFFFFTF)Nrrhr)FN)r()NNNlongestNT)_r]r^r_r`rqrrartrrrsrrwr@ryrzr1rUrrsrrsetterrrHrrrrrr rrrr ryrrr classmethodrPathLiker"r staticmethodrTr8rhrArr~rrrENCODE_KWARGS_DOCSTRING TextInputPreTokenizedInput EncodedInputrerrr'ENCODE_PLUS_ADDITIONAL_KWARGS_DOCSTRINGrrrrrmr TextInputPairPreTokenizedInputPairEncodedInputPairrrrrrrrrrrrrrrrrrrrrr rrrrs@r8rprpzsA )+tCH~*<>S$sCx.%8 9>!%K#%$UtCyTL#"OS"2ZhRRR QQQ ##  $ "" xs x# x0C0$d3 ?&;$$ # $$ $4S> $8<48'+&+',5: $(;?!-259o!Dc3h0$tDcN7K2LLMo!U4>234o!Dc3h01 o!  } o! $ o!!%o!o!tS/12o!o!SMo!!sJ!78o!o!'+o!#4S>2o!" sDItCy$tCy/=H I#o!h@D6 c3h6 'tDcN';<6 c 6 p4x}4HUYZ^U_L`4lo4l 8<$!&,0S ',S"++-='>S E#r{{"234 S  S  S c4i()S S S j@@D  uZ_'=$5?c2;;./5?5?"# 5?  5?t)-)-! gc2;;./g ~g"# g  g sCxgZ)-)- !?c2;;./!?#s(O!? ~ !? "# !? sCx !?F"c"HSM"]bcfhkck]l"$"S" "RV"mqrumv"(  RV#'5:AE$(&*;?)+I0,>?)+E)-> "LMN)+! )+ tS/12 )+ $%7=> )+SM)+)+sm)+!sJ!78)+ c)+ )+V"d"s"aegIR/1XYeimqko #'5:AE$($),0&*;?0404*/+0',#/_I0$y/4HYCZ\``a_E)->YQUVgQh"hij_9&7i$O`Jacggh _ # ).YFWAXX Y _!_tS/12_$%7=>_SM__"_%SM_sm_ !sJ!78!_" (~#_$ (~%_&$('_(%))_*!%+_,-_./_2 3_Z_Hnr#'5:AE$($),0&*;?0404*/+0',#%*)tI0$y/4HYCZZ[tE)->YQUVgQh"hijt! t tS/12 t $%7=> tSMtt"t%SMtsmt!sJ!78t (~t (~t$(t %)!t"!%#t$%t&'t(#)t, -tl/1XYRV#'5:AE$($),0&*;?0404*/+0',#'H I0,>?H E)-> "LMNH ! H tS/12 H $%7=> H SMH H "H %SMH smH !sJ!78H  (~H  (~H $(H %)!H "!%#H $%H &'H * +H ZH ZRV#',;,F,F2D2T2T$($),0&*;?0404*/+0',#%*)"I0,>?"E)-> "LMN"! " * " 0 "SM""""%SM"sm"!sJ!78" (~" (~"$(" %)!""!%#"$%"&'"(#)", -"2/1XY$(5:AE$($),0&*;?0404*/+0',#%*5K "' O   " # & '   ! "  $# K !K tS/12K $%7=>K SMK K "K %SM!K "sm#K $!sJ!78%K & (~'K ( (~)K *$(+K ,%)-K .!%/K 01K 23K 4#5K 8 9K ZK n$(,;,F,F2D2T2T$($),0&*;?0404*/+0',#%*5""' O   " # & '   ! "  $# "!"*"0"SM"""" %SM!""sm#"$!sJ!78%"& (~'"( (~)"*$(+",%)-".!%/"01"23"4#5"8 9"P6:$(,0&*04;?qH    l" # d<(( ) c<'( )  + qHtS/12qHSMqH%SMqHsmqH (~qH!sJ!78qHqH !qHhJNa9a3;DI3Fa ca2JN)9)3;DI3F) c)(/1XY)-#'5:AE$(,0&*;?0404*/+0',##('G #YG49%G! G tS/12 G $%7=> GSMGG%SMGsmG!sJ!78G (~G (~G$(G%)G !%!G"#G$%G&!'G* +GZGX)-$%>M z3 #Yz349%z3" z3 #3(:#:; z3  z3 tCy$s)T#Y. /z3~%),;,F,F,0&*04Pd3 #45}DEPSMP* P %SM P sm P (~P Pd "tCy "S " %*7;  cDcO\>S^^_  "  '/tn  c  J%*7; # d3i^CD# "# '/tn # # P%*7; "d3i("""'/tn " "sx#9#3;DI3F#ko# c#>##0e$s)eQYZ]Q^eime(  %% %%0*.$(+/ (,x9xDI&xSM x $C= x  x! xx xr:rptokenization_filesrc6i}|D]2}tj|}||jd}|||<4t|j }t }t jt}|D]"}t j||kr||}!|S|S)z Get the tokenization file to use for this version of transformers. Args: tokenization_files (`list[str]`): The list of available configuration files. Returns: `str`: The tokenization file to use. r) _re_tokenizer_filesearchgroupsrrlrrparser)r)tokenizer_files_map file_namer,ravailable_versionsrtransformers_versions r8r r ^s'/ #**95   "A%.  " /  3 8 8 :;)N"==5  == 3 303N   r:rr(ztokenizer files)r object_class object_files)r})pr`rr rrer collectionsrcollections.abcrrr contextlibr dataclassesrpathlibr typingr r r r rrrr packagingrr}rdynamic_module_utilsrutilsrrrrrrrrrrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/utils.chat_template_utilsr0utils.import_utilsr1rrrrrr9r@rC tokenizersrGrrH get_loggerr]rrsrrrar"r#r$rAr&r'r(r AudioInputrrrrcompiler+rerorwryrr!r%INIT_TOKENIZER_DOCSTRINGrpr rjr6rbr:r8rFs   44%!LL4<=5G 3%e%  & 4   H %YD   ICy c3h d3ic23cDI-.2::~tBJJ/?nAUU V 4'/'RZZ 89 ( (  z     D HD Nprprf :x4+'nCL,-`+0.`+.`+FWS c@'00G0S0S&T#&&..:2I2U2U2]2]2d2dGX3e3''/;r: