L iLddlmZmZddlmZddlmZejeZ GddeZ Gdde Z Gd d Z Gd d e Z Gd dZy))ABCabstractmethod)Optional)loggingceZdZdZdZdZedZedefdZ edefdZ edZ ed Z ed d Z y ) ConstraintayAbstract base class for all constraints that can be applied during generation. It must define how the constraint can be satisfied. All classes that inherit Constraint must follow the requirement that ```py completed = False while not completed: _, completed = constraint.update(constraint.advance()) ``` will always terminate (halt). cNtjd|jy)Na Importing `Constraint` classes is deprecated and will be removed in v4.58.0. Constrained beam search has been moved to the Hub: https://hf.co/transformers-community/constrained-beam-search. Please import using `from transformers.generation import Constraint` instead.)logger warning_oncetestselfs n/mnt/ssd/data/python-lab/Trading/venv/lib/python3.12/site-packages/transformers/generation/beam_constraints.py__init__zConstraint.__init__s! Z  c(d}d}|sn|dk(r|j|j}|j|s td|j |\}}}|dz }|dkDr td|sn|j dk7r tdy) zJ Tests whether this constraint has been properly defined. rFz[Custom Constraint is not defined correctly. self.does_advance(self.advance()) must be true.i'z)update() does not fulfill the constraint.z+Custom Constraint is not defined correctly.N)resetadvance does_advance Exceptionupdate remaining)rcounter completedrsteppedrs rr zConstraint.test!s !| llnG$$W-q)- G(< %GY qLG KLL >> q IJ J !rc2t|jd)a When called, returns the token(s) that would take this constraint one step closer to being fulfilled. Return: token_ids (Union[int, list[int], None]): - A single token ID (int) that advances the constraint, or - A list of token IDs that could advance the constraint - None if the constraint is completed or cannot be advanced H is an abstract class. Only classes inheriting this class can be called.NotImplementedError __class__rs rrzConstraint.advance9!"~~f g  rtoken_idc2t|jd)zK Reads in a token and returns whether it creates progress. rr rr$s rrzConstraint.does_advanceH! "~~f g  rc2t|jd)a Reads in a token and returns booleans that indicate the progress made by it. This function will update the state of this object unlikes `does_advance(self, token_id: int)`. This isn't to test whether a certain token will advance the progress; it's to update its state as if it has been generated. This becomes important if token_id != desired token (refer to else statement in PhrasalConstraint) Args: token_id(`int`): The id of a newly generated token in the beam search. Return: stepped(`bool`): Whether this constraint has become one step closer to being fulfuilled. completed(`bool`): Whether this constraint has been completely fulfilled by this token being generated. reset (`bool`): Whether this constraint has reset its progress by this token being generated. rr r&s rrzConstraint.updateQs!*"~~f g  rc2t|jd)z Resets the state of this constraint to its initialization. We would call this in cases where the fulfillment of a constraint is abrupted by an unwanted token. rr rs rrzConstraint.resetjs! "~~f g  rc2t|jd)zl Returns the number of remaining steps of `advance()` in order to complete this constraint. rr rs rrzConstraint.remainingtr'rc2t|jd)a  Creates a new instance of this constraint. Args: stateful(`bool`): Whether to not only copy the constraint for new instance, but also its state. Return: constraint(`Constraint`): The same constraint as the one being called from. rr )rstatefuls rcopyzConstraint.copy}r#rNF)__name__ __module__ __qualname____doc__rr rrintrrrrr-rrr r s K0     S   s  0        rr c`eZdZdZdeeffd ZdZdefdZdefdZ dZ d Z d d Z xZ S) PhrasalConstraintz [`Constraint`] enforcing that an ordered sequence of tokens is included in the output. Args: token_ids (`list[int]`): The id of the token that must be generated by the output. token_idsc$tt| t|trt |dk(rt d|dtd|Drt d|d||_t |j|_ d|_ d|_ y)Nrz/`token_ids` has to be a non-empty list, but is .c3LK|]}t|t xs|dkywrN isinstancer3.0r$s r z-PhrasalConstraint.__init__..s&Y8Jx--=A=Y"$zGEach list in `token_ids` has to be a list of positive integers, but is F) superr rr=listlen ValueErroranyr7seqlen fulfilled_idxr)rr7r"s rrzPhrasalConstraint.__init__s j$(*)T*c)n.ANykYZ[\ \ YyY Yfgpfqqrst t"$..) rcT|jry|j|jdzSNr)rr7rIrs rrzPhrasalConstraint.advances& >>~~d001455rr$ct|tstd|dt||jry||j |j dzk(S)N&`token_id` has to be an `int`, but is of type Fr)r=r3 TypeErrortyperr7rIr&s rrzPhrasalConstraint.does_advancesV(C(DXJiX\]eXfWghi i >>4>>$*<*>: %)[[N "+/+=+=N ('+~~N $rr.r/r0r1r2rDr3rrrrrrr- __classcell__r"s@rr6r6sF $s) 6 BSB)s)(6rr6c>eZdZddeeefdZdZdZdZdZ y) DisjunctiveTrienested_token_idsct|Dcgc] }t|c}|_i}|D]%}|}t|D]\}}||vri||<||}'|r!|j ||rt d|d||_ycc}w)ze A helper class that builds a trie with the words represented in `nested_token_ids`. zSEach list in `nested_token_ids` can't be a complete subset of another list, but is r9N)maxrE max_height enumerate has_subsetsrFtrie) rr_ no_subsetsonerootr7leveltidxr$s rrzDisjunctiveTrie.__init__s3CDCs3xDE) (IE"+I"6 (h5(&(E(Oh  ( ( $**41AB$%Q(   #EsA?ch|j}|D]}||} t|j}|S)z~ The next possible tokens that will progress the trie, given the current sequence of tokens in `current_seq`. )rerDkeys)r current_seqstart current_token next_tokenss rrpzDisjunctiveTrie.next_tokenss> ( )M-(E )5::<( rc@|j|}t|dk(SNr)rprE)rrmrps r reached_leafzDisjunctiveTrie.reached_leafs"&&{3 ;1$$rct|j}t|dk(ryt|Dcgc]}|j |c}Scc}w)Nrr)rDvaluesrEsum count_leaves)rrh next_nodesnns rrwzDisjunctiveTrie.count_leavessE$++-( z?a  C"))"-CD DCsAc@|j|}t||k7S)zh Returns whether # of leaves == # of words. Otherwise some word is a subset of another. )rwrE)rrer_ leaf_counts rrdzDisjunctiveTrie.has_subsets s%&&t, #$ 22rNT) r/r0r1rDr3rrprsrwrdr4rrr^r^s+d3i. % E3rr^cfeZdZdZdeeeffd ZdZdefdZdefdZ dZ d Z d d Z xZ S) DisjunctiveConstrainta3 A special [`Constraint`] that is fulfilled by fulfilling just one of several constraints. Args: nested_token_ids (`list[list[int]]`): A list of words, where each word is a list of ids. This constraint is fulfilled by generating just one from the list of words. r_ctt| t|trt |dk(rt d|dtd|Drt d|dtd|Drt d|dt||_ ||_ |jj|_ g|_ d|_y) Nrz6`nested_token_ids` has to be a non-empty list, but is r9c3>K|]}t|t ywrV)r=rDr?r7s rr@z1DisjunctiveConstraint.__init__.. sQ9:i..Qsz5`nested_token_ids` has to be a list of lists, but is c3@K|]}td|Dyw)c3LK|]}t|t xs|dkywr;r<r>s rr@z;DisjunctiveConstraint.__init__...#s&ZHZ#..>(Q,>ZrAN)rGrs rr@z1DisjunctiveConstraint.__init__.."s$  ZPYZ Z szNEach list in `nested_token_ids` has to be a list of positive integers, but is F)rCr rr=rDrErFrGr^rer7rbrHrmr)rr_r"s rrzDisjunctiveConstraint.__init__s j$(**D1S9I5Ja5OUVfUgghij j Q@PQ QTUeTffghi i  -  `aq`rrst $$45 )ii** rcn|jj|j}t|dk(ry|Srr)rerprmrE)r token_lists rrzDisjunctiveConstraint.advance1s1YY**4+;+;< z?a  rr$ct|tstd|dt||jj |j }||vS)N0`token_id` is supposed to be type `int`, but is rN)r=r3rOrPrerprm)rr$rps rrz"DisjunctiveConstraint.does_advance9sS(C(NxjXabfgobpaqrs sii++D,<,<= ;&&rcFt|tstd|dt|d}d}d}|j |r|j j |d}nd}|j|jj|j }||_ |||fS)NrrNFT) r=r3rOrPrrmappendrrersrrRs rrzDisjunctiveConstraint.updateAs(C(NxjXabfgobpaqrs s    X &    # #H -GE JJLII**4+;+;< " 5((rc d|_g|_yNF)rrmrs rrzDisjunctiveConstraint.resetUsrc`|jry|jt|jz Srr)rrHrErmrs rrzDisjunctiveConstraint.remainingYs& >>;;T%5%5!66 6rct|j}|r3|j|_|j|_|j |_|SrV)r~r7rHrWrmrrXs rr-zDisjunctiveConstraint.copy`s?.t~~> %)[[N ")-)9)9N &'+~~N $rr.rZr\s@rr~r~sHd3i,'S')s)(7rr~c`eZdZdZdeefdZdZdZdZ de ee fdZ d e fd Z d d Zy )ConstraintListStatez A class for beam scorers to track its progress through a list of constraints. Args: constraints (`list[Constraint]`): A list of [`Constraint`] objects that must be fulfilled by the beam scorer. constraintsc||_t|Dcgc]}|jc}|_t ||_d|_|jycc}wr)rrarH max_seqlenrE n_constraintsr init_state)rrcs rrzConstraintListState.__init__tsJ&=Aqxx=> -  >sAcg|_d|_|jDcgc]}|jdc}|_ycc}w)NFr,)complete_constraintsinprogress_constraintrr-pending_constraints)r constraints rrzConstraintListState.init_state~s9$&!%)"VZVfVf#g JOOUO$C#g #gs?cd}|jr*||j|jjz z }t|j|jz|zSrr)rrrrEr)radds rget_bankzConstraintListState.get_banksQ  % % 4??T%?%?%I%I%KK KCD--.@CGGrcg}|jf|jD]V}|j}t|tr|j |5t|t sF|j|Xn]|jj}t|tr|j |n!t|t r|j|t|dk(ry|S)aThe list of tokens to generate such that we can make progress. By "list" we don't mean the list of token that will fully fulfill a constraint. Given constraints `c_i = {t_ij | j == # of tokens}`, If we're not in the middle of progressing through a specific constraint `c_i`, we return: `[t_k1 for k in indices of unfulfilled constraints]` If we are in the middle of a constraint, then we return: `[t_ij]`, where `i` is the index of the inprogress constraint, `j` is the next step for the constraint. Though we don't care which constraint is fulfilled first, if we are in the progress of fulfilling a constraint, that's the only one we'll return. Nr) rrrr=r3rrDextendrE)rrrrs rrzConstraintListState.advances  % % -"66 / $,,.gs+%%g..%%g.  /0088:G'3'!!'*GT*!!'* z?a  rr7c||j|*|D]$}|j|\}}|js$yyy)zr token_ids: the tokens generated thus far to reset the state of the progress through constraints. N)rrr)rr7tokencompleters rrzConstraintListState.resetsF   " $(HHUO!'>>   !rr$ct|tstd|dd\}}|jrd}d}||fS|j|jj |\}}}|r<|j j|jjdd|_|rK|jj|jd|_t|j dk(rd|_||fSt|j D]\}}|j|s|j |\}}}|s td|r"|jj|d|_|s |r||_|s|sn|j d||j |d zdz|_t|j dk(r|jd|_||fS||fS) Nz'`token_id` should be an `int`, but is `z`.)FFTFrrz|`constraint.update(token_id)` is not yielding incremental progress, even though `constraint.does_advance(token_id)` is true.r)r=r3rOrrrrrr-rrErcrr)rr$rrrcidxpending_constraints rrzConstraintListState.adds(C(EhZrRS S(' >>HGW$ $  % % 1(,'A'A'H'H'R $GXu((//0J0J0O0OY^0O0_`-1* ))001K1KL-1*t//0A5%)DNL  A-6d6N6N,O ((%228V>VW[^_W_Wa>bb0t778A=$B\B\Bd.2DN  A @  rcZt|j}|r|jDcgc]}|jdc}|_|j!|jjd|_|j Dcgc]}|jc}|_|Scc}wcc}w)NTr)rrrr-rr)rr, new_staters rr-zConstraintListState.copy s'(8(89  AEAZAZ.3= ..I *))5262L2L2Q2Q[_2Q2` /QUQiQi,j:Z__->,jI ). -ks B#B(Nr|)r/r0r1r2rDr rrrrrr3rrr-r4rrrrksSD$4h H!F xS 2 J!CJ!X rrN)abcrrtypingrutilsr get_loggerr/r r r6r^r~rr4rrrsf#   H %  DH HV6363rWJWtjjr