JL iMzdZddlmZddlmZddlmZddlmZdZ GddZ Gd d Z Gd d Z y )a Common methods and classes for all IBM models. See ``IBMModel1``, ``IBMModel2``, ``IBMModel3``, ``IBMModel4``, and ``IBMModel5`` for specific implementations. The IBM models are a series of generative models that learn lexical translation probabilities, p(target language word|source language word), given a sentence-aligned parallel corpus. The models increase in sophistication from model 1 to 5. Typically, the output of lower models is used to seed the higher models. All models use the Expectation-Maximization (EM) algorithm to learn various probability tables. Words in a sentence are one-indexed. The first word of a sentence has position 1, not 0. Index 0 is reserved in the source sentence for the NULL token. The concept of position does not apply to NULL, but it is indexed at 0 by convention. Each target word is aligned to exactly one source word or the NULL token. References: Philipp Koehn. 2010. Statistical Machine Translation. Cambridge University Press, New York. Peter E Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. The Mathematics of Statistical Machine Translation: Parameter Estimation. Computational Linguistics, 19 (2), 263-311. ) insort_left defaultdict)deepcopy)ceilcZd}|D]#}t|j}t||}%|S)z :param sentence_aligned_corpus: Parallel corpus under consideration :type sentence_aligned_corpus: list(AlignedSent) :return: Number of words in the longest target language sentence of ``sentence_aligned_corpus`` r)lenwordsmax)sentence_aligned_corpusmax_maligned_sentencems ^/mnt/ssd/data/python-lab/Trading/venv/lib/python3.12/site-packages/nltk/translate/ibm_model.pylongest_target_sentence_lengthr.s< E3  && 'Au  LcheZdZdZdZdZdZdZdZdZ dd Z dd Z dd Z d Z d ZdZdZdZy)IBMModelz0 Abstract base class for all IBM models g-q=cF|j||jyN) init_vocabreset_probabilitiesselfr s r__init__zIBMModel.__init__Hs /0   "rctd_ td_ tfd_ d_y)NctdS)Nc"tjSrrMIN_PROBrrz@IBMModel.reset_probabilities....Ns (9(9rrr!rrr"z.IBMModel.reset_probabilities..Ns K 9:rctdS)NctdS)NctdS)Nc"tjSrrr!rrr"zdIBMModel.reset_probabilities........Ws @Q@Qrrr!rrr"zRIBMModel.reset_probabilities......Ws K8Q,Rrrr!rrr"z@IBMModel.reset_probabilities....Ws $RSrrr!rrr"z.IBMModel.reset_probabilities..VsKSrc tfdS)NcjSr)r rsrr"z@IBMModel.reset_probabilities....`s t}}rrr)srr"z.IBMModel.reset_probabilities..`s ;?T3Urg?)rtranslation_tablealignment_tablefertility_tablep1r)s`rrzIBMModel.reset_probabilitiesLsT!, :"  +    ++UV  rcy)z Initialize probability tables to a uniform distribution Derived classes should implement this accordingly. Nr!rs rset_uniform_probabilitiesz"IBMModel.set_uniform_probabilitiesns rct}t}|D]8}|j|j|j|j:|j d||_ ||_yr)setupdater motsadd src_vocab trg_vocab)rr r5r6rs rrzIBMModel.init_vocabvsrE E 7 4    -33 4   -22 3 4  d" # rct}t|j}t|j}|j |}|j |}|j |j||}td|dzD]z}td|dzD]f} |j ||| }|j ||}|j||} |j | |j|jkDse|}h|||fS)a Sample the most probable alignments from the entire alignment space First, determine the best alignment according to IBM Model 2. With this initial alignment, use hill climbing to determine the best alignment according to a higher IBM Model. Add this alignment and its neighbors to the sample set. Repeat this process with other initial alignments obtained by pegging an alignment point. Hill climbing may be stuck in a local maxima, hence the pegging and trying out of different alignments. :param sentence_pair: Source and target language sentence pair to generate a sample of alignments from :type sentence_pair: AlignedSent :return: A set of best alignments represented by their ``AlignmentInfo`` and the best alignment of the set for convenience :rtype: set(AlignmentInfo), AlignmentInfo r) r1r r3r best_model2_alignment hillclimbr2 neighboringrangescore) r sentence_pairsampled_alignmentslrinitial_alignmentpotential_alignmentbest_alignmentji neighborss rsamplezIBMModel.samples.!U  "" #  ## $!66}E"nn->?!!$"2"23F"GH,q!a% 9A1a!e_ 9$($>$>}aQR$S!&*nn5F&J# ,,-@!D ")))4&,,~/C/CC%8N  9 9">11rNcRdg|jz}dg|jz}t|dz }t|dz }dg|dzz}t|dzD cgc]} g} } td|dzD]} | |k(r|} ned} tj } || }td|dzD]<} || }|j |||j| | ||z}|| k\s9|} | } >| || <| | j| tt|t|t|| Scc} w)aT Finds the best alignment according to IBM Model 2 Used as a starting point for hill climbing in Models 3 and above, because it is easier to compute than the best alignments in higher models :param sentence_pair: Source and target language sentence pair to be word-aligned :type sentence_pair: AlignedSent :param j_pegged: If specified, the alignment point of j_pegged will be fixed to i_pegged :type j_pegged: int :param i_pegged: Alignment point to j_pegged :type i_pegged: int NUNUSEDr8r) r3r r r<rr r*r+append AlignmentInfotuple)rr>j_peggedi_pegged src_sentence trg_sentencer@r alignmentrEceptsrDbest_imax_alignment_probtsalignment_probs rr9zIBMModel.best_model2_alignmentsi&v 2 22  zM$7$77   !   !C1q5M "1q5\***q!a% $AH}!%-%6%6" Oq!a%#A$QA..q1!4t7K7KA7Nq7QRS7TUV7WW#&);;-;*!"#"IaL &M  #) $, ) eL153F  1+s D$c|}|j|} |}|j||D]}|j|}||kDs|}|}||k(rn;||_|S)a, Starting from the alignment in ``alignment_info``, look at neighboring alignments iteratively for the best one There is no guarantee that the best alignment in the alignment space will be found, because the algorithm might be stuck in a local maximum. :param j_pegged: If specified, the search will be constrained to alignments where ``j_pegged`` remains unchanged :type j_pegged: int :return: The best alignment found from hill climbing :rtype: AlignmentInfo )prob_t_a_given_sr;r=)ralignment_inforMrQmax_probability old_alignmentneighbor_alignmentneighbor_probabilitys rr:zIBMModel.hillclimbs # // :%M&*&6&6y(&K ;"'+'<'<=O'P$'/9 2I&:O  ;M)* rct}t|jdz }t|jdz }|j}|j }t d|dzD]}||k7s t d|dzD]} t|} t|} ||} | | |<t| | || | j|tt| |j|j| } |j| t d|dzD]}||k7s t d|dzD]}||k7s ||k7st|} t|} ||}||} || |<| | |<| |j|t| ||| | j|t| | |tt| |j|j| } |j| |S)a Determine the neighbors of ``alignment_info``, obtained by moving or swapping one alignment point :param j_pegged: If specified, neighbors that have a different alignment point from j_pegged will not be considered :type j_pegged: int :return: A set neighboring alignments represented by their ``AlignmentInfo`` :rtype: set(AlignmentInfo) r8r)r1r rOrPrQrRr<listrrremoverKrLr4)rrZrMrFr@roriginal_alignmentoriginal_ceptsrDrE new_alignment new_ceptsold_inew_alignment_infoother_jother_is rr;zIBMModel.neighboringsE ++ ,q 0 ++ ,q 0+55'--q!a% 6AH}q!a%6A$();$  # ) )     rc2t|j|S)zL Fertility of word in position ``i`` of the source sentence )r rR)rrEs rfertility_of_izAlignmentInfo.fertility_of_is4::a=!!rcJ|j|}|j|d|k(S)zo :return: Whether the word in position ``j`` of the target sentence is a head word r)rQrR)rrDrEs r is_head_wordzAlignmentInfo.is_head_words* NN1 zz!}Q1$$rc|yt|j|t|j|z }tt |S)z :return: The ceiling of the average positions of the words in the tablet of cept ``i``, or 0 if ``i`` is None r)sumrRr intr)rrEaverage_positions rcenter_of_ceptzAlignmentInfo.center_of_ceptsB 9tzz!}-DJJqM0BB4()**rc|j|}|dk(r td|dz }|dkDr3|j|dk(r|dz}|dkDr|j|dk(r|dkrd}|S)zm :return: The previous cept of ``j``, or None if ``j`` belongs to the first cept rzNWords aligned to NULL cannot have a previous cept because NULL has no positionr8N)rQ ValueErrorr)rrDrE previous_cepts rrzAlignmentInfo.previous_cepts NN1  64 A aD$7$7 $F!$K Q MaD$7$7 $F!$K A  Mrc|j|}|j|j|}|dk(ry|j||dz S)z :return: The position of the previous word that is in the same tablet as ``j``, or None if ``j`` is the first word of the tablet rNr8)rQrRindex)rrDrEtablet_positions rprevious_in_tabletz AlignmentInfo.previous_in_tabletsM NN1 **Q---a0 a zz!}_q011rcg}tdt|jD]1}|j|dz }|dkrd}|j |dz |f3|S)z :return: Zero-indexed alignment, suitable for use in external ``nltk.translate`` modules like ``nltk.translate.Alignment`` :rtype: list(tuple) r8rN)r<r rPrQrJ)rzero_indexed_alignmentrDrEs rrz$AlignmentInfo.zero_indexed_alignmentsh "$q#d//01 6Aq!A%A1u " ) )1q5!* 5  6 &%rc4|j|jk(Sr)rQrothers r__eq__zAlignmentInfo.__eq__s~~00rc||k( Srr!rs r__ne__zAlignmentInfo.__ne__s5=  rc,t|jSr)hashrQr)s r__hash__zAlignmentInfo.__hash__sDNN##rN)rrrrrrrrrrrrrrr!rrrKrKs: ( T" % +& 2 &1!$rrKc(eZdZdZdZdZdZdZy)CountszK Data object to store counts of various parameters during training ctd|_tt|_d|_d|_td|_tt|_y)Nc ttSrrfloatr!rrr"z!Counts.__init__..  [-?rrc ttSrrr!rrr"z!Counts.__init__..rr)rrkrrmrxr-rsrtr)s rrzCounts.__init__ sD$%?@(/$%?@%0%7"rc|j|}|j|}|j|}|j||xx|z cc<|j|xx|z cc<yr)rQrPrOrkrm)rcountrZrDrErUrVs rupdate_lexical_translationz!Counts.update_lexical_translationsa  $ $Q '  ' ' *  ' ' * q!% 1&rct|jdz }|jd}|xj||zz c_|xj|d|zz |zz c_y)Nr8r)r rPrr-rx)rrrZrfertility_of_nulls rupdate_null_generationzCounts.update_null_generationsZ ++ ,q 0*99!< $u,, A---66rctdt|jD]S}|j|}|j|}|j||xx|z cc<|j |xx|z cc<Uyr})r<r rOrrsrt)rrrZrErVrus rupdate_fertilityzCounts.update_fertility srq#n99:; 3A++A.A //2C NN3  "e + "  & &q )U 2 )  3rN)rrrrrrrrr!rrrrs8'7 3rrN) rbisectr collectionsrcopyrmathrrrrKrr!rrrsB@# @@F D$D$N33r