JL idZddlmZmZddlmZmZmZGddeZGddeZ Gdd e Z Gd d eZ Gd d eZ Gdde Z Gdde ZGdde Zy)zLanguage Models) LanguageModel Smoothing)AbsoluteDiscounting KneserNey WittenBellceZdZdZddZy)MLEzbClass for providing MLE ngram model scores. Inherits initialization from BaseNgramModel. NcB|j|j|S)zReturns the MLE score for a word given a context. Args: - word is expected to be a string - context is expected to be something reasonably convertible to a tuple )context_countsfreq)selfwordcontexts T/mnt/ssd/data/python-lab/Trading/venv/lib/python3.12/site-packages/nltk/lm/models.pyunmasked_scorezMLE.unmasked_scores ""7+0066N)__name__ __module__ __qualname____doc__rrrr r s  7rr c*eZdZdZfdZddZxZS)LidstonezProvides Lidstone-smoothed scores. In addition to initialization arguments from BaseNgramModel also requires a number by which to increase the counts, gamma. c2t||i|||_yr)super__init__gamma)r rargskwargs __class__s rrzLidstone.__init__% $)&) rc|j|}||}|j}||jz|t|j|jzzz S)ztAdd-one smoothing: Lidstone or Laplace. To see what kind, look at `gamma` attribute on the class. )r Nrlenvocabr rrcounts word_count norm_counts rrzLidstone.unmasked_score)sR $$W-D\ XXZ TZZ'JTZZ4::9U,UVVrrrrrrrr __classcell__r!s@rrrs  Wrrc"eZdZdZfdZxZS)LaplacezwImplements Laplace (add one) smoothing. Initialization identical to BaseNgramModel because gamma is always 1. c,t|dg|i|y)N)rr)r rr r!s rrzLaplace.__init__;s ,T,V,rrrrrrr,r-s@rr/r/5s --rr/c,eZdZdZdfd ZddZxZS) StupidBackoffa8Provides StupidBackoff scores. In addition to initialization arguments from BaseNgramModel also requires a parameter alpha with which we scale the lower order probabilities. Note that this is not a true probability distribution as scores for ngrams of the same order do not sum up to unity. c2t||i|||_yr)rralpha)r r6rr r!s rrzStupidBackoff.__init__Hr"rc|s%|jjj|S|j|}||}|j }|dkDr||z S|j |j ||ddzS)Nrr1)r(unigramsr r r$r6rr's rrzStupidBackoff.unmasked_scoreLsx;;'',,T2 2$$W-D\ XXZ > * *:: 3 3D'!"+ FF Fr)g?rr+r-s@rr4r4?s Grr4c*eZdZdZfdZddZxZS)InterpolatedLanguageModelzLogic common to all interpolated language models. The idea to abstract this comes from Chen & Goodman 1995. Do not instantiate this class directly! c |jdi}t||fi|||j|jfi||_y)Nparams)poprrr&r( estimator)r smoothing_clsorderr r<r!s rrz"InterpolatedLanguageModel.__init__`s@Hb) )&)&tzz4;;I&Irc|s|jj|S|j|sd\}}n|jj||\}}|||j ||ddzzS)N)rr1r1)r> unigram_scorer( alpha_gammar)r rrr6rs rrz(InterpolatedLanguageModel.unmasked_scoreesn>>//5 5{{7# LE5>>55dGDLE5ut224EEEErrr+r-s@rr:r:Ys J Frr:c"eZdZdZfdZxZS)WittenBellInterpolatedz.Interpolated version of Witten-Bell smoothing.c 0t|t|fi|yr)rrr)r r@r r!s rrzWittenBellInterpolated.__init__vs U5f5rr2r-s@rrErEss866rrEc$eZdZdZdfd ZxZS)AbsoluteDiscountingInterpolatedz9Interpolated version of smoothing with absolute discount.c 8t|t|fdd|ii|y)Nr<discount)rrrr r@rJr r!s rrz(AbsoluteDiscountingInterpolated.__init__}s*   0:H/E IO r)g?r2r-s@rrHrHzsC  rrHc$eZdZdZdfd ZxZS)KneserNeyInterpolatedz-Interpolated version of Kneser-Ney smoothing.c ~d|cxkrdkstdtdt|t|fd||di|y)Nrr1zCDiscount must be between 0 and 1 for probabilities to sum to unity.r<)rJr@) ValueErrorrrrrKs rrzKneserNeyInterpolated.__init__s^X""U #U   u 2:U%K OU r)g?r2r-s@rrMrMs7  rrMN)r nltk.lm.apirrnltk.lm.smoothingrrrr rr/r4r:rErHrMrrrrRs|0HH 7- 7 W}W.-h-GMG4F F4666 &?   5  r