JL i8rdZddlmZdZdZdZdZdZdZd Z Gd d Z d Z e d k(re yy)aCounts Paice's performance statistics for evaluating stemming algorithms. What is required: - A dictionary of words grouped by their real lemmas - A dictionary of words grouped by stems from a stemming algorithm When these are given, Understemming Index (UI), Overstemming Index (OI), Stemming Weight (SW) and Error-rate relative to truncation (ERRT) are counted. References: Chris D. Paice (1994). An evaluation method for stemming algorithms. In Proceedings of SIGIR, 42--50. )sqrtcbt}|D]}|jt||!|S)a5 Get original set of words used for analysis. :param lemmas: A dictionary where keys are lemmas and values are sets or lists of words corresponding to that lemma. :type lemmas: dict(str): list(str) :return: Set of words that exist as values in the dictionary :rtype: set(str) )setupdate)lemmaswordslemmas X/mnt/ssd/data/python-lab/Trading/venv/lib/python3.12/site-packages/nltk/metrics/paice.pyget_words_from_dictionaryr s4 EE) S'() Lcxi}|D]}|d|} ||j|g|S#t$r |h||<Y3wxYw)aGroup words by stems defined by truncating them at given length. :param words: Set of words used for analysis :param cutlength: Words are stemmed by cutting at this length. :type words: set(str) or list(str) :type cutlength: int :return: Dictionary where keys are stems and values are sets of words corresponding to that stem. :rtype: dict(str): set(str) N)rKeyError)r cutlengthstemswordstems r _truncater*s` E!JY ! $K  v &! L !&E$K !s '99cR|d\}}|d\}}|d\}}|d\}} ||z || z z||z ||z zz } | dk(r||cxk(r|cxk(r|cxk(rdk(rd| fS||z||zz ||z z||z || z||zz zz | z } ||z||zz || z z||z || z||zz zz | z } | | fS)a{Count intersection between two line segments defined by coordinate pairs. :param l1: Tuple of two coordinate pairs defining the first line segment :param l2: Tuple of two coordinate pairs defining the second line segment :type l1: tuple(float, float) :type l2: tuple(float, float) :return: Coordinates of the intersection :rtype: tuple(float, float) r) l1l2x1y1x2y2x3y3x4y4 denominatorxys r _count_intersectionr%@sUFB UFB UFB UFB7rBw'27rBw*??Kc  &r &R &3 & 9  ' b27 rBw'27rBwb7H*II A b27 rBw'27rBwb7H*II A q6Mr cN |d|dz S#t$rtdcYSwxYw)zGet derivative of the line from (0,0) to given coordinates. :param coordinates: A coordinate pair :type coordinates: tuple(float, float) :return: Derivative; inf if x is zero :rtype: float rrinfZeroDivisionErrorfloat) coordinatess r _get_derivativer,cs31~ A.. U|s $$cd\}}|D]W}t|t||z}|s t|}t||}||t||z zz }||||z zz }Y||fS)a Count understemmed and overstemmed pairs for (lemma, stem) pair with common words. :param lemmawords: Set or list of words corresponding to certain lemma. :param stems: A dictionary where keys are stems and values are sets or lists of words corresponding to that stem. :type lemmawords: set(str) or list(str) :type stems: dict(str): set(str) :return: Amount of understemmed and overstemmed pairs contributed by words existing in both lemmawords and stems. :rtype: tuple(float, float) rr)rlen) lemmawordsrumtwmtrcutcutcount stemcounts r _calculate_cutr6qsHC5*oE$K 00 3xHE$K(I 8s:9: :C 8y834 4C5 :r ctfdD}d\}}}}D]B}t|}|||dz zz }||||z zz }t||\} } || z }|| z }D|dz |dz |dz |dz fS)aCalculate actual and maximum possible amounts of understemmed and overstemmed word pairs. :param lemmas: A dictionary where keys are lemmas and values are sets or lists of words corresponding to that lemma. :param stems: A dictionary where keys are stems and values are sets or lists of words corresponding to that stem. :type lemmas: dict(str): list(str) :type stems: dict(str): set(str) :return: Global unachieved merge total (gumt), global desired merge total (gdmt), global wrongly merged total (gwmt) and global desired non-merge total (gdnt). :rtype: tuple(float, float, float, float) c3:K|]}t|ywNr/).0rrs r z_calculate..s 1$Ct  1s)rrrrr)sumr/r6) rrngdmtgdntgumtgwmtr lemmacountr1r2s ` r _calculaterEs 1& 11A1D$d'  j1n-- a*n--"&-7S    $ 1HdQhq$( 33r c ||z } ||z } ||z }|||fS#t$rd}YwxYw#t$rd}Y)wxYw#t$r|dk(r td}n td}YNwxYw)aCount Understemming Index (UI), Overstemming Index (OI) and Stemming Weight (SW). :param gumt, gdmt, gwmt, gdnt: Global unachieved merge total (gumt), global desired merge total (gdmt), global wrongly merged total (gwmt) and global desired non-merge total (gdnt). :type gumt, gdmt, gwmt, gdnt: float :return: Understemming Index (UI), Overstemming Index (OI) and Stemming Weight (SW). :rtype: tuple(float, float, float) rnanr'r()rBr@rCrAuioisws r _indexesrKs D[ D[ "W B<#      9uBuB s'): && 77%A"!A"c6eZdZdZdZdZdZd dZdZdZ y) Paicez7Class for storing lemmas, stems and evaluation metrics.c||_||_g|_d\|_|_|_|_d\|_|_|_ d|_ |jy)al :param lemmas: A dictionary where keys are lemmas and values are sets or lists of words corresponding to that lemma. :param stems: A dictionary where keys are stems and values are sets or lists of words corresponding to that stem. :type lemmas: dict(str): list(str) :type stems: dict(str): set(str) )NNNN)NNNN) rrcoordsrBr@rCrArHrIrJerrtr)selfrrs r __init__zPaice.__init__sQ   5M2 49di$6!$'  r crd|jzg}|jd|jz|jd|jz|jd|jz|jd|j z|jd|j z|jd|jz|jd|jzd j|jDcgc]}d |z c}}|jd |zd j|Scc}w) Nz)Global Unachieved Merge Total (GUMT): %s z&Global Desired Merge Total (GDMT): %s z'Global Wrongly-Merged Total (GWMT): %s z*Global Desired Non-merge Total (GDNT): %s z&Understemming Index (GUMT / GDMT): %s z%Overstemming Index (GWMT / GDNT): %s zStemming Weight (OI / UI): %s z.Error-Rate Relative to Truncation (ERRT): %s  z(%s, %s)zTruncation line: %s) rBappendr@rCrArHrIrJrPjoinrO)rQtextitemr+s r __str__z Paice.__str__sJK ADIIMN =GH .s4dD 4sr.r=rrr)r rmaxr]rVr/r,rJ)rQrr maxlengthrOpair derivative1 derivative2s r _get_truncation_coordinatesz!Paice._get_truncation_coordinates s*$++64e44 9$//yAD6! d#z! 6{aDGcM-fRj9 -fRj9 $''8[8!M9 NI)9$* r c|j|_d|jvr1|j|jfdk7r t dSt dS|j|jfdk(ryt d|j|jff|jdd}t |jdz|jdzz}t |d dz|d dzz}||z S) aCount Error-Rate Relative to Truncation (ERRT). :return: ERRT, length of the line from origo to (UI, OI) divided by the length of the line from origo to the point defined by the same line when extended until the truncation line. :rtype: float r.r'rGr)rrr`Nr=rr)rgrOrHrIr*r%r)rQ intersectionopots r _errtz Paice._errt0s668  $!Z/U|#U|# GGTWW  ++ dggtww' ($++bc*: $''1*twwz) * ,q/Q&aA)== >Bwr c2t|j|j\|_|_|_|_t|j|j|j |j \|_|_ |_ |j|_ y)z7Update statistics after lemmas and stems have been set.N) rErrrBr@rCrArKrHrIrJrlrP)rQs r rz Paice.updatePsa5? TZZ5X2 49di$,TYY 499dii$X!$'JJL r N)r) __name__ __module__ __qualname____doc__rRrZr]rgrlrrr r rMrMs%A"  %N@!r rMc ddgddggdd}dgdggddgd gd }td t|D]/}td j|d j||1ttdt|D]/}td j|d j||1tt ||}t|tdgdgdgddgdgd gd}tdt|D]/}td j|d j||1t||_|j t|y)zDemonstration of the module.kneelkneltrangeranged)ringrangrung)rsrurw)rxrurvrwry)rsrtrxrwryzWords grouped by their lemmas:z{} => {}rTz+Same words grouped by a stemming algorithm:rx)rsrtrxrurwryz/Counting stats after changing stemming results:N)printsortedformatrWrMrr)rrr rps r demor~Wsf7#8$(F+  E *+A jsxxu '>?@A G 78u > jchhuT{&;<=> G feA !H G8$  E ;<u > jchhuT{&;<=> GAGHHJ !Hr __main__N) rqmathrr rr%r,r6rErKrMr~rnrr r rs\   , F 2&4R"J|!|!~* Z zFr