JL i:ddlZddlmZddlmZGddeZy)N)Tuple)StemmerIceZdZdZej dZej dZej dZej dZ ej dZ ej dZ ej dZ dd e fd Zed ed efd Zed ed efdZd ed efdZd ed eeeffdZd ede fdZy)Cistema: CISTEM Stemmer for German This is the official Python implementation of the CISTEM stemmer. It is based on the paper Leonie Weissweiler, Alexander Fraser (2017). Developing a Stemmer for German Based on a Comparative Analysis of Publicly Available Stemmers. In Proceedings of the German Society for Computational Linguistics and Language Technology (GSCL) which can be read here: https://www.cis.lmu.de/~weissweiler/cistem/ In the paper, we conducted an analysis of publicly available stemmers, developed two gold standards for German stemming and evaluated the stemmers based on the two gold standards. We then proposed the stemmer implemented here and show that it achieves slightly better f-measure than the other stemmers and is thrice as fast as the Snowball stemmer for German while being about as fast as most other stemmers. case_insensitive is a a boolean specifying if case-insensitive stemming should be used. Case insensitivity improves performance only if words in the text may be incorrectly upper case. For all-lowercase and correctly cased text, best performance is achieved by setting case_insensitive for false. :param case_insensitive: if True, the stemming is case insensitive. False by default. :type case_insensitive: bool z ^ge(.{4,})z(.)\1ze[mr]$znd$zt$z[esn]$z(.)\*case_insensitivec||_y)N)_case_insensitive)selfrs V/mnt/ssd/data/python-lab/Trading/venv/lib/python3.12/site-packages/nltk/stem/cistem.py__init__zCistem.__init__5s !1wordreturnc|jdd}|jdd}|jdd}tjjd|}|S)Nsch$ei%ie&z\1*)replacerrepl_xxsubrs r replace_tozCistem.replace_to8sM||E3'||D#&||D#&~~!!&$/ r ctjjd|}|jdd}|jdd}|jdd}|S)Nz\1\1rrrrrr)r repl_xx_backrrrs r replace_backzCistem.replace_backAsO""&&w5||C&||C&||C' r cbt|dk(r|S|dj}|j}|jdd}|jdd}|jdd}|jdd }tj j d |}|j||dS) uStems the input word. :param word: The word that is to be stemmed. :type word: str :return: The stemmed word. :rtype: str >>> from nltk.stem.cistem import Cistem >>> stemmer = Cistem() >>> s1 = "Speicherbehältern" >>> stemmer.stem(s1) 'speicherbehalt' >>> s2 = "Grenzpostens" >>> stemmer.stem(s2) 'grenzpost' >>> s3 = "Ausgefeiltere" >>> stemmer.stem(s3) 'ausgefeilt' >>> stemmer = Cistem(True) >>> stemmer.stem(s1) 'speicherbehal' >>> stemmer.stem(s2) 'grenzpo' >>> stemmer.stem(s3) 'ausgefeil' rüuöoäaßssz\1)lenisupperlowerrrstrip_ger_segment_innerr ruppers r stemz Cistem.stemJs6 t9>KQ!zz|||D#&||D#&||D#&||D$'""5$/""4/22r ct|dk(ry|dj}|j}|j||S)u This method works very similarly to stem (:func:'cistem.stem'). The difference is that in addition to returning the stem, it also returns the rest that was removed at the end. To be able to return the stem unchanged so the stem and the rest can be concatenated to form the original word, all subsitutions that altered the stem in any other way than by removing letters at the end were left out. :param word: The word that is to be stemmed. :type word: str :return: A tuple of the stemmed word and the removed suffix. :rtype: Tuple[str, str] >>> from nltk.stem.cistem import Cistem >>> stemmer = Cistem() >>> s1 = "Speicherbehältern" >>> stemmer.segment(s1) ('speicherbehält', 'ern') >>> s2 = "Grenzpostens" >>> stemmer.segment(s2) ('grenzpost', 'ens') >>> s3 = "Ausgefeiltere" >>> stemmer.segment(s3) ('ausgefeilt', 'ere') >>> stemmer = Cistem(True) >>> stemmer.segment(s1) ('speicherbehäl', 'tern') >>> stemmer.segment(s2) ('grenzpo', 'stens') >>> stemmer.segment(s3) ('ausgefeil', 'tere') r)r1)r(r)r*r,r-s r segmentzCistem.segmenttsA@ t9>Q!zz|""4//r r.cFd}|dd}tj|}d}t|dkDrt|dkDr\tjj d|\}}|dk7r|dz }Jtj j d|\}}|dk7r|dz }x|r |j r.tjj d|\}}|dk7r|dz }tjj d|\}}|dk7r|dz } tj|}|r|| d}||fS)a5Inner method for iteratively applying the code stemming regexes. This method receives a pre-processed variant of the word to be stemmed, or the word to be segmented, and returns a tuple of the word and the removed suffix. :param word: A pre-processed variant of the word that is to be stemmed. :type word: str :param upper: Whether the original word started with a capital letter. :type upper: bool :return: A tuple of the stemmed word and the removed suffix. :rtype: Tuple[str, str] rNr1) rrr( strip_emrsubnstrip_ndr strip_t strip_esnr)r rr. rest_length word_copyrestns r r,zCistem._segment_inners8 G   &$i!m4y1} **//D9a61$K //..r48a61$KD22 ..--b$7a61$K&&++B5GD!Avq ""4( k\]+Dd|r N)F)__name__ __module__ __qualname____doc__recompiler+rr8r:r;r<rboolr staticmethodstrrrr/rr2r,r r rrs 8rzz-(Hbjj"G 9%Irzz&!HbjjG 9%I2::h'L2233(3(3(3T&0C&0E#s(O&0P535t5r r)rEtypingr nltk.stem.apirrrJr r rMs "AXAr