MpjyCxddlmZmZmZddlmZddlmZddlmZddlm Z m Z ddl m Z ddl mZddlZdd lmZmZmZmZddlZej,d ZGd d eZGd deZdde deedfdefdZdeeefdeedffdZGddeZGddeZGddeZ de!de"fdZ#dZ$y))UnionListTuple) TypedDict) Generator) Transformer)v_argsDiscard)Tree)TokenN)log_htmlrtf_strippinglog_htmlrtf_statelog_htmlrtf_evaluated_chars is_logger_onRTFDEcXeZdZdZeddefdZedede e dffd Z y) StripNonVisibleRTFGroupszVisits each Token in provided RTF Trees and strips out any RTF groups which are non-visible when de-encapsulated into HTML. T)treerc6|j}t|dk(ry|d}dg}t|tr|j|vr|Sdg}t|tr|j|vrygd}|j |}||vryg}d} t|trHt|jdk7r0|jd} t| t r| jdk(rd} d} | dur7|d } t| t r"| jd k(r| } | j|vr|Sy|S) aTransformer which aggressively seeks out possible non-visible RTF groups and replaces them with empty strings. NOTE: Currently deleting all groups that don't have an htmltag. Please file an issue if you find one that should be included in de-encapsulated HTML. I will refine what gets deleted and what is converted based on identified needs for greater functionality or specific issues which need to be addressed. Args: tree: A .rtf group (Tree object) which needs its contents decoded. r htmltag_groupmhtmltag_group)s\contexts \colortbls\fonttblN STAR_ESCAPET CONTROLWORD) childrenlen isinstancer dataget_first_controlwordr typevalue) selfrr first_childknown_control_groups known_non_visible_control_groupsnon_visible_control_words first_controlunderstood_commandsis_star_escaped first_item control_word first_tokens `/mnt/ssd/data/Dropbox/adrian/scripts/msg_venv/lib/python3.12/site-packages/RTFDE/transformers.pygroupzStripNonVisibleRTFGroups.group$s:== x=A qk /0 k4 (#77 ,<+=( k4 (#CC%P!228< 5 5 *, {D ) %% &! +$--a0J*e,??m3&*O d ""1+K+u-##}4#.L#))-@@#  rrreturnNcl|D] } |jdk(r|jcS"y#t$rY/wxYw)zExtracts the first control word from a .rtf group. Args: children: A list of child objects within a .rtf group Returns: The first controlword found in a group. Returns None if no controls words are found. rN)r!r"AttributeError)ris r.r z.StripNonVisibleRTFGroups.get_first_controlwordZsK A 66]*77N+  "  s ' 33) __name__ __module__ __qualname____doc__r r r/ staticmethodrrstrr rr.rr sR 3$33js4xrrcLeZdZdZdedefdZdedefdZdedefdZ dedefd Z d edefd Z dedefd Z dedefd Z dedefdZdefdZdedefdZdedefdZdedefdZdedefdZdedefdZdedefdZdedefdZdedefdZdedefdZdedefdZdedefdZy) RTFCleanerzVisits each Token in provided RTF Trees. Converts all tokens that need converting. Deletes all tokens that shouldn't be visible. And, joins all strings that are left into one final string. argsr0c$dj|S)zJoins the .rtf object's string representations together at highest level object `start`. This is the final string combination. rjoinr#r=s r.startzRTFCleaner.startpsxx~rstringc4|j |jSy)z*Convert a string object into a raw string.rr"r#rCs r.STRINGzRTFCleaner.STRINGvs << #<< rc|jSNrErFs r. SPACE_SAVEzRTFCleaner.SPACE_SAVE|s ||rstringsc$dj|S)zGConvert all string objects withing a string group into a single string.rr?r#rKs r.rCzRTFCleaner.stringsxx  rgrpcg}|D]-}t|ttr|j|/dj |S)z&Join the strings in all group objects.r)rr!r appendr@)r#rN _new_childrenr3s r.r/zRTFCleaner.groupsF  (A!T']+$$Q'  ( xx &&rcR|Dcgc]}|| }}dj|Scc}w)z_Join the all the strings in an .rtf object into a single string representation of the document.rr?)r#r=r3s r.documentzRTFCleaner.documents,1a1=11xx~2s$$cy)zDelete all open parens.rr:rAs r. OPENPARENzRTFCleaner.OPENPARENrcy)zDelete all closed parens.rr:rAs r. CLOSEPARENzRTFCleaner.CLOSEPARENrVrctS)zProcess MHTMLTAG groups Currently discarding because they don't need to be processed. Returns: Always returns a discard object.r rAs r.rzRTFCleaner.mhtmltag_groups rc$dj|S)zYHTMLTAG processing. Takes any string values within an HTMLTAG and returns them. rr?rMs r.rzRTFCleaner.htmltag_groups xx  rhtmltagcy)zDelete all HTMLTAG objectsrr:)r#r\s r.HTMLTAGzRTFCleaner.HTMLTAGrVrcharcy)zDelete all star escape objectsrr:)r#r_s r.rzRTFCleaner.STAR_ESCAPEsrsymbolsc$dj|S)z7Join all visible symbols from in control symbol groups.rr?)r#ras r.control_symbolzRTFCleaner.control_symbolsxx  rc"djS)z8Convert non-breaking spaces into visible representation. encoderAs r.NONBREAKING_SPACEzRTFCleaner.NONBREAKING_SPACE!!rc"djS)z9Convert non-breaking hyphens into visible representation.­rfrAs r.NONBREAKING_HYPHENzRTFCleaner.NONBREAKING_HYPHENrirc"djS)z8Convert hyphen control char into visible representation.u‧rfrAs r.OPTIONAL_HYPHENzRTFCleaner.OPTIONAL_HYPHENrircy)aConvert a formula character into an empty string. If we are attempting to represent formula characters the scope for this library has grown too inclusive. This was only used by Word 5.1 for the Macintosh as the beginning delimiter for a string of formula typesetting commands.rr:rAs r.FORMULA_CHARACTERzRTFCleaner.FORMULA_CHARACTERrcy)zProcess index subentry items Discard index sub-entries. Because, we don't care about indexes when de-encapsulating at this time.rr:rAs r.INDEX_SUBENTRYzRTFCleaner.INDEX_SUBENTRYrqrcPdddd}|j|jd}||Sy)zzConvert encoded chars which are mis-categorized as control symbols into their respective chars. Delete all the other ones.{}\)s\{s\}s\\Nr)getr")r#r=ra replacements r. CONTROLSYMBOLzRTFCleaner.CONTROLSYMBOLs9  kk$**d3  " rc 0ddddjddjddjddjddjdd jdd jdd }|j|jd }||Sy )zConvert encoded chars which are mis-categorized as control words into their respective chars. Delete all the other ones.   u‘zutf-8u’u“u”u•u–u—) s\pars\tabs\lines\lquotes\rquotes \ldblquotes \rdblquotes\bullets\endashs\emdashNr)rgrxr")r#r=wordsrys r.rzRTFCleaner.CONTROLWORDs!1!1$OOG4$OOG4!1!1!1  ii D1  " rN)r4r5r6r7rbytesrBr rGrJrCr/rSrUrXrrr^rrcrhrlrnrprsrzrr:rr.r<r<ls|$5 Uu 5!d!u! ''%'Te eu4!T!e!u% !d!u!"e"" "u"" "E"e" e 5U % E %rr<r current_stater0c#$K|dg}n|g}|jD]t}d}t|trt||dD]}|.t |}||j ||Q|ddusYt ddur t||vyw)aGet a list of Tokens which should be suppressed by HTMLRTF control words. NOTE: This de-encapsulation supports the HTMLRTF control word within nested groups. The state of the HTMLRTF control word transfers when entering groups and is restored when exiting groups, as specified in [MSFT-RTF]. Returns: A list of Tokens which should be suppressed by HTMLRTF control words. NFTRTFDE.HTMLRTF_Stripping_logger)rrr get_stripped_HTMLRTF_valuestoggle_htmlrtfrPrr )rr htmlrtf_stackchild is_htmlrtftoggles r.rrs &  eT "5e]2=NO   (.J%$$Z0 r"d* @ATI)%0 s A.B1Brc&t|tr|jdk(rr|jj j }t |dkDr"|ddk(rtddur td|ytddur td |yy ) zIdentify if htmlrtf is being turned on or off. Returns: Bool representing if htmlrtf is being enabled or disabled. None if object is not an HTMLRTF token. HTMLRTFrr0z$RTFDE.HTMLRTF_Stripping_state_loggerTOFFFONN) rr r!r"decodestriprrr)r htmlrtfstrs r.rrs % :: "++-335JJ!# 2#(= FG4O%e4BCtK!B/ rc:eZdZdZdeeffd ZdefdZxZS)DeleteTokensFromTreeaSRemoves a series of tokens from a Tree. Parameters: tokens_to_delete: A list of tokens to delete from the Tree object. (sets self.to_delete) Attributes: to_delete: A list of tokens to delete from the Tree object. delete_start_pos: The starting position for all the identified tokens. Used to identify which tokens to delete. tokens_to_deletect|||_|jDchc]}|jc}|_ycc}w)zSetup attributes including token start_pos tracking. Args: tokens_to_delete: A list of tokens to delete from the Tree object. (sets self.to_delete) N)super__init__ to_delete start_posdelete_start_pos)r#rr3 __class__s r.rzDeleteTokensFromTree.__init__/s5 )6:nn E E EsAtokenctddur | t|t|tr|j|j vrt |jD]\}}|j|jk(s |j|jk(s:|j|jk(sTtddur t||jj|tcS|S)zDiscard any identified tokens. Args: token: All tokens within the transformed tree. Returns: Returns all non-identified tokens. Returns Discard objects for any identified tokens. z&RTFDE.HTMLRTF_Stripping_logger_VERBOSETr) rrrr rr enumeraterend_posr"r popr )r#renr3s r.__default_token__z&DeleteTokensFromTree.__default_token__9s @ AT I +E2 eU #$"7"77&t~~6'EB u6 U]]25;;.'(HITQ1%8**2.&' r) r4r5r6r7rr rr __classcell__)rs@r.rr$s&FeFurrc4eZdZdZdefdZdefdZdefdZy)StripUnusedSpecialCharactersaStrip all unused tokens which lark has extracted from the RTF. These tokens are largely artifacts of the RTF format. We have to do this because we use the "keep_all_tokens" option in our lark parser. It's better to be explicit then to allow for ambiguity because of the grammar. rctSzERemove RTF braces. Returns: Always returns a discard object.rZr#rs r._LBRACEz$StripUnusedSpecialCharacters._LBRACEY rctSrrZrs r._RBRACEz$StripUnusedSpecialCharacters._RBRACE`rrctS)zRemove spaces which are not a part of the content These are mostly spaces used to separate control words from the content they precede. Returns: Always returns a discard object. rZrs r. _SPACE_DELETEz*StripUnusedSpecialCharacters._SPACE_DELETEgs rN)r4r5r6r7r rrrr:rr.rrQs+UU5rrceZdZdZdefdZy)StripControlWordszGVisits each control word and strips the whitespace from around it. rcZ|j|jj}|S)zStrips the whitespace from around a provided control word. Args: token: A CONTROLWORD token to strip whitespace from. rE)updater"r)r#rtoks r.rzStripControlWords.CONTROLWORDvs& ll!2!2!4l5 rN)r4r5r6r7r rr:rr.rrrsrrraw_rtfcRg}d}tj||D]d}t|d}|jd}||||z|j |jd||z|d}|j |fd}d}|D]} |||| dz }| d}|||d z }||fS) aExtracts binary objects from a rtf file. Parameters: raw_rtf: (bytes): It's the raw RTF file as bytes. Returns: A tuple containing (new_raw, found_bytes) new_raw: (bytes) A bytes object where any binary data has been removed. found_bytes: (list) List of dictionaries containing binary data extracted from the rtf file. Each dictionary includes the data extracted, where it was extracted from in the original rtf file and where it can be inserted back into the stripped output. Description of found_bytes dictionaries: "bytes": (bytes) The binary data contained which was extracted. "ctrl_char": (tuple) Tuple containing the binary control word and its numeric parameter "start_pos": (int) The position (in the original raw rtf data) where the binary control word started. "bin_start_pos": (int) The position (in the original raw rtf data) where the binary data starts. "end_pos": (int) The position (in the original raw rtf data) where the binary data ends. Here is an example of what this looks like (by displaying the printable representation so you can see the bytes and then splitting the dict keys on new lines to make it readable.) >> print(repr(found_bytes)) "{'bytes': b'\xf4UP\x13\xdb\xe4\xe6CO\xa8\x16\x10\x8b\n\xfbA\x9d\xc5\xd1C', 'ctrl_char': (b'\\bin', b'20'), 'start_pos': 56, 'end_pos': 83, 'bin_start_pos': 63}" s(\\bin)([0-9]+)[ ]?rr)r ctrl_charrr bin_start_posrrrN)refinditerintspangroupsrP) r found_bytes byte_finder matchitemparamrbyte_objnew_raw start_buffer new_bytess r.strip_binary_objectsrs8K)K[[g6 % IaL!!(, $]=3FG!*!1!1!3!*!1!!4,U2%2  8$ %GL , 7< +(>?? + , w|}%%G [ !!rcP|dk(r |jdjd}|S)Nhtmlres )replacerg)content content_types r.transform_based_on_content_typers(v//)"2"2"4i@ NrrI)%typingrrrrcollections.abcr lark.visitorsrr r lark.treer lark.lexerr r RTFDE.utilsr rrrlogging getLoggerlogrr<boolrr9rrrrrtuplerrr:rr.rs &%%%) kkg J{JXGGRd5d;KW`:%c *uT$Y/?$+;+Z;B  /"%/"E/"dr