MpjO<ddlmZmZmZmZmZddlmZddlm Z ddl m Z ddl m Z ddlmZddlmZmZddlmZdd lmZdd lmZdd lmZdd lmZmZmZdd lmZmZmZddlmZddl m!Z!ddl m"Z"ddl#m$Z$m%Z%m&Z&ddl'Z'e'jPdZ)GddZ*y))UnionAnyStrTupleDictAny)BufferedReader)Lark)Tree)Token)UnexpectedInput) RTFCleanerStripControlWords)StripNonVisibleRTFGroups)StripUnusedSpecialCharacters)transform_based_on_content_type)encode_escaped_control_chars)log_validatorslog_transformations is_logger_on)get_stripped_HTMLRTF_valuesDeleteTokensFromTreestrip_binary_objects)make_concise_grammar) TextDecoder)validate_ansi_cpg)NotEncapsulatedRtfMalformedEncapsulatedRtf MalformedRtfNRTFDEc eZdZdZddedeedffdZdZdde defd Z d Z d Z d Z d ZdeedffdZdefdZdefdZdefdZdZedededeeedefffdZedefdZedefdZy)DeEncapsulatoraDe-Encapsulating RTF converter of HTML/TEXT found in .msg files. De-encapsulation enables previously encapsulated HTML and plain text content to be extracted and rendered as HTML and plain text instead of the encapsulating RTF content. After de-encapsulation, the HTML and plain text should differ only minimally from the original HTML or plain text content. Parameters: raw_rtf: (bytes): It's the raw RTF file as bytes. grammar: (str): OPTIONAL - Lark parsing grammar which defines the RTF language. https://github.com/lark-parser/lark If you think my grammar is shoddy this is your chance to test out a better one and make a pull request. Attributes: content: (bytes) The deencapsulated content no matter what format it is in. Populated by the `deencapsulate` function. html: (bytes) The deencapsulated content IF it is HTML content. Populated by the `set_content` function. text: (bytes) The deencapsulated content IF it is plain text content. Populated by the `set_content` function. found_binary: List of dictionaries containing binary data extracted from the rtf file. content_type: The type of content encapsulated in .rtf data (html or text). Populated by the `get_content_type` function. full_tree: The full .rtf object parsed into an object Tree using the grammar. Populated by the `parse_rtf` function. doc_tree: The `document` portion of the .rtf full_tree object. raw_rtf: The raw encapsulated .rtf data in byte format. grammar: The Lark parsing grammer used to parse the .rtf data. content_type_token: The .rtf header token identifying the content type. (\fromhtml1 OR \fromtext) parser: The lark parser. Should not need to be manipulated directly. But, useful for debugging and saving the parsed object. Nraw_rtfgrammarc0||||||||||j|t|tr|}n td|j d}|j dd}|j dd}||_|||_yt|_y)a[Load in the Encapsulated test and setup the grammar used to parse the encapsulated RTF. NOTE: This does not do the parsing in the init so that you can initiate the object and do the parsing step by step. Parameters: raw_rtf: (bytes): It's the raw RTF string. grammar: (str): OPTIONAL - Lark parsing grammar which defines the RTF language. https://github.com/lark-parser/lark If you think my grammar is shoddy this is your chance to test out a better one and make a pull request. Raises: TypeError: The raw_rtf data passed is not the correct type of data (string/byte string). zGDeEncapssulator only accepts RTF files in string or byte-string formatss   N) catch_common_validation_issues isinstancebytes TypeErrorrstripreplacer"r#r)selfr"r# raw_rtf_bytess a/mnt/ssd/data/Dropbox/adrian/scripts/msg_venv/lib/python3.12/site-packages/RTFDE/deencapsulate.py__init__zDeEncapsulator.__init__@s          ++G4 gu %#Mef f%,,W5 %--ge< %--eE: +   'DL/1DLct|j}|d}|d}t|dkDr||_tj dt |}tddurtdt| |j|t}|j|j|j|j!|j#}t%j'|}t)} | j'|} t+} | j'| } t-d } | j'| }||_|j1y#t$r}td|d}~wwxYw) aDe-encapsulate the RTF content loaded into the De-Encapsulator. Once you have loaded in the raw rtf this function will set the properties containing the encapsulated content. The `content` property will store the content no matter what format it is in. The `html` and `text` properties will be populated based on the type of content that is extracted. (self.html will be populated if it is html and self.text if it is plain text.) rz.Binary data found and extracted from rtf file.RTFDE.transform_loggerTzEncoding Escaped Rtfz&Malformed encapsulated RTF discovered:N) visit_tokens)rr"len found_binaryloginforrr parse_rtfr rrupdate_children full_tree get_doc_treevalidate_encapsulationstrip_htmlrtf_tokensr transformrrr content set_content)r. stripped_datanon_binary_rtfr8 escaped_rtf_eDecoderhtmlrtf_strippedcontrol_strippedspecial_strippernon_special_treestripper stripped_treecleaner cleaned_texts r0 deencapsulatezDeEncapsulator.deencapsulateds` -T\\: &q)$Q' | q ,D  HHE F2>B 0 1T 9  6 7  , ^ NN; '-/  ##% 446,.889IJ79+556FG+- **+;< $/((7 #  - ^*-SU[] ] ^s9E## E=, E88E=fallback_to_defaultreturnc|j}|D]}|jdvs|cStjd|dur t dtj dtj dtjdy) a\Validate and return the RTF charset keyword from the RTF streams header. Args: fallback_to_default (bool): Allows you to force the use of the default charset "\ansi" if one is not found. Raises: MalformedRtf: RTF stream does not include charset control word. Returns: The RTF charset keyword from the RTF streams header. )\ansis\macs\pcs\pcazAcceptable charset not found as the second token in the RTF stream. The control word for the character set must precede any plain text or any table control words. So, if this stream doesn't have one it is malformed or corrupted.Fz1RTF stream does not include charset control word.zThe fallback_to_default option on _get_charset is considered DANGEROUS if used on possibly malicious samples. Make sure you know what you are doing before using it.zAttempting to decode RTF using the default charset ansi. This is not recommended and could have unforeseen consequences for the resulting file and your systems security.zYou have a malformed RTF stream. Are you sure you really want to be parsing it? It might not just be corrupted. It could be maliciously constructed.rU)+get_header_control_words_before_first_groupvaluer9debugrwarningr:)r.rR main_headerstokens r0validate_charsetzDeEncapsulator.validate_charsetsGGI ! E{{FF   y z % 'RS S { | } ~ i jr2c|j|_|jdk(r&t|j|j|_y|j|_y)zlPopulate the html or text content based on the content type. Populates self.html and/or self.text variables.htmlN)get_content_type content_typerrBr^textr.s r0rCzDeEncapsulator.set_contentsD 113    &7 dFWFWXDI DIr2c|jjdjdk(r|jjd|_yt d)zExtract the document portion of the .rtf full_tree object. Populates the classes doc_tree attribute. Raises: ValueError: The .rtf document object is missing or mis-located in the .rtf's full_tree object. r4documentz1Document object in the wrong place after parsing.N)r=childrendatadoc_tree ValueErrorrbs r0r>zDeEncapsulator.get_doc_treesC >> " "1 % * *j 8 NN33A6DMPQ Qr2c|j|jtd|jdk(ry|jdk(rytd)ahProvide the type of content encapsulated in RTF. NOTE: This function will only work after the header validation has completed. Header validation also extracts the content type of the encapsulated data. Raises: NotEncapsulatedRtf: The .rtf object is missing an encapsulated content type header. Which means that it is likely just a regular .rtf file. \fromhtml1r^ \fromtextrazCData is missing encapsulated content type header (the FROM header).)content_type_tokenvalidate_FROM_in_doc_headerrrbs r0r_zDeEncapsulator.get_content_typesZ  " " *  , , . !!fgg  $ $ 6  $ $ 5 !fggr2c|j|j|j|j|j }|t |j yy)zhRuns simple tests to validate that the file in question is an rtf document which contains encapsulation.N)validate_rtf_doc_headerrgr\rmget_ansicpg_headerrrW)r.ansicpgs r0r?z%DeEncapsulator.validate_encapsulationsQ $$T]]3  ((*))+   gmm , r2cp|j}|D]!}|jjds|cSy)aExtract the ansicpg control word from the .rtf header. Returns: A lark CONTROLWORD Token with the `\ansicpg` value. Returns None if the `\ansicpg` control word is not included as this is only required if there is Unicode which needs to be converted to ANSI within a .rtf file. s\ansicpgN)rVrW startswith)r.headersitems r0rpz!DeEncapsulator.get_ansicpg_headers= BBD Dzz$$\2  r2rtfct|jdddd|_|jj||_t ddur!t dt |jyy)zParse RTF file's header and document and extract the objects within the RTF into a Tree. Populates the self.full_tree attribute. Args: rtf: The .rtf string to parse with the projects lark grammar. lalrT)parserkeep_all_tokens use_bytespropagate_positionsr5z Parsed RTFN)r r#ryparser=rr)r.rvs r0r;zDeEncapsulator.parse_rtfsa4<<"(+/%)/3 5 **3/ 0 1T 9  -  / :r2ct|j}t|}t|}|j |j}|S)zStrip tokens from with htmlrtf regions of the doc_tree as they were not part of the original HTML content. Returns: .rtf doc_tree stripped of all non-original tokens. )rrglistrrA)r.delete_generatortokens_to_deletedeleterhtmlrtf_cleaned_trees r0r@z#DeEncapsulator.strip_htmlrtf_tokenssE7t}}E 01&'78&00?##r2cg}|jjddD]'}t|tr|j |%|cS|S)a-Extracts all the control words in the first 20 tokens of the document or all the tokens which occur before the first group (whichever comes first.) This is used to extract initial header values for validation functions. Returns: A list containing the header tokens in the .rtf data. N)rgrer)r append)r.initial_control_wordsr[s r0rVz:DeEncapsulator.get_header_control_words_before_first_groupsP!#]]++CR0 -E%'%,,U3,,  - %$r2cddddd}tj|j}|jdd}g}d}|D]B}t |t r|j |%|t|jdz }Dtddurtd||D];}|j|| \}}|d dur|d dur td |5||_ =|d dur tjd t!dy)aInspect the header to identify what type of content (html/plain text) is encapsulated within the document. NOTE: The de-encapsulating RTF reader inspects no more than the first 10 RTF tokens (that is, begin group marks and control words) in the input RTF document, in sequence, starting from the beginning of the RTF document. If one of the control words is the FROMHTML control word, the de-encapsulating RTF reader will conclude that the RTF document contains an encapsulated HTML document and stop further inspection. If one of the control words is the FROMTEXT control word, the de-encapsulating RTF reader concludes that the RTF document was produced from a plain text document and stops further inspection. - MS-OXRTFEX Raises: MalformedEncapsulatedRtf: The .rtf headers are malformed. NotEncapsulatedRtf: The .rtf object is missing an encapsulated content type header. Which means that it is likely just a regular .rtf file. F)rtf1fromfonttbl malformedN c |jdk(S)N CONTROLWORD)type)ts r0z.+sQVV}E\r2RTFDE.validation_loggerTzHeader tokens being evaluated: )r[cw_foundrrzRTF file looks like is was supposed to be encapsulated HTML/TEXT but the headers are malformed. Turn on debugging to see specific informationzpFROMHTML/TEXT control word not found in first 10 RTF tokens. This is not an HTML/TEXT encapsulated RTF document.z%FROMHTML/TEXT control word not found.)rrArgrer)r rr scan_valuesrrcheck_from_tokenrrlr9rXr)r.r decoded_treefirst_ten_tokensoperating_tokens found_tokenr[s r0rmz*DeEncapsulator.validate_FROM_in_doc_headersM! # %' )*44T]]C '00"5 % _E%' ''. D):):;\)]$^^  _ 1 2d : <=M, "from":, "fonttbl":, "malformed":}` Returns: cw_found: Updated state dictionary found_token: The content_type_token found in the header. rjrk\rtf1Ns\fonttblrrTrzVMultiple FROM HTML/TXT tokens found in the header. This encapsulated RTF is malformed.rz^FROMHTML/TEXT control word found before rtf1 control word. That's not allowed in the RTF spec.rz\fonttbl code word found before FROMTML/TEXT was defined. This is not allowed for encapsulated HTML/TEXT. So... this is not encapsulated HTML/TEXT or it was badly encapsulated.)rrWstripr9rX)r[rfrom_cwsrtf1_cwr fonttbl_cws r0rzDeEncapsulator.check_from_token;sC #M2 ! :: &{{  "h.F#t+,0H[)IIvwF#t+'+HV$"'++K$$II~'+HV$,0H[)$$""$/#' $$ ""$ 2&*#F#4/IIRS,0H[)$$r2rgc|jdj}|dk7r;tjdt ddurt d|t dy) aCheck if doc starts with a valid RTF header `\rtf1`. "Before the de-encapsulating RTF reader tries to recognize the encapsulation, the reader SHOULD ensure that the document has a valid RTF document heading according to [MSFT-RTF] (that is, it starts with the character sequence "{\rtf1")." - MS-OXRTFEX Raises: MalformedRtf: The .rtf headers do not include \rtf1. rrz_RTF stream does not contain valid valid RTF document heading. The file must start with "{\rtf1"rTz(First child object in document tree is: z%RTF stream does not start with {\rtf1N)rerWr9rXrrr)rg first_tokens r0roz&DeEncapsulator.validate_rtf_doc_headerfs^''*00 ) # IIz {56$>!I+YZGH H $r2ct|tr td| td|dddk(r td|dvr tdy) afChecks for likely common valid input mistakes that may occur when folks try to use this library and raises exceptions to try and help identify them. Args: raw_rtf: A raw .rtf string or byte-string. Raises: TypeError: The data passed is the wrong type of data. MalformedRtf: The data passed is not a correctly formatted .rtf string. zFData passed as file pointer. DeEncapsulator only accepts byte objects.Nzr_r?r rpr;r r@rrVrm staticmethoddictrrrrorr(r2r0r!r!(s ."2u"2uSX"2H'R456% Rh"- E%*$5 0S0* $d $%T%"$NL'%u'%t'%d5c?>R8S'%'%T I$ I IRRRr2r!)+typingrrrrriorlarkr lark.treer lark.lexerr lark.exceptionsr RTFDE.transformersr rrrr RTFDE.utilsrrrrrrr RTFDE.grammarrRTFDE.text_extractionrrRTFDE.exceptionsrrrlogging getLoggerr9r!rr2r0rsl32+<7;>4IIff.-3XWg `R`Rr2