tag, if the del looks like (DEL_START,
'Text
', DEL_END)
r r usr+ r insdelzUnexpected delete tag: uerk r z tag that was not in the original document.
If cleanup is true, make sure there's no or , and get
rid of any and tags.
T)
create_parent) cleanup_htmlr )rx r s r r r y s D! t4 88r
z z
zc t | }|r| |j d } t | }|r| d|j } t d| } | S )z This 'cleans' the HTML, meaning that any page structure is removed
(only the contents of are used, if there is any and tags are removed. Nr< ) _search_bodyend_search_end_bodystart_replace_ins_del)rx matchs r r r sS
E EIIKL! T "E NU[[]# B %D Kr
c H t | j }| d| | |d fS )zP
This function takes a word, such as 'test
' and returns ('test','
')
r N)rd rstrip)wordstripped_lengths r split_trailing_whitespacer s. $++-(O / "D )9$: ::r
c
g }d}g }| D ]- }t |t rq|d dk( r:|d }t |d \ }}t d|||| }g }|j | n.|d dk( r&|d }t ||d }g }|j | t
| r0t | \ }}t ||| }g }|j | t | r|j | t | rF|r|j | |sJ d
|d |d |d
| |j j | .J |st d| gS |d j j | |S )zM
This function takes a list of chunks and produces a list of tokens.
Nr imgr r, )r rl rr hrefr )rl rr zWeird state, cur_word=z , result=z , chunks=z of r< )rl rk )
isinstancetupler r ro r is_wordrt is_start_tag
is_end_tagrm r ) r tag_accumcur_wordrI r rf r rr r s r r r s IH
F $ eU #Qx5 Ah+DU1X+N(($UC3.79L N
h'qV# Qx%dYTWX
h' 5>)B5)I &E& UYL_`HI MM( #
%
U #
' 9 89x ""))%0 5I$L b9-..r
##I. Mr
)address
blockquotecenterdirdivdlfieldsetformh1h2h3h4h5h6hrisindexmenunoframesnoscriptolppretableul)
dddtframesetlitbodytdtfootththeadtrc # b K |s<| j dk( r d| j d t | f n
t | | j t v r$| j st | s
| j syt | j }|D ] }t | | D ] }t || E d{ | j dk( r(| j d r|rd| j d f |s7t | t | j }|D ] }t | yy7 ww)a Takes an lxml element el, and generates all the text chunks for
that tag. Each start tag is a chunk, each word is a chunk, and each
end tag is a chunk.
If skip_tag is true, then the outermost container tag is
not returned (just its contents).r rf NrK rR r ) r get start_tagr r% rd tail split_wordsr6 r end_tag)elrL r start_wordsr child end_wordss r r r s
66U?"&&-2 7 7B- vvBGGCGBGG bgg&K $ B e=AAAB vv} Mrvvf~&& bk ( $D d# # $ Bs B3D/5D-6A8D/z \S+(?:\s+|$)c D | r| j sg S t | }|S )z_ Splits some text into words. Includes trailing whitespace
on each word when appropriate. )rB _find_words)r% wordss r r, r, # s# tzz| E Lr
z
^[ \t\n\r]c dj | j j D cg c] \ }}d| dt | d c}} }d| j | dS c c}}w )z=
The text representation of the start tag for a tag.
r< r z="r/ r+ r- )rA attribitemsr6 r )r. r r
attributess r r* r* . sj 99??, D% D6K&'q) J rvvhzl! $$ s A
c ` | j }|r
t | rdnd}d| j d| S )zg The text representation of an end tag for a tag. Includes
trailing whitespace when appropriate. r r< r? s r r r B s >>$ r
c L | j d xr | j d S )Nr+ r: r> r? s r r r E s" >># ;s~~d';#;;r
c P t | d }t | t |d } | S )z Given an html string, move any or tags inside of any
block-level elements, e.g. transform , etc.
TF)rO any_block_level_tag)r. s r rP rP p s% bgg*+ r
c z | j } | j t D ] }|| us nB || }| j |_ d| _ |j t | |g| dd yt | D ] }t
| rKt || |j s' || }|j |_ d|_ |j | Y || }| j || |j | | j r3 || }| j |_ d| _ | j d| yy)zt helper for _fixup_ins_del_tags; actually takes the etc tags
and moves them inside any block-level tags. Nr )
makeelementrO rT r% r rN rP rQ r+ addnextr3 ro r ) r. r rV block_level_el children_tagr0 tail_tag child_tagtext_tags r rQ rQ y s$ ..K!"''#67
#
#3' GG DH% 1b $ $U + !% -zz&s+ %
!
h'#C(I JJui ( U # $
ww s#
!X r
c | j }| j }| j }|r3t | s
|xs d|z }n | d j xs d|z | d _ |j | }|rC| j }||j xs d|z |_ n|j xs d|z |_ | j
|||dz y)z
Removes an element, but merges its contents into its place, e.g.,
given r s2
,%w'334 Z\ \B ) J
''i'' (9(9?Rx?R?RST?UV c U _b & *$ #1 &#P -1
#"&8%P# $ < S S $ 4 4 4 $ H4 HV 2t 2 2j C B 8" " 2 9 rzz,RTT 299 2::mRTT"$$Y7>> 2:: 12449=AA ;2l V^^It
7
!6>>)Y 8 . 6 ,V^^Iy B 8 %fnnUE&1133 -4
$6 bjj "$$/77 #
=177 %!# <& F-0 & z&L K : : ]F" , ??+,s" M: 4N :NNN65N6
word
toword
Fr T) skip_outer)r _fixup_ins_del_tagsserialize_html_fragment)rx rE s r r r H s) T5 )C "34 8D Kr c t | t r J d| t j | dd }|r;||j d dz d }|d|j d }|j S |S ) z Serialize a single lxml element as HTML. The serialized form includes the elements tail. If skip_outer is true, then don't serialize the outermost tag z1You should pass in an element, not a string like rx unicode)methodencodingr- r Nr+ )r r r tostringfindrfindrB )r. rC rx s r rE rE Q s| "c" D ;B6BD " >>"Vi @DDIIcN1$%&$TZZ_%zz| r c t | j dd D ]5 }t | st ||j |j 7 y)z?fixup_ins_del_tags that works on an lxml document in-place r r )r N)listiter_contains_block_level_tag_move_el_inside_blockr drop_tag)rE r. s r rD rD d sE 388E5)* (, bbff- r c 6 | j t D ] } y y)zPTrue if the element contains any block-level elements, like,
Hi there!
, if you remove the element you getHi there!
r< rk Nr ) getparentr% r+ rd r getprevious getchildren)r. parentr% r+ r previouss r _merge_element_contentsrc s \\^F 77D 77D 2wJB$&D b6;;,"4BrFK LL E >># !;;,"4FK %]] 0bD 8HM NN ,F5qr c > e Zd ZdZdZej defd Zy)rU zt Acts like SequenceMatcher, but tries not to find very small equal blocks amidst large spans of changes r, r' c t t | j t | j }| j }t ||dz }t j | }|D cg c] }|d |kD s|d s| c}S c c}w )Nr0 r, )minrd rS thresholdr get_matching_blocks)r sizerg actualr s r rh z.InsensitiveSequenceMatcher.get_matching_blocks sy "%c$&&k3tvv;"?'+~~ 419- 44T:!' 7Y&Aw s 'B N) r r r r rg cythonr rN rh r r r rU rU s+ I \\ T r rU __main__) _diffcommand))z&z<z>z"z')F)T)brk ImportErrorr r< r difflibinspect isfunctionget_close_matches"cython.cimports.lxml.html._difflibr r itertools functoolsoperatorrelxmlr lxml.htmlr r! __all__partialgroupby itemgetterr r r r r6 rx r7 r: r" r= r> rW r? rN rn r@ r# r{ r r r r r r r r rt r r rM r compileISsearchr r subr r r r r frozensetr block_level_tagsblock_level_container_tagssortedrT r Ufindallr3 r, r r; r* r- r r r r rE rD rP rQ rc finalr rU r rm mainr r r