'VfizdZddlmZddlmZddlZddlZddlmZddlm Z m Z m Z ddl Z ddl mZddlmZdd lmZmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddl m!Z!ddl"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-ddl.m/Z/ddl0m1Z1e rddl2m3Z3m4Z4ddl5m6Z6m7Z7m8Z8m9Z9m:Z:m;Z;ddlme>fdbdZ?dcd#Z@ddd+ZAGd,d-ZBGd.d/eBZCded0ZDd1d2iZEGd3d4eBZFdfd6ZGd7ZHeFeFeCeCd8ZIdgd<ZJded=ZKd>ZLd?ZMee%d(@dAddddddBdCddDdddEdEdejNddFdhdaZOdS)izW :mod:`pandas.io.html` is a module containing functionality for dealing with HTML IO. ) annotations)abcN)Pattern) TYPE_CHECKINGLiteralcast)lib)import_optional_dependency)AbstractMethodErrorEmptyDataError)doc)find_stack_level)check_dtype_backend) is_list_likeisna)Index) MultiIndex)Series) _shared_docs) file_exists get_handle is_file_like is_fsspec_urlis_urlstringify_pathvalidate_header_arg pprint_thing) TextParser)IterableSequence) BaseBuffer DtypeBackendFilePath HTMLFlavors ReadBufferStorageOptions) DataFramez[\r\n]+|\s{2,}sstrregexrreturncR|d|S)a Replace extra whitespace inside of a string with a single space. Parameters ---------- s : str or unicode The string from which to remove extra whitespace. regex : re.Pattern The regular expression to use to remove extra whitespace. Returns ------- subd : str or unicode `s` with all extra whitespace replaced with a single space.  )substrip)r*r,s =c:\PYTHON\DbComparer\venv\Lib\site-packages\pandas/io/html.py_remove_whitespacer3Is 99S!'')) $ $$skiprows"int | Sequence[int] | slice | Noneint | Sequence[int]cXt|tr5|jpd|jpd}}t t ||j|St|tjst|rtd|S|dStt|j d)a Get an iterator given an integer, slice or container. Parameters ---------- skiprows : int, slice, container The iterator to use to skip rows; can also be a slice. Raises ------ TypeError * If `skiprows` is not a slice, integer, or Container Returns ------- it : iterable A proper iterator to use to skip rows of a DataFrame. rr7Nz& is not a valid type for skipping rows) isinstanceslicestartsteplistrangestopnumbersIntegralrr TypeErrortype__name__)r5r<r=s r2 _get_skiprowsrF\s&(E""n)8=+=AtE%55666 Hg. / /<3I3I)8444  q tH~~.VVV W WWr4objFilePath | BaseBufferencoding str | Nonestorage_optionsStorageOptions | None str | bytesct|s4t|ds$t|trTt |rEt |d||5}|j}dddn #1swxYwYnDt|ttfr|}n%tdt|j d|S)z Try to read from a url, file or string. Parameters ---------- obj : str, unicode, path object, or file-like object Returns ------- raw_text : str readr)rIrKNzCannot read object of type '') rhasattrr:r+rrhandlerObytesrCrDrE)rGrIrKhandlestexts r2_readrWys$ s  N 3   N sC  N&1%5%5 N  x    ) >&&((D ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) C#u & &NLtCyy7ILLLMMM KsA==BBczeZdZdZ d&d'dZdZdZd(dZdZdZ dZ dZ dZ dZ d)dZdZd Zd*d#Zd+d%ZdS),_HtmlFrameParseraZ Base class for parsers that parse HTML into DataFrames. Parameters ---------- io : str or file-like This can be either a string of raw HTML, a valid URL using the HTTP, FTP, or FILE protocols or a file-like object. match : str or regex The text to match in the document. attrs : dict List of HTML element attributes to match. encoding : str Encoding to be used by parser displayed_only : bool Whether or not items with "display:none" should be ignored extract_links : {None, "all", "header", "body", "footer"} Table elements in the specified section(s) with tags will have their href extracted. .. versionadded:: 1.5.0 Attributes ---------- io : str or file-like raw HTML, URL, or file-like object match : regex The text to match in the raw HTML attrs : dict-like A dictionary of valid table attributes to use to search for table elements. encoding : str Encoding to be used by parser displayed_only : bool Whether or not items with "display:none" should be ignored extract_links : {None, "all", "header", "body", "footer"} Table elements in the specified section(s) with tags will have their href extracted. .. versionadded:: 1.5.0 Notes ----- To subclass this class effectively you must override the following methods: * :func:`_build_doc` * :func:`_attr_getter` * :func:`_href_getter` * :func:`_text_getter` * :func:`_parse_td` * :func:`_parse_thead_tr` * :func:`_parse_tbody_tr` * :func:`_parse_tfoot_tr` * :func:`_parse_tables` * :func:`_equals_tag` See each method's respective documentation for details on their functionality. Nio.FilePath | ReadBuffer[str] | ReadBuffer[bytes]match str | Patternattrsdict[str, str] | NonerIr+displayed_onlybool extract_links0Literal[None, 'header', 'footer', 'body', 'all']rKr(r-Nonech||_||_||_||_||_||_||_dSN)rZr\r^rIr`rbrK)selfrZr\r^rIr`rbrKs r2__init__z_HtmlFrameParser.__init__s>    ,*.r4cjj}fd|DS)z Parse and return all tables from the DOM. Returns ------- list of parsed (header, body, footer) tuples from tables. c3BK|]}|VdSrf)_parse_thead_tbody_tfoot).0tablergs r2 z0_HtmlFrameParser.parse_tables..s1II--e44IIIIIIr4) _parse_tables _build_docr\r^)rgtabless` r2 parse_tablesz_HtmlFrameParser.parse_tablessD##DOO$5$5tz4:NNIIII&IIIIr4c,||S)aC Return the attribute value of an individual DOM node. Parameters ---------- obj : node-like A DOM node. attr : str or unicode The attribute, such as "colspan" Returns ------- str or unicode The attribute value. )get)rgrGattrs r2 _attr_getterz_HtmlFrameParser._attr_getters$wwt}}r4rJc t|)a Return a href if the DOM node contains a child or None. Parameters ---------- obj : node-like A DOM node. Returns ------- href : str or unicode The href from the child of the DOM node. r rgrGs r2 _href_getterz_HtmlFrameParser._href_getter"$'''r4c t|)a Return the text of an individual DOM node. Parameters ---------- obj : node-like A DOM node. Returns ------- text : str or unicode The text from an individual DOM node. rxrys r2 _text_getterz_HtmlFrameParser._text_getter r{r4c t|)a Return the td elements from a row element. Parameters ---------- obj : node-like A DOM node. Returns ------- list of node-like These are the elements of each row, i.e., the columns. rxrys r2 _parse_tdz_HtmlFrameParser._parse_td0r{r4c t|)a; Return the list of thead row elements from the parsed table element. Parameters ---------- table : a table element that contains zero or more thead elements. Returns ------- list of node-like These are the row elements of a table. rxrgrms r2_parse_thead_trz _HtmlFrameParser._parse_thead_tr@"$'''r4c t|)a Return the list of tbody row elements from the parsed table element. HTML5 table bodies consist of either 0 or more elements (which only contain elements) or 0 or more elements. This method checks for both structures. Parameters ---------- table : a table element that contains row elements. Returns ------- list of node-like These are the row elements of a table. rxrs r2_parse_tbody_trz _HtmlFrameParser._parse_tbody_trO""$'''r4c t|)a, Return the list of tfoot row elements from the parsed table element. Parameters ---------- table : a table element that contains row elements. Returns ------- list of node-like These are the row elements of a table. rxrs r2_parse_tfoot_trz _HtmlFrameParser._parse_tfoot_trbrr4c t|)ai Return all tables from the parsed DOM. Parameters ---------- document : the DOM from which to parse the table element. match : str or regular expression The text to search for in the DOM tree. attrs : dict A dictionary of table attributes that can be used to disambiguate multiple tables on a page. Raises ------ ValueError : `match` does not match any text in the document. Returns ------- list of node-like HTML
elements to be parsed into raw data. rx)rgdocumentr\r^s r2roz_HtmlFrameParser._parse_tablesqs0"$'''r4c t|)a? Return whether an individual DOM node matches a tag Parameters ---------- obj : node-like A DOM node. tag : str Tag name to be checked for equality. Returns ------- boolean Whether `obj`'s tag name is `tag` rxrgrGtags r2 _equals_tagz_HtmlFrameParser._equals_tagrr4c t|)z Return a tree-like object that can be used to iterate over the DOM. Returns ------- node-like The DOM from which to parse the table element. rx)rgs r2rpz_HtmlFrameParser._build_docs"$'''r4c|}|}|}fd}|sN|rL||dr;||d|r||d;|d}|d}|d}|||fS)a  Given a table, return parsed header, body, and foot. Parameters ---------- table_html : node-like Returns ------- tuple of (header, body, footer), each a list of list-of-text rows. Notes ----- Header and body are lists-of-lists. Top level list is a list of rows. Each row is a list of str text. Logic: Use , , elements to identify header, body, and footer, otherwise: - Put all rows into body - Move rows from top of body to header only if all elements inside row are s, return a list of text rows. Parameters ---------- rows : list of node-like List of s section : the section that the rows belong to (header, body or footer). Returns ------- list of list Each returned row is a list of str text, or tuple (text, link) if extract_links is not None. Notes ----- Any cell with ``rowspan`` or ``colspan`` will have its contents copied to subsequent cells. rr9rrowspancolspan) rrrr3r}rbrzintrvr?)rgrowsr all_texts remaindertrtextsnext_remainderindextdstdprev_i prev_text prev_rowspanrVhrefrr_s r2rz(_HtmlFrameParser._expand_colspan_rowspans.   % '% 'BENE..$$C   IaLOu$<$<6?mmA6F6F3FI|LL+++#a''&--vy,QRBR.STTTQJE  IaLOu$<$<*$*;*;B*?*?@@%%)999,,R00D $>C!DDd//I>>C!DDwALL&&&{{&--udGaK.HIIIQJEE 4= Q Q/ < Y'''!##"))69lQ>N*OPPP   U # # #&II 'NE3< Q Q/ < Y'''!##"))69lQ>N*OPPP   U # # #&I 'r4 attr_namec2|js|Sfd|DS)a Return list of tables, potentially removing hidden elements Parameters ---------- tbl_list : list of node-like Type of list elements will vary depending upon parser used attr_name : str Name of the accessor for retrieving HTML attributes Returns ------- list of node-like Return type matches `tbl_list` cg|]>}dt|ddddv<|?S) display:nonestyler/)getattrrtreplace)rlxrs r2 z:_HtmlFrameParser._handle_hidden_tables..;sh   1i((,,Wb99AA#rJJKK KKKr4)r`)rgtbl_listrs `r2_handle_hidden_tablesz&_HtmlFrameParser._handle_hidden_tables(s@ " O        r4rf)rZr[r\r]r^r_rIr+r`rarbrcrKr(r-rdr-rJr-ra)rr)rr+)rE __module__ __qualname____doc__rhrrrvrzr}rrrrrorrprkrrr4r2rYrYs.BBX+//////$ J J J((((( ((( ((( ( ( ((((& ( ( ((((4((((& ( ( (+$+$+$ZPPPPd      r4rYcReZdZdZdZddZdZddZd Zd Z d Z d Z d Z dZ dS)!_BeautifulSoupHtml5LibFrameParsera/ HTML to DataFrame parser that uses BeautifulSoup under the hood. See Also -------- pandas.io.html._HtmlFrameParser pandas.io.html._LxmlFrameParser Notes ----- Documentation strings for this class are in the base class :class:`pandas.io.html._HtmlFrameParser`. chd}|||}|stdg}t}||d}|D]}|jrk|dD]} | |t jdD]} | ||vr+||| || ||s$td t|j |S) Nrm)r^zNo tables foundr^rzdisplay:\s*none)r)stringz!No tables found matching pattern ) find_all ValueErrorsetrr` decomposerecompilefindraddreprpattern) rgrr\r^ element_namerqresult unique_tablesrmelems r2roz/_BeautifulSoupHtml5LibFrameParser._parse_tablesRsR ""g|]\}}d|dt|S)@=)r)rlkvs r2rz%_build_xpath_expr..s3BBBDAq'!''d1gg''BBBr4[])rjoinitems)r^r*s r2_build_xpath_exprrsY 58,,g BBEKKMMBBBCCA q888Or4rz$http://exslt.org/regular-expressionscLeZdZdZddZdZdZdZdd Zd Z d Z d Z d Z dS)_LxmlFrameParseraY HTML to DataFrame parser that uses lxml under the hood. Warning ------- This parser can only handle HTTP, FTP, and FILE urls. See Also -------- _HtmlFrameParser _BeautifulSoupLxmlFrameParser Notes ----- Documentation strings for this class are in the base class :class:`_HtmlFrameParser`. r-rJcD|d}|sdn|dS)Nz .//a/@hrefrxpath)rgrGrs r2rzz_LxmlFrameParser._href_getters&yy&&,ttT!W,r4c*|Srf) text_contentrys r2r}z_LxmlFrameParser._text_getters!!!r4c,|dS)N ./td|./thr rs r2rz_LxmlFrameParser._parse_tdsyy%%%r4c2|j}dt|d}|r|t|z }||t}||d}|jr|D]}|dD]}||dD]G}d|j dd  d d vr|H|std t||S) Nz//table[.//text()[re:test(., z)]]) namespacesattribz.//stylez .//*[@style]rrrr/zNo tables found matching regex ) rrrr  _re_namespacerr` drop_treerrtrr) rgrr\kwargsr xpath_exprrqrmrs r2roz_LxmlFrameParser._parse_tabless=-HT']]GGG   4 +F33 3J }EE++FH==   ) ) )"KK 33%%DNN$$$$!KK77))D%")E)E)M)McSU)V)VVV((() PNtG}}NNOO O r4rac|j|kSrf)rrs r2rz_LxmlFrameParser._equals_tagsw#~r4cddlm}ddlm}m}m}|d|j} t|jrGt|jd|j 5}||j |}d d d n #1swxYwYn||j|} | }n#t$rYnwxYwt|d s|d dddnl#tt f$rX}t|js8||j|} | }n#t$rYnwxYw|Yd }~nd }~wwxYw|d D]} d | jpdz| _|S)a Raises ------ ValueError * If a URL that lxml cannot parse is passed. Exception * Any other ``Exception`` thrown. For example, trying to parse a URL that is syntactically correct on a machine with no internet connection will fail. See Also -------- pandas.io.html._HtmlFrameParser._build_doc r)XMLSyntaxError) HTMLParser fromstringparseT)recoverrIrPrK)parserNrzno text parsed from documentz*//brrr) lxml.etreer lxml.htmlrrrrIrrZrrKrSgetrootAttributeErrorrRUnicodeDecodeErrorOSErrorr tail) rgrrrrr!frPers r2rpz_LxmlFrameParser._build_docsA .-----          D4=AAA Ndg 2GS$2F7ahv666A777777777777777 E$'&111 IIKK!    1n-- N$n%CQ1MMM N#G,   $'?? Jtwv666 AA%D '''"" - -Bbgm,BGGs0CA3' C3A77C:A7;CB'&C' B41C3B44CD?'&D:D#"D:# D0-D:/D00D::D?cg}|dD]V}||d|d}|r||W|S)Nz.//thead./trr)r extendr)rgrmrtheadelements_at_roots r2rz _LxmlFrameParser._parse_thead_tr0st[[,, # #E KK F++ , , , %{{;77  # E""" r4c`|d}|d}||zS)Nz .//tbody//trr,r rs r2rz _LxmlFrameParser._parse_tbody_trBs/[[00 KK'' I%%r4c,|dS)Nz .//tfoot//trr rs r2rz _LxmlFrameParser._parse_tfoot_trHs{{>***r4Nrr) rErrrrzr}rrorrprrrrr4r2r r s$----"""&&& :888t$&&& +++++r4r rdcd|D}t|}|}|||k}dg}|D]\}}||xx|||z zz cc<dS)Nc,g|]}t|Sr)len)rlrs r2rz$_expand_elements..Ms ' ' '$CII ' ' 'r4r)rmaxr)rdatalenslens_maxnot_maxemptyindlengths r2_expand_elementsr=Ls ' '$ ' ' 'D $<..ds"@V@V$@V@V@V@V@V@Vr4)any)rlirs r2rz"_data_to_frame..ds9WWW3S@V@VRU@V@V@V=V=VW!WWWr4)rrFr4 enumerater=r rO)rheadrfootrtps r2_data_to_framerGWs5zz&))D$ ZZ ! !F&vj'9::F: Xd{ >4yyA~~XW)D//WWW   T D 2 2 26 2 2bwwyys$CC  C )lxmlNrrflavorHTMLFlavors | Nonetype[_HtmlFrameParser]ctt}||vr!tt |d||dvrt dt dnt dt|S)a Choose the parser based on the input flavor. Parameters ---------- flavor : {{"lxml", "html5lib", "bs4"}} or None The type of parser to use. This must be a valid backend. Returns ------- cls : _HtmlFrameParser subclass The parser class based on the requested input flavor. Raises ------ ValueError * If `flavor` is not a valid backend. ImportError * If you do not have the requested `flavor` z* is not a valid flavor, valid flavors are )rrrrz lxml.etree)r>_valid_parserskeysrrr )rI valid_parserss r2_parser_dispatchrPws*,,..//M ]""F|| V V} V V   $$$":..."5))))"<000 & !!r4cLdd|D}d|dS)Nz, c,g|]}t|Srr)rlels r2rz!_print_as_set..s 222"\"%%222r4{})r)r*args r2 _print_as_setrWs0 ))22222 3 3C <<<r4cf|d}nt|tr|f}nt|tjrLt d|Ds2t dt t|jdnGt|trt |nt|}|dz }t|t|}tt}t|}||zs.tt|dt||S)N)rHrc3@K|]}t|tVdSrf)r:r+)rlflavs r2rnz#_validate_flavor..s,<`__. :: attrs = {{'asdf': 'table'}} is *not* a valid attribute dictionary because 'asdf' is not a valid HTML attribute even if it is a valid XML attribute. Valid HTML 4.01 table attributes can be found `here `__. A working draft of the HTML 5 spec can be found `here `__. It contains the latest information on table attributes for the modern web. parse_dates : bool, optional See :func:`~read_csv` for more details. thousands : str, optional Separator to use to parse thousands. Defaults to ``','``. encoding : str, optional The encoding used to decode the web page. Defaults to ``None``.``None`` preserves the previous encoding behavior, which depends on the underlying parser library (e.g., the parser library will try to use the encoding provided by the document). decimal : str, default '.' Character to recognize as decimal point (e.g. use ',' for European data). converters : dict, default None Dict of functions for converting values in certain columns. Keys can either be integers or column labels, values are functions that take one input argument, the cell (not column) content, and return the transformed content. na_values : iterable, default None Custom NA values. keep_default_na : bool, default True If na_values are specified and keep_default_na is False the default NaN values are overridden, otherwise they're appended to. displayed_only : bool, default True Whether elements with "display: none" should be parsed. extract_links : {{None, "all", "header", "body", "footer"}} Table elements in the specified section(s) with tags will have their href extracted. .. versionadded:: 1.5.0 dtype_backend : {{'numpy_nullable', 'pyarrow'}}, default 'numpy_nullable' Back-end data type applied to the resultant :class:`DataFrame` (still experimental). Behaviour is as follows: * ``"numpy_nullable"``: returns nullable-dtype-backed :class:`DataFrame` (default). * ``"pyarrow"``: returns pyarrow-backed nullable :class:`ArrowDtype` DataFrame. .. versionadded:: 2.0 {storage_options} .. versionadded:: 2.1.0 Returns ------- dfs A list of DataFrames. See Also -------- read_csv : Read a comma-separated values (csv) file into DataFrame. Notes ----- Before using this function you should read the :ref:`gotchas about the HTML parsing libraries `. Expect to do some cleanup after you call this function. For example, you might need to manually assign column names if the column names are converted to NaN when you pass the `header=0` argument. We try to assume as little as possible about the structure of the table and push the idiosyncrasies of the HTML contained in the table to the user. This function searches for ``
- Move rows from bottom of body to footer only if all elements inside row are c`tfd|DS)Nc3DK|]}|dVdS)thN)r)rltrgs r2rnzS_HtmlFrameParser._parse_thead_tbody_tfoot..row_is_all_th..s3NNQt''400NNNNNNr4)allr)rowrgs r2 row_is_all_thz@_HtmlFrameParser._parse_thead_tbody_tfoot..row_is_all_ths2NNNN$..:M:MNNNNN Nr4rheader)sectionbodyfooter)rrrappendpop_expand_colspan_rowspan) rg table_html header_rows body_rows footer_rowsrrrrs ` r2rkz)_HtmlFrameParser._parse_thead_tbody_tfoots2**:66 ((44 **:66  O O O O O 5 5 il ; ; 5""9==#3#3444 5 il ; ; 5--k8-LL++Iv+FF--k8-LLtV##r4r#Literal['header', 'footer', 'body']c zg}g}|D]}g}g}d}||} | D]\} |rz|dd|krh|d\} } } || | dkr|| | | dz f|dz }|r|dd|kht|| }|jd|fvr|| }||f}t|| dpd}t|| dpd}t|D]=}|||dkr||||dz f|dz }>^|D]<\} } } || | dkr|| | | dz f=|||}|r\g}g}|D]<\} } } || | dkr|| | | dz f=|||}|\|S)a* Given a list of
`` elements and only for ```` and ```` or ```` argument, it is used to construct the header, otherwise the function attempts to find the header within the body (by putting rows with only ``
`` rows and ```` elements within each ``
`` element in the table. ```` stands for "table data". This function attempts to properly handle ``colspan`` and ``rowspan`` attributes. If the function has a ``
`` elements into the header). Similar to :func:`~read_csv` the `header` argument is applied **after** `skiprows` is applied. This function will *always* return a list of :class:`DataFrame` *or* it will fail, e.g., it will *not* return an empty list. Examples -------- See the :ref:`read_html documentation in the IO section of the docs ` for some examples of reading in HTML tables. rzPcannot skip rows starting from the end of the data (you passed a negative value))NrrrrzO`extract_links` must be one of {None, "header", "footer", "body", "all"}, got ""zPassing literal html to 'read_html' is deprecated and will be removed in a future version. To read from a literal string, wrap it in a 'StringIO' object.) stacklevelrIrZr\rrpr5rqrrr^rIrsrtrurvr`rbrwrKr)r:rArBrrrrr+rArrrrwarningswarn FutureWarningrrm)rZr\rIrrpr5r^rqrrrIrsrtrurvr`rbrwrKs r2 read_htmlrsj(G,-- (Q,, 1   EEE ! ! ! !    &&&   B"c 3    OO 2JJ "    $$    > '))         v  2 e v   )      K ) e   : ) ( &~  $m! "$m# $(% r4)r*r+r,rr-r+)r5r6r-r7)rGrHrIrJrKrLr-rM)r-r+)r-rd)rIrJr-rK)&rZrxr\r]rIryrrzrprzr5r6r^r_rqrarrrJrIrJrsr+rtr{rur|rvrar`rarbrcrwr}rKr(r-r~)Pr __future__r collectionsrrArrtypingrrrr pandas._libsr pandas.compat._optionalr pandas.errorsr r pandas.util._decoratorsr pandas.util._exceptionsrpandas.util._validatorsrpandas.core.dtypes.commonrpandasrpandas.core.indexes.baserpandas.core.indexes.multirpandas.core.seriesrpandas.core.shared_docsrpandas.io.commonrrrrrrrpandas.io.formats.printingrpandas.io.parsersr collections.abcr!r"pandas._typingr#r$r%r&r'r(r)r_RE_WHITESPACEr3rFrWrYrrrr r=rGrMrPrWr_rm no_defaultrrr4r2rs #"""""  >>>>>>('''''444444777777222222******000000%%%%%%000000433333((((((! !     -..1?%%%%%&XXXX:Bf f f f f f f f R WWWWW(8WWWt.=> R+R+R+R+R+'R+R+R+j11112  1 ,  " " " "F 8DDDN\"34555 9=)-,037#'")- FJ25.&*'hhhhh65hhhr4