L iu^dZddlZddlZddlZddlZddlZddlmZddlm Z m Z ddlm Z  ddl Z ddlmZddlmZdd lmZmZdd lmZmZmZmZd d lmZej<d Zej<dZ ejBdZ"GddejFjHZ%GddejLZ'GddZ(GddZ)GddeZ*GddeZ+ej<dZ,dZ-GddeZ.d Z/d$d!Z0d"Z1e1d#Z2y#e eef$rdZ YwxYw)%z(This file is largely copied from http.pyN)copy)dumpsloads)urlparseF)_DEFAULT_CALLBACK)register_implementation)AbstractBufferedFileAbstractFileSystem)DEFAULT_BLOCK_SIZE isfilelike nullcontexttokenize)AllBytesz6<(a|A)\s+(?:[^>]*?\s+)?(href|HREF)=["'](?P[^"']+)z-(?Phttp[s]?://[-a-zA-Z0-9@:%_+.~#?&/=]+)z fsspec.httpc eZdZy)JsHttpExceptionN__name__ __module__ __qualname__f/mnt/ssd/data/python-lab/Trading/venv/lib/python3.12/site-packages/fsspec/implementations/http_sync.pyrrsrrc eZdZy)StreamIONrrrrrr!srrceZdZdZddZedZdZedZedZ dZ dZ ed Z ed Z ed Zed Zed ZdZy) ResponseProxyzLooks like a requests responsec<||_||_d|_d|_yN)requeststream_data_headers)selfreqr!s r__init__zResponseProxy.__init__*s    rc|jS|jjj}|jrt ||_|jS||_|jSr)r"r responseto_bytesr!r)r$bs rrawzResponseProxy.raw0sU ::  %%..0A{{%a[ zz zzrc"t|dr|`yy)Nr")hasattrr"r$s rclosezResponseProxy.close:s 4 !  "rc|jbt|jjj j dDcgc]}|j dc}|_|jScc}w)Nz : )r#dictr getAllResponseHeadersstripsplit)r$_s rheaderszResponseProxy.headers>sk ==  "\\??AGGIOOPVWGGDMDM }} s A;c@t|jjSr)intr statusr.s r status_codezResponseProxy.status_codeIs4<<&&''rc|js7t|j|j|j|j dyr)okrurlr;reasonr7r.s rraise_for_statuszResponseProxy.raise_for_statusMs8ww!$**DKKt rc/TK |jj|}|r|ny$wr)r+read)r$ chunksizer6__outs r iter_contentzResponseProxy.iter_contentSs+((-- *C  s&(c.|jjSr)r statusTextr.s rr?zResponseProxy.reason[s||&&&rc |jdkS)N)r;r.s rr=zResponseProxy.ok_s#%%rcB|jjjSr)r r( responseURLr.s rr>zResponseProxy.urlcs||$$000rc6|jjSr)contentdecoder.s rtextzResponseProxy.textgs||""$$rc(d|_|jSNF)r!r+r.s rrNzResponseProxy.contentls xxrc,t|jSr)rrPr.s rjsonzResponseProxy.jsonqsTYYrN)F)rrr__doc__r&propertyr+r/r7r;r@rFr?r=r>rPrNrTrrrrr's( (( ''&&11%% rrcZeZdZdZ d dZdZdZdZdZdZ d Z y) RequestsSessionShimci|_yr)r7r.s rr&zRequestsSessionShim.__init__vs  rNcddlm}m}tj d|||s |s| s|s|s| rt |r |r t d|j}|r|nd}|r$|dtjj|}|j||dg|| r| |_ |r*|jD]\}}|j|||jdd d |_|r4|jt!|gt"d i}|j%|nh|rUt'|t(j*r|j-}|j|gt"d i}|j%|n|j%dt/|| S) Nr)BlobXMLHttpRequestzJS request: %s %szUse json= or data=, not bothr?FAcceptzapplication/octet-stream arraybufferzapplication/json)r!)jsr[r\loggerdebugNotImplementedError ValueErrornewurllibparse urlencodeopentimeoutitemssetRequestHeader responseTypertypesend isinstanceioIOBaserBr)r$methodr>paramsdatar7cookiesfilesauthrjallow_redirectsproxieshooksr!verifycertrTr[r\r%extrakvblobs rr zRequestsSessionShim.requestys_& , (&#6 6WE% % D;< <  "" E6<<11&9:;Ce,e, !CK   +1$$Q* + X'AB( 88U4[MD2D+EFD HHTN $ *yy{88TFT+E$FGD HHTN HHTNS00rc *|jd|fi|S)NGETr r$r>kwargss rgetzRequestsSessionShim.gett||E31&11rc *|jd|fi|S)NHEADrrs rheadzRequestsSessionShim.headst||FC2622rc *|jd|fi|S)NzPOST}rrs rpostzRequestsSessionShim.postt||GS3F33rc *|jd|fi|S)NPUTrrs rputzRequestsSessionShim.putrrc *|jd|fi|S)NPATCHrrs rpatchzRequestsSessionShim.patchrrc *|jd|fi|S)NDELETErrs rdeletezRequestsSessionShim.deletest||Hc4V44r)NNNNNNNNNNNNNN) rrrr&r rrrrrrrrrrXrXusW    #21h234245rrXceZdZdZdZdZ dfd ZedZdZ e de de fd Z e fd Z dd Zdd Zd ZddZdefdZdedfdZdZdZdZ ddZdZdZddZdZxZS) HTTPFileSystema< Simple File-System for fetching data via HTTP(S) This is the BLOCKING version of the normal HTTPFileSystem. It uses requests in normal python and the JS runtime in pyodide. ***This implementation is extremely experimental, do not use unless you are testing pyodide/pyscript integration*** )httphttps sync-http sync-https/c pt ||fi|||nt|_||_||_||_||_|xsi|_||_ ||_ ddl } tjdt|_d|_ t'|} | j)dd|_| j)dd| j)d d| j)d d| |_ y#t $rF} ddl} tjd| | j$d i|xsi|_d|_ Yd} ~ d} ~ wwxYw) a Parameters ---------- block_size: int Blocks to read bytes; if 0, will default to raw requests file-like objects instead of HTTPFile instances simple_links: bool If True, will consider both HTML tags and anything that looks like a URL; if False, will consider only the former. same_scheme: True When doing ls/glob, if this is True, only consider paths that have http/https matching the input URLs. size_policy: this argument is deprecated client_kwargs: dict Passed to aiohttp.ClientSession, see https://docs.aiohttp.org/en/stable/client_reference.html For example, ``{'auth': aiohttp.BasicAuth('user', 'pass')}`` storage_options: key-value Any other parameters passed on to requests cache_type, cache_options: defaults used in open NrzStarting JS sessionTz'Starting cpython session because of: %sFuse_listings_cachelistings_expiry_time max_pathsskip_instance_cacher)superr&r block_size simple_links same_schema cache_type cache_options client_kwargsencodedrr`rarbrXsession ExceptionrequestsSessionrpopr)r$rr same_schemerrrrstorage_optionsr`errequest_options __class__s rr&zHTTPFileSystem.__init__s+B 11(2(>*DV(&$**0b %    LL. /.0DLDG/"1"5"56JE"R2D9K.148%    LLBA F+8++D}/BDDLDGG  s/C&& D5/s r encode_urlzHTTPFileSystem.encode_urls 88C6 6 rpathreturncJ|jddjdd}|S)z-For HTTP, we always want to keep the full URLz sync-http://zhttp://z sync-https://zhttps://)replace)clsrs r_strip_protocolzHTTPFileSystem._strip_protocols+||NI6>> Z  rcDt||}t|dkDr|Sy)N)r_parentlen)rrparrs rrzHTTPFileSystem._parents&god# s8a<Jrc |jj}|j|tj ||j j |j|fi|j}|j|||j}|jr=tj|tj|Dcgc]}|d c}z}n%tj|Dcgc]}|d }}t} t|} |D]D} t!| t"r| d} | j%dr-t'| dkDr| j(dz| j*z| z} | j%dr|j,r5| j%|j/ddzr| j1| | j3ddj%|j3ddj/ddzs| j1| | dvs| j1dj5|j/d| j7dgG| s3|j9dr"|j;|j/dd } |r'| Dcgc]}|d|j9drd nd d c}St=| Scc}wcc}wcc}w) Nrrz://rr)z..z../F)detail directoryfilenamesizern)rrupdaterarbrrr_raise_not_found_for_statusrPrex2findallexsetrrptuple startswithrschemenetlocrrstripaddrjoinlstripendswith_ls_realsorted) r$r>rrkwrrPulinksrEpartsls rrzHTTPFileSystem._ls_real&sS [[    & S DLL  T__S1 AT[[ A ((C0vv   KK%rzz$7G(H!1(HHE#%::d#34aQqT4E4e  HA!U#aD||C SVaZLL5(5<<7!;||F### SZZ_s5J(KGGAJYYw/::KK077C     +,::c?K #; G)I40s K 5 K Kc |jr||jvr|j|}|S|j|fd|i|}||j|<|S)Nr)rdircacher)r$r>rrrEs rlszHTTPFileSystem.lsTsZ  " "sdmm';--$C  $--=F=f=C!$DMM#  rcX|jdk(r t||jy)zU Raises FileNotFoundError for 404s, otherwise uses raise_for_status. iN)r;FileNotFoundErrorr@)r$r(r>s rrz*HTTPFileSystem._raise_not_found_for_status\s)   3 &#C( (!!#rc |jj}|j|tj |||A||k(ry|j dij}|j ||||d<||d<|jj|j|fi|}|j|||jS)Nrr7Range) rrrrarbr_process_limitsrrrrrN)r$r>startendrrr7rs rcat_filezHTTPFileSystem.cat_fileds [[    & S  |ffY+002G#33CDGG #ByM DLL  T__S1 8R 8 ((C0yyriPc |jj}|j|tj ||j j |j|fi|} t|jj ddxs|jj dd}|j||j||t|s t!|d}|j#|dD]-} |j%| |j't)| /y#tttf$rd}YwxYw)Ncontent-lengthContent-LengthwbF)decode_unicode)rrrrarbrrrr9r7rdKeyError TypeErrorset_sizerr rirFwriterelative_updater) r$rpathlpath chunk_sizecallbackrrrrchunks rget_filezHTTPFileSystem.get_filets[[    & U DLL  T__U3 :r :  .5999==!148D $ ((E2% %E^^Ju^E 1E KK   $ $SZ 0 1Hi0 D s.AD//EErc "fd}|jj}|j||j}|dvrt d|t |j |} | |fd|i|} |j| |y)Nc3Kttjrt}d}nt d}d}|5}|r3j |j dd|j dnj t|dd|j}|r2|jt||j}|r2dddy#1swYyxYww)NFrbTrrr) rprqrrr rirseekgetattrrBrr)contextuse_seekfrrrrs r gen_chunksz+HTTPFileSystem.put_file..gen_chunkss%+%e, ud+ /A%%affQl3FF1I%%ga&>?z*K,,SZ8FF:.E / / /s9C(BC C(C%!C()rrz.method has to be either 'post' or 'put', not: ru)rrrlowerrdrrr) r$rrrrrsrr rmethresps ` `` rput_filezHTTPFileSystem.put_files /.[[    &  (@ K t||V,E3 33 ((u5rcd}d}|-|dkr(|| }d}d}n"|xs|j|d}||z}n|d}|s@|%|dkr |"|xs|j|d}||z}n|d}t|tr|dz}d|d |S) z"Helper for "Range"-based _cat_fileNFrrTrrbytes=-)inforpr9)r$r>rrrsuffs rrzHTTPFileSystem._process_limitss  {f5tyy~f5u  ]E37$9499S> D*C#s#qwau%%rc |jj}|j| tj ||j j |j|fi|}|jdkS#t$rYywxYw)NrJF) rrrrarbrrrr;r)r$rrrrs rexistszHTTPFileSystem.existssv [[    &  LL     !6="=A==3& &  sAA== B B c (|j|fi|Sr)r)r$rrs risfilezHTTPFileSystem.isfilest{{4*6**rc r|dk7rt||n |j}|jj} | j ||xs|j |fi|d}|r<|r:t ||f|j||||xs |j|xs |jd| St||f||jd| S)aMake a file-like object Parameters ---------- path: str Full URL with protocol mode: string must be "rb" block_size: int or None Bytes to download in one request; use instance value if None. If zero, will return a streaming Requests file-like instance. kwargs: key-value Any other parameters, passed to requests calls rr)rrmoderrr)rr) rcrrrrrHTTPFilerrrHTTPStreamFile) r$rrr autocommitrrrrrs r_openzHTTPFileSystem._opens2 4<% %#-#9Zt [[    &8ytyy008 $  %%8+At/A/A   "    rcDt||j|jS)z;Unique identifier; assume HTTP files are static, unchanging)rrprotocolrs rukeyzHTTPFileSystem.ukeysT[[$--88rc bi}dD]Y} |jt|j|f||jd|j||j dn[|dd|ddiS#t $r9}|dk(r t||tjt|Yd}~d}~wwxYw) aHGet info of URL Tries to access location via HEAD, and then GET methods, but does not fetch the data. It is possible that the server does not supply any size information, in which case size will be given as None (and certain operations on the corresponding file will not work). )rr) size_policyrrNrrrrnr) r _file_inforrrrrrrarbstr)r$r>rrpolicyexcs rrzHTTPFileSystem.infos% 'F ' ,$* $ ++  ! 88F#/0 '&TBTB66BB  'U?+C0c9 SX&&  'sAA,, B.5/B))B.c ddl}|jd}|j|}|jddk\r|jdn t |}|jddk\r|jdn t |}t ||}|j dd} t|s;|} d} |r|d z }n|j|r| s|gS||j|iS| sgSiSd|d|vr=|d|jd} |d| dz} d |vrdn|| dzdjddz} n"d } d |vrdn||dzdjddz} |j| f|xs| d d d |} d|jddjddjddjddjddjddjddjddjddjdd jd!d"jdzdz}|jd#d$|}|jd%d&|}|j|jd$d'}t!| Dcic]8}|j#|jddjdr|| |:}}| r|St%|Scc}w)(a Find files by glob-matching. This implementation is idntical to the one in AbstractFileSystem, but "?" is not considered as a character for globbing, because it is so common in URLs, often identifying the "query" part. rNr*[rFrz/*z**rT)maxdepthwithdirsr^\z\\.z\.+z\+z//(z\()z\)|z\|z\^$z\${z\{}z\}z[*]{2}z =PLACEHOLDER=z[*]z[^/]*z.*)rerrfindrminr has_magicrrrindexcountrrsubcompilermatchlist)r$rr,rr8endsindstarindbraceindrrootdepthind2allpathspatternprEs rglobzHTTPFileSystem.glob7s }}S!##D)$(IIcNa$7$))C.SY%)YYs^q%8499S>c$i'8$He,DE T" 6M $))D/22II D#J :$$S)D $(#D DLDd4!8:.>.D.DS.IA.MED DLDd379o.C.CC.H1.LE499  #,utD LR   T5)e$e$s#e$e$e$e$e$e$e$   $"&&?G<"&&1"**W___dCDH% }}QYYtS188=> x{N  J9  s =Kcb t|j|S#ttf$rYywxYwrR)boolrrrd)r$rs risdirzHTTPFileSystem.isdirs1  & &!:.  s ..)TNT readaheadNNF)T)NN)rNNNNNr)rrrrUr sepr&rVrr classmethodr&rrrrrrrrrrrrrr!rrLrO __classcell__rs@rrrsrdetailsrr&) r$rWr>rrrrrrrrs rr&zHTTPFile.__init__sb 4<%&?@ @ #T6B   !!'   rc>|dkrD|jdk(r5|j|j|jks|j|j|dkr4|jn#t |j|jz |}t ||S)a5Read bytes from file Parameters ---------- length: int Read up to this many bytes. If negative, read all content to end of file. If the server has not supplied the filesize, attempting to read only part of the data will raise a ValueError. r)locr blocksize _fetch_allr:rrB)r$lengthrs rrBz HTTPFile.reads|aZDHHMYY*tyyDNN/J OO  99 z!TXX-v6Fw|F##rctjd|t|jts|j j |jj|jfi|j}|j|j}t t|dd||_t||_yy)zRead whole file in one shot, without caching This is only called when position is still at zero, and read() is called without a byte-count. zFetch all for N)rfetcherr[ru)rarbrpcacherrrrWrr>rr@rNrr)r$rrEs rr\zHTTPFile._fetch_alls  ~dV,-$**h/    !3!3DHH!=MMA   ))C!s3xTWXDJCDI 0rc|jdd}tjd|}|sy|ddk(rdx}}n.|djdDcgc] }t |c}\}}|d dk(rdn t |d }|||fScc}w) zParse the Content-Range header Content-Rangerzbytes (\d+-\d+|\*)/(\d+|\*))NNNrr*Nrr)rr8r@r5r9)r$r7smrrxtotals r_parse_content_rangezHTTPFile._parse_content_ranges KK , HH3Q 7# Q43; EC*+A$**S/:Q#a&:JE3! QqTc5  ;sBctjd|d|d||jj}|j dij}d|d|dz |d<tjd|j |d|j j|jj|j fd|i|}|jd k(ry |j|jjd |jjd |dz}|jd k(xs4|j|jd|k(xst|||z k}|r|j}|S|dkDr t!dd}g}|j#ddD]!} |j%| |t'| z }#d j)|d||z }|S)a3Download a block of data The expectation is that the server returns only the requested bytes, with HTTP code 206. If this is not the case, we first check the headers, and then stream the output - if the data size is bigger than we requested, an exception is raised. zFetch range for r1rr7rrrz%s : %sirrrrzThe HTTP server doesn't appear to support range requests. Only reading this file from the beginning is supported. Open with block_size=0 for a streaming file interface.iFN)rarbrrrr>rrrWrr;r@r7rgr9rNrdrFappendrr) r$rrrr7rclresponse_is_rangerErs r _fetch_rangezHTTPFile._fetch_ranges  'vRwau=>!!#**Y+002#E7!C!G95 Y''*:; DLL  TWW//9 U7 Uf U ==C   YY]]+QYY]];KSSTW-U V MMS  &((3A6%? &2w#+%  ))C  QYI BCu5 ! 5!c%j  !((3- #+.C r)NNrbytesNN) rrrrUr&rBr\rgrmrSrTs@rrrs58   6$. ! !0rrz([*[])c4tj|}|duSr) magic_checksearch)rcr@s rr;r;,s   q !E  rc4eZdZdfd ZdZddZdZxZS)rc f||_||_|dk7rt|dd|_t |d |||dd||jj |jj|fddi|}|jj|||jdd|_ d |_ ||_ y) Nrr$rP)rWrrrr!TiFrr)r>rrdrXrr&rrWrrrFitleftoverr)r$rWr>rrrrrs rr&zHTTPStreamFile.__init__2s 4<  #T2  VBSt VvV DLL  TWW//4 LT LV L ++As3..u- rctd)NzCannot seek streaming HTTP file)rd)r$argsrs rrzHTTPStreamFile.seekAs:;;rc|jg}t|j}||ks|dkrJ |jj}|r|j |nn|t|z }||krD|dkrJdj |}|dk\r||d|_|d|}nd|_|xjt|z c_|S#t$rYXwxYw)Nrr)rwrrv__next__ StopIterationrjrrZ)r$numbufslengrEs rrBzHTTPStreamFile.readDs 4==!SjC!G gg&&( C  CH DSjC!Ghhtn !8IDMds)CDM CH !  sC CCcF|jjd|_y)NT)rr/closedr.s rr/zHTTPStreamFile.closeZs   r)rNro)rrrr&rrBr/rSrTs@rrr1s <,rrc |j}|jdij}d|d|dz |d<|j|fd|i|}|j|jS)Nr7rrrr)rrrr@rN)rr>rrrr7rs r get_ranger_sq [[]FjjB',,.Gwaay1GG C33F3A 99rc ltjd||j}|jdd}|j dij}||d<i}|dk(r|j |fd|i|}n)|dk(r|j|fd|i|}nt d||jd|jvrt|jd|d <nd |jvr.t|jd jd d |d <ned |jvrt|jd |d <n;d|jvr-t|jdjd d |d <dD]0}|jj |s|j|||<2|S)zCall HEAD on the server to get details about the file (size/checksum etc.) Default operation is to explicitly allow redirects and use encoding 'identity' (no compression) to get the true size of the target. zRetrieve file size for %sryTr7rrz)size_policy must be "head" or "get", got rrrbrrrz content-range)ETagz Content-MD5Digest) rarbrrrrrr@r7r9r5) r>rr#rarrrrchecksum_fields rr%r%is  LL,c2 [[]F %t ,B ::i $ ) ) +DF9 Df GLL ;b ;F ;   GKK :R :6 :CK=QRR 199$199%567V AII %199_5;;C@CDV QYY &199%567V AII %199_5;;C@CDV ;= 99== (#$99^#rs2.  !/3@NNRZZMNbjjEF   = )3fll,,2rzz K K \H5H5VL'L^W#Wtbjj"  +)+\(XH Cc ('2 DsD D,+D,