L iX!UddlmZddlZddlZddlZddlmZddlmZ ddl Z er&ddl m Z ddlmZmZddlmZdd lmZeeefZd ed <Gd d Zy#e $r esddl Z Y@wxYw)) annotationsN) TYPE_CHECKING) atomic_write)Iterator)AnyLiteral) TypeAlias)CachingFileSystemr Detailc~eZdZdZddZddZddZ d ddZ ddZddZ ddZ dd Z dd Z dd Z dd Zy ) CacheMetadataa\Cache metadata. All reading and writing of cache metadata is performed by this class, accessing the cached files and blocks is not. Metadata is stored in a single file per storage directory in JSON format. For backward compatibility, also reads metadata stored in pickle format which is converted to JSON when next saved. cJ|s td||_ig|_d|_y)z Parameters ---------- storage: list[str] Directories containing cached files, must be at least one. Metadata is stored in the last of these directories by convention. z3CacheMetadata expects at least one storage locationFN) ValueError_storage cached_files_force_save_pickle)selfstorages k/mnt/ssd/data/python-lab/Trading/venv/lib/python3.12/site-packages/fsspec/implementations/cache_metadata.py__init__zCacheMetadata.__init__&s.RS S +-$#(c t|d5}tj|}dddj D]3}t |jdts#t|d|d<5|S#1swYQxYw#t$r:t|d5}t j|}dddn #1swYnxYwYwxYw)z6Low-level function to load metadata from specific filerNrbblocks) openjsonloadrpicklevalues isinstancegetlistset)rfnfloadedcs r_loadzCacheMetadata._load9s (b# &!1 &  /A!%%/40!!H+.(  /  & & (b$ (1Q ( ( ( (s? BA4B4A=9BCB4+ C4B= 9CCc|jr+t|5}tj||dddyt|d5}t j||dddy#1swYyxYw#1swYyxYw)z4Low-level function to save metadata to specific fileNw)mode)rrr dumpr)rmetadata_to_saver&r's r_savezCacheMetadata._saveFsu  " "b! 1Q ,a0 1 1bs+ /q *A. / / 1 1 / /sA%A1%A.1A:c#Kt|j}t|jD]7\}}||dz k(}|r|stjj |d||f9yw)aYield locations (filenames) where metadata is stored, and whether writable or not. Parameters ---------- writable: bool Set to True to only yield writable locations. Returns ------- Yields (str, str, bool) r cacheN)lenr enumerateospathjoin)r writable_onlynirwritables r_scan_locationszCacheMetadata._scan_locationsOse  #DMM2 DJAwAEzHX'',,w0'8C C  DsA&A(ct|j|jD]\\}}}}||vr||j}|b|jr"|d|j j |k7rR|jr(tj|dz |jkDrtjj||d}tjj|s||fcSy)zIf path is in cache return its details, otherwise return ``False``. If the optional CachingFileSystem is specified then it is used to perform extra checks to reject possible matches, such as if they are too old. uidtimer&F) zipr<rcopy check_filesfsukeyexpiryr?r5r6r7exists)rr6cfsr&base_r2details r check_filezCacheMetadata.check_filees%((<(<(>@Q@Q$R " MRq55 4[%%'F??ve} D8I'I::$))+v">"KdF4L1Bww~~b!rz! " rcg}|jdjjD]\}}tj|dz |kDs$|j dd}|st d|t jj|jd|}|j||jdj||jdrLt jj|jdd}|j|jd||jd }||fS)zRemove expired metadata from the cache. Returns names of files corresponding to expired metadata and a boolean flag indicating whether the writable cache is empty. Caller is responsible for deleting the expired files. r?r&z)Cache metadata does not contain 'fn' for r2) rrAitemsr?r# RuntimeErrorr5r6r7rappendpopr0)r expiry_time expired_filesr6rJr& cache_pathwritable_cache_emptys r clear_expiredzCacheMetadata.clear_expireds&  --b1668>>@ 0LD&yy{VF^+k9ZZb)&CD6JWW\\$--"3R8$$R(!!"%))$/ 0   R dmmB&7AJ JJt((,j 9#'#4#4R#88222rcg}|jD]W\}}}tjj|r!|j |j |G|j iY|xsig|_y)z>Load all metadata from disk and store in ``self.cached_files``N)r<r5r6rFrQr*r)rrr&rIs rrzCacheMetadata.loadsj ,,. (HB1ww~~b!##DJJrN3##B'  ( )0RDrc|jd|}|ddur/t|d|jz|jk\rd|d<yyy)zPerform side-effect actions on closing a cached file. The actual closing of the file is the responsibility of the caller. rMrTN)rr3 blocksizesize)rr'r6r)s ron_close_cached_filez"CacheMetadata.on_close_cached_filesT   b !$ ' X;d "s1X;'7!++'E'OAhK(P "rc|j|d}|sy|\}}|j|jdr0|jdj ||j |St d)zRemove metadata of cached file. If path is in the cache, return the filename of the cached file, otherwise return ``None``. Caller is responsible for deleting the cached file. NrMzrMN)r@r<rr5r6rFr*rOupdatemaxrAr!r"r%r$r0) rr&rIr;r2rkr)rvs rr_zCacheMetadata.saves(+D,@,@,BDDUDU(V 1 $ RHuww~~b!#zz"~ (..0 3DAqEzX;$.%(82D2L*.AhK &+1Xh%7F"MM!H+6*0AhK$'& 58F3C$D& #(8E?% 3 "KKM,DAq ,*+ Q, % -9-?-?-ABTQQ[BEB\\^ 4ak3/"&q{"3AhK 4 JJub !$0D  b !A 16CsFc(||jd|<y)z8Update metadata for specific file in memory, do not saverMN)r)rr6rJs r update_filezCacheMetadata.update_files&,"d#rN)rz list[str])r&strreturnr )r/r r&rjrkNone)F)r8boolrkzIterator[tuple[str, str, bool]])r6rjrGzCachingFileSystem | Nonerkz#Literal[False] | tuple[Detail, str])rSintrkztuple[list[str], bool])rkrl)r'rr6rjrkrl)r6rjrkz str | None)r6rjrJr rkrl)__name__ __module__ __qualname____doc__rr*r0r<rKrWrr\rbr_rirrrrso(& /%*D!D (D,6 ,634 1("1H-rr) __future__rr5r r?typingr fsspec.utilsrujsonr ImportErrorcollections.abcrrrtyping_extensionsr cachedr dictrjr __annotations__rrsrrr~sf" % (#+)S#XFI&N-N- sA A&%A&