wL iC dZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z ddl m Z ejeZeGdd Zed Gd d ZeGddZeGddZde dedefdZde dedefdZde dede efdZde dedefdZde dedededdf dZde de fdZdedefdZy)u$Contains utilities to handle the `../.cache/huggingface` folder in local directories. First discussed in https://github.com/huggingface/huggingface_hub/issues/1738 to store download metadata when downloading files from the hub to a local directory (without using the cache). ./.cache/huggingface folder structure: [4.0K] data ├── [4.0K] .cache │ └── [4.0K] huggingface │ └── [4.0K] download │ ├── [ 16] file.parquet.metadata │ ├── [ 16] file.txt.metadata │ └── [4.0K] folder │ └── [ 16] file.parquet.metadata │ ├── [6.5G] file.parquet ├── [1.5K] file.txt └── [4.0K] folder └── [ 16] file.parquet Download metadata file structure: ``` # file.txt.metadata 11c5a3d5811f50298f278a704980280950aedb10 a16a55fda99d2f2e7b69cce5cf93ff4ad3049930 1712656091.123 # file.parquet.metadata 11c5a3d5811f50298f278a704980280950aedb10 7c5d3f4b8b76583b422fcb9189ad6c89d5d97a094541ce8932dce3ecabde1421 1712656091.123 } ``` N) dataclass)Path)Optional) WeakFileLockc@eZdZUdZeed<eed<eed<dedefdZy) LocalDownloadFilePathsa Paths to the files related to a download process in a local dir. Returned by [`get_local_download_paths`]. Attributes: file_path (`Path`): Path where the file will be saved. lock_path (`Path`): Path to the lock file used to ensure atomicity when reading/writing metadata. metadata_path (`Path`): Path to the metadata file. file_path lock_path metadata_pathetagreturnc0|jjt|jjd|dz }t |j }t jdk(r-t|dkDr|jdstd|z}|S)z^Return the path where a file will be temporarily downloaded before being moved to `file_path`..z .incompletent\\?\) r parent _short_hashnamestrresolveoslen startswithr)selfr path resolved_paths c/mnt/ssd/data/python-lab/Trading/venv/lib/python3.12/site-packages/huggingface_hub/_local_folder.pyincomplete_pathz&LocalDownloadFilePaths.incomplete_pathWs!!((k$:L:L:Q:Q.R-SSTUYTZZe+ffDLLN+  77d?s=1C7 @X@XYb@c M12D N)__name__ __module__ __qualname____doc__r__annotations__rr r!rr r Cs, OOCDr!r T)frozenc:eZdZUdZeed<eed<eed<eed<y)LocalUploadFilePathsa Paths to the files related to an upload process in a local dir. Returned by [`get_local_upload_paths`]. Attributes: path_in_repo (`str`): Path of the file in the repo. file_path (`Path`): Path where the file will be saved. lock_path (`Path`): Path to the lock file used to ensure atomicity when reading/writing metadata. metadata_path (`Path`): Path to the metadata file. path_in_repor r r N)r"r#r$r%rr&rr'r!rr*r*bs OOr!r*c:eZdZUdZeed<eed<eed<eed<y)LocalDownloadFileMetadataaG Metadata about a file in the local directory related to a download process. Attributes: filename (`str`): Path of the file in the repo. commit_hash (`str`): Commit hash of the file in the repo. etag (`str`): ETag of the file in the repo. Used to check if the file has changed. For LFS files, this is the sha256 of the file. For regular files, it corresponds to the git hash. timestamp (`int`): Unix timestamp of when the metadata was saved i.e. when the metadata was accurate. filename commit_hashr timestampN)r"r#r$r%rr&floatr'r!rr-r-zs M Ir!r-ceZdZUdZeed<dZeeed<dZ ee ed<dZ ee ed<dZ ee ed<dZee ed<d Ze ed <d Ze ed <d ed dfdZy)LocalUploadFileMetadatazT Metadata about a file in the local directory related to an upload process. sizeNr0 should_ignoresha256 upload_mode remote_oidF is_uploaded is_committedpathsrc 6t|j5|jjd5}t j}|j t |dz|j t |j|j d|j-|j t t|j|j d|j|j |j|j d|j|j |j|j d|j|j |j|j d|j t t|jdz|j t t|jdzddd|_dddy#1swYxYw#1swYyxYw)zSave the metadata to disk.w N)rr r opentimewriterr4r5intr6r7r8r9r:r0)rr;f new_timestamps rsavezLocalUploadFileMetadata.saves %// * +$$))#. ( FHSM !%K#% $J $KL$+.+4+r!r3 local_dirr.rc\tjj|jd}tjdk(r$|j dsd|vrt d|d||z }t|dz |dz }|jd }tjdk(rt|j d sttjj|d kDrtd tjj|z}td tjj|z}td tjj|z}|jjd d |jjd d t|||S)aCompute paths to the files related to a download process. Folders containing the paths are all guaranteed to exist. Args: local_dir (`Path`): Path to the local directory in which files are downloaded. filename (`str`): Path of the file in the repo. Return: [`LocalDownloadFilePaths`]: the paths to the files (file_path, lock_path, metadata_path, incomplete_path). /r..\\..\*Invalid filename: cannot handle filename 'B' on Windows. Please ask the repository owner to rename this file.download .metadata.lockrrTparentsexist_ok)r r r )rrjoinsplitrr ValueError_huggingface_dir with_suffixrrabspathrrmkdirr rGr.sanitized_filenamer r r s rget_local_download_pathsr]sl x~~c':; ww$  ( ( 0H@R4R<=Oabc=!D ?   k,U-@-@,AA3Fklk''..0 kNN%NuObObNccefgeh#ijjk k$% C!!> ?!D sG-EA3D7 E7GG7E <E G &G3F G G%G<GGG G-GG- G*G-)G**G--G6c $t||}t|j5|jj rc |jj 5}t |jj}t|jj}|jj}|dk(rdntt|}|jj}|dk(rdn|} |jj} | dk(rdn| } | dvrtd|jd| |jj} | dk(rdn| } tt|jj}tt|jj}t|||| | | ||}dddj$C|j&r7|j(s+t+j*|j$z d kDrd |_ |j$<|j,j/j0|j$kr |cdddStj3d |d dddt|j,j/j6S#1swYxYw#t$r}tj!d|jd|d |jj#n;#t$r/}tj!d |jd|Yd}~nd}~wwxYwYd}~d}~wwxYw#t4$rYwxYw#1swYxYw)aRead metadata about a file in the local directory related to an upload process. TODO: factorize logic with `read_download_metadata`. Args: local_dir (`Path`): Path to the local directory in which files are downloaded. filename (`str`): Path of the file in the repo. Return: `[LocalUploadFileMetadata]` or `None`: the metadata if it exists, `None` otherwise. N)Nregularlfsz Invalid upload mode in metadata rc)r0r4r5r6r7r8r9r:rbrdrei@Frfrg)r4)r`rr r rhr?r1rirjrBrFrVr+r3rkrlrmrnr0r9r:r@r rorprqrrst_size)rGr.r;rCr0r4_should_ignorer5_sha256r6 _upload_moder7 _remote_oidr8r9r:rsrts rread_upload_metadatarFs #9h 7E eoo &=    % % ') k((--/1 %ajjl&8&8&: ;Iqzz|1134D%&ZZ\%7%7%9N,:b,@Dd3~K^F_Mjjl002G%,]TF#$::<#5#5#7L*6"*<$,K"*BB(+KEL^L^K__abman)opp"#**,"4"4"6K)4): J"&s1::<+=+=+?'@"AK#'AJJL,>,>,@(A#BL6"+!&3%$/#-$/%1 H-X"".(( --IIK("4"44y@',$ %%1eoo6J6J6L6U6UYaYkYk6k#s==t 4XJ>abcu=@ #(<(<(>(F(F GG{@ k,U-@-@,AA3Fklk''..0 kNN%NuObObNccefgeh#ijjk k0%  w==sNK#FK(K#0AN>M7M7K K## M4,&M/L.-M/. M&7%M!M/!M&&M/)N/M44N7 NNNNNr/r c (t||}t|j5|jj d5}|j |d|dt j dddddddy#1swYxYw#1swYyxYw)zWrite metadata about a file in the local directory related to a download process. Args: local_dir (`Path`): Path to the local directory in which files are downloaded. r=r>N)r]rr r r?rAr@)rGr.r/r r;rCs rwrite_download_metadatars %Y 9E eoo &?  % %c * ?a GG{m2dV2diik]"= > ??? ? ???s"B-A<+B<B BBcP|dz dz }|jdd|dz }|dz }|js; t|d5|jd d d d  |j |S|S#1swYxYw#t$rY,t $rY6wxYw#t $rY|SwxYw) zKReturn the path to the `.cache/huggingface` directory in a local directory.z.cache huggingfaceT)rSrRz .gitignorez.gitignore.lockg?)timeout*N)rZrhr write_text IndexErrorOSErrorrn)rGr gitignoregitignore_locks rrWrWs x - /DJJdJ+|#I--N     nc: *$$S) *    ! ! # K4K * *        K sA A?A3A?B3A<8A?? B BB B%$B%ctjtj|j j j S)N)base64urlsafe_b64encodehashlibsha1encodedigestdecode)r.s rrrs5  # #GLL1B$C$J$J$L M T T VVr!)r%rrloggingrr@ dataclassesrpathlibrtypingrutilsr getLoggerr"rlr r*r-r3rr]r`rurrrWrr'r!rrsz#J !   8 $  < $.  , -+-+ -+`%i%i%i@V%iP'd'c'>R'T/d/c/hG`>a/dOHDOHCOH