o 0i @sUddlZddlZddlZddlZddlmZddlmZddlm Z ddl m Z m Z ddl m Z ddlmZddlmZmZdd lmZmZmZmZmZmZdd lmZmZddlZddlZdd lm Z m!Z!m"Z"dd l#m$Z$d dl%m&Z&d dl'm(Z(m)Z)d dl*m+Z+m,Z,m-Z-m.Z.m/Z/d dl0m1Z1m2Z2d dl3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9d dl:m;Z;mZ>d dl?m@Z@eAdejBZCe GdddZDe GdddeDZEe GdddeDZFeGejHZIeeJd<GdddeIZKGdd d ejHeKd!ZLGd"d#d#ejMjNZOGd$d%d%ejMjNZPd&eQd'eQfd(d)ZRd*eQd'eQfd+d,ZSd-eQd.eeTd'efd/d0ZUd1eLd-eQd2eQd3eVd4eQf d5d6ZWd7d8ZXeLZYdS)9N)deque) ExitStack)deepcopy) dataclassfield)datetime)chain)Path PurePosixPath)AnyIterableIteratorNoReturnOptionalUnion)quoteunquote)_DEFAULT_CALLBACK NoOpCallback TqdmCallback) isfilelike) constants)CommitOperationCopyCommitOperationDelete)BucketNotFoundErrorEntryNotFoundErrorHfHubHTTPErrorRepositoryNotFoundErrorRevisionNotFoundError) hf_hub_urlhttp_get) BucketFile BucketFolderHfApiLastCommitInfoRepoFile RepoFolder)HFValidationErrorhf_raise_for_status http_backoffhttp_stream_backoff)md5zy (^refs\/convert\/\w+) # `refs/convert/parquet` revisions | (^refs\/pr\/\d+) # PR revisions c@s0eZdZUdZeed<eed<defddZdS)HfFileSystemResolvedPathz_Top level Data structure containing information about a resolved Hugging Face file system path.rootpathreturncCs|jd|jdS)N/)r.r/rstripselfr5X/home/kim/smarthome/.venv/lib/python3.10/site-packages/huggingface_hub/hf_file_system.py unresolve5sz"HfFileSystemResolvedPath.unresolveN)__name__ __module__ __qualname____doc__str__annotations__r7r5r5r5r6r-.s r-c@sveZdZUdZeed<eed<eed<eed<eddZeed<eddZeed <ed dd Z e eed <d dZ d S)"HfFileSystemResolvedRepositoryPathzLData structure containing information about a resolved path in a repository. repo_typerepo_idrevision path_in_repoFinitr.r/N)defaultrepr _raw_revisioncCsftj|jd|j}|jr|d|j|_n|jtjkr*|dt |j|_n||_|j |_ dS)N@) rREPO_TYPES_URL_PREFIXESgetr?r@rGr.rADEFAULT_REVISION safe_revisionrBr/)r4 repo_pathr5r5r6 __post_init__Gs  z0HfFileSystemResolvedRepositoryPath.__post_init__) r8r9r:r;r<r=rr.r/rGrrOr5r5r5r6r>9s  r>c@s4eZdZUdZeed<eddZeed<ddZdS) HfFileSystemResolvedBucketPathzHData structure containing information about a resolved path in a bucket. bucket_idFrCr.cCsd|j|_dS)Nzbuckets/)rQr.r3r5r5r6rOYsz,HfFileSystemResolvedBucketPath.__post_init__N) r8r9r:r;r<r=rr.rOr5r5r5r6rPRs  rP _cached_basecs(eZdZdZfddZddZZS)_Cacheda Metaclass for caching HfFileSystem instances according to the args. This creates an additional reference to the filesystem, which prevents the filesystem from being garbage collected when all *user* references go away. A call to the :meth:`AbstractFileSystem.clear_instance_cache` must *also* be made for a filesystem instance to be garbage collected. This is a slightly modified version of `fsspec.spec._Cached` to improve it. In particular in `_tokenize` the pid isn't taken into account for the `fs_token` used to identify cached instances. The `fs_token` logic is also robust to defaults values and the order of the args. Finally new instances reuse the states from sister instances in the main thread. cstj|i|i|_dSN)super__init___cache)clsargskwargs __class__r5r6rVqs z_Cached.__init__c Os|dd}|j|tg|Ri|}|j|tjg|Ri|}|s7|jr7||jvr7||_|j|St j |g|Ri|}|sc|jrc||jvrc|j| }| D] \}} t ||| qX||_||_||_|jry|sy||_||j|<|S)NZskip_instance_cacheF)pop _tokenize threading get_ident main_threadidentZcachablerWZ_latesttype__call___get_instance_stateitemssetattrZ _fs_token_ storage_argsstorage_options) rXrYrZskipZfs_tokenZfs_token_main_threadobjinstance_stateattr state_valuer5r5r6rdys$     z_Cached.__call__)r8r9r:r;rVrd __classcell__r5r5r[r6rSas rScspeZdZdZdZdZddddddeedee edfdee d ee ffd d Z e d e d efddZ dededeed ee eeffddZded ee eeffddZ dTdedeed eeeffddZdTdeed dfddZ   dUdededee deed edf d d!ZdTdedeed dfd"d#Z $  dVded%e d&ee deed df d'd(Z dWded*e d+e deed eeeeeefff d,d-Z $ $   dXded%e d+e deed ee d&ee f d.d/Zded0ed%e d eeee ffd1d2Z!ded e"eeeeeefffd3d4 Z#dTded&ee d eeffd5d6 Z$  $ $ $ dYded&ee d7e d*e d+e deed eeeeeeeeffffd8d9Z%dTd:ed;edeed dfdd?Z(dZded+e deed eeeffd@dAZ)dBdCZ*dDdEZ+dFdGZ,ded efdHdIZ-e.dfd[fdJdK Z/e0dLdMZ1dNdOZ2dPdQZ3dRdSZ4Z5S)\ HfFileSystemaz Access a remote Hugging Face Hub repository as if were a local file system. > [!WARNING] > [`HfFileSystem`] provides fsspec compatibility, which is useful for libraries that require it (e.g., reading > Hugging Face datasets directly with `pandas`). However, it introduces additional overhead due to this compatibility > layer. For better performance and reliability, it's recommended to use `HfApi` methods when possible. The file system supports paths for the `hf://` protocol, which follows those URL schemes: * Models, Datasets and Spaces repositories: ``` hf://[@]/ hf://datasets/[@]/ hf://spaces/[@]/ ``` * Buckets (generic storage): ``` hf://buckets// ``` Note: when using the [`HfFileSystem`] directly, passing the `hf://` protocol prefix is optional in paths. Args: endpoint (`str`, *optional*): Endpoint of the Hub. Defaults to . token (`bool` or `str`, *optional*): A valid user access token (string). Defaults to the locally saved token, which is the recommended method for authentication (see https://huggingface.co/docs/huggingface_hub/quick-start#authentication). To disable authentication, pass `False`. block_size (`int`, *optional*): Block size for reading and writing files. expand_info (`bool`, *optional*): Whether to expand the information of the files. **storage_options (`dict`, *optional*): Additional options for the filesystem. See [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.__init__). Usage: ```python >>> from huggingface_hub import hffs >>> # List files >>> hffs.glob("my-username/my-model/*.bin") ['my-username/my-model/pytorch_model.bin'] >>> hffs.ls("datasets/my-username/my-dataset", detail=False) ['datasets/my-username/my-dataset/.gitattributes', 'datasets/my-username/my-dataset/README.md', 'datasets/my-username/my-dataset/data.json'] >>> # Read/write files >>> with hffs.open("my-username/my-model/pytorch_model.bin") as f: ... data = f.read() >>> with hffs.open("my-username/my-model/pytorch_model.bin", "wb") as f: ... f.write(data) ``` Specify a token for authentication: ```python >>> from huggingface_hub import HfFileSystem >>> hffs = HfFileSystem(token=token) ``` rHZhfN)endpointtoken block_size expand_inforqrrrsrtcsTtj|i||p tj|_||_t||d|_||_||_ i|_ i|_ i|_ dS)N)rqrr) rUrVrENDPOINTrqrrr$_apirsrt_repo_and_revision_exists_cache_bucket_exists_cachedircache)r4rqrrrsrtrYrir[r5r6rVs  zHfFileSystem.__init__threading_identr0cs\dptjd<dd<fddtD|||f}tt|}|S)zDeterministic token for cachingrqrrci|]}||qSr5r5).0keyrZr5r6 z*HfFileSystem._tokenize..)rKrrusortedr,r<encode hexdigest)rXrzrYrZZ tokenize_argshr5r~r6r^s  zHfFileSystem._tokenizer?r@rAc Cs|||f|jvrqz |jj|||tjdWnKttfy=}zd|f|j|||f<d|f|j||df<WYd}~n8d}~wty`}zd|f|j|||f<d|j||df<WYd}~nd}~wwd|j|||f<d|j||df<|j|||fS)N)rAr?timeoutFTN)rwrvZ repo_inforZHF_HUB_ETAG_TIMEOUTrr(r)r4r?r@rAer5r5r6_repo_and_revision_exists"   z%HfFileSystem._repo_and_revision_existrQc Csb||jvr,z|j|Wnty&}z d|f|j|<WYd}~n d}~wwd|j|<|j|S)NFr)rxrvZ bucket_infor)r4rQrr5r5r6_bucket_existss   zHfFileSystem._bucket_existsr/cCsdttdttdttfdd}||}|std|ddd krQd|dd d }d|dd d }||\}}|sKt||t||d S|dddt j vrud|vrgtd|dd \}}t j |}nt j }|ddkr=dd|dd dvr|dd \}} d| vrt| } | d ur|d | fvrtd| d} | } n | dd \} } nd} |t| |}||||\} }| st||nd } d|dd d} d|ddd }|dd}d|dd d }| }|} ||||\} }| s._align_revision_in_path_with_revisionz)r4r/rArrQZ bucket_existserrr?r@rmatchrBZrepo_and_revision_existZrepo_id_with_namespaceZpath_in_repo_with_namespaceZrepo_id_without_namespaceZpath_in_repo_without_namespace_r5r5r6 resolve_paths           zHfFileSystem.resolve_pathcCs|s|j|jdS||}|}|r'|j|d||}|s|jsTt|t rJ|j|j |j dfd|j|j |j |j fddS|j |jddSdS)ac Clear the cache for a given path. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.invalidate_cache). Args: path (`str`, *optional*): Path to clear from cache. If not provided, clear the entire cache. N)ryclearrwrr7r]_parentr/rr>r?r@rArxrQ)r4r/ resolved_pathr5r5r6invalidate_caches&    zHfFileSystem.invalidate_cacherbmode)HfFileSystemFileHfFileSystemStreamFilecKsj|dur|n|j}|dur||d<d|vrtd|dkr)t||f||d|St||f||d|S)Nrsaz/Appending to remote files is not yet supported.r)rrA)rsrrr)r4r/rrsrArZr5r5r6_openszHfFileSystem._openc Kst|j||d}t|tr|jj|j|jgdn|jj|j|j |j |j |j | d| dd|j|ddS)NrAdeletecommit_messagecommit_description)rBr@rrr?rArrr/)rrrPrvbatch_bucket_filesrQr/Z delete_filerBr@rrr?rArKrr7)r4r/rArZrr5r5r6_rms  zHfFileSystem._rmF recursivemaxdepthc sj||d}j||||d}t|tr(fdd|D}jj|j|dnDfdd|D} dd| D} d|d } | |rCd nd 7} | |d urQd |d nd 7} jj|j|j j | |j | d| | ddj |dd S)a Delete files from a repository. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.rm). > [!WARNING] > Note: When possible, use `HfApi.delete_file()` for better performance. Args: path (`str`): Path to delete. recursive (`bool`, *optional*): If True, delete directory and all its contents. Defaults to False. maxdepth (`int`, *optional*): Maximum number of subdirectories to visit when deleting recursively. revision (`str`, *optional*): The git revision to delete from. r)rrrAc"g|] }|s|jqSr5isdirrr/r|r/r3r5r6 "z#HfFileSystem.rm..rcrr5rrr3r5r6rrcSsg|]}t|dqS))rB)r)r|rBr5r5r6rrzDelete  z recursively rHNz up to depth rr)r@r?rr operationsrArrr)rZ expand_pathrrPrvrrQ create_commitr@r?rrrArKrr7) r4r/rrrArZrpathsrZ paths_in_reporrr5r3r6rms(    zHfFileSystem.rmTdetailrefreshc s|j|d}|z|jf||d|}WnBty[|js(tdz|j|f||d|}Wn tyDg}Ynwfdd|D}t|dkrYtdYnw|r`|Sdd|DS)a List the contents of a directory. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.ls). > [!WARNING] > Note: When possible, use `HfApi.list_repo_tree()` for better performance. Args: path (`str`): Path to the directory. detail (`bool`, *optional*): If True, returns a list of dictionaries containing file information. If False, returns a list of file paths. Defaults to True. refresh (`bool`, *optional*): If True, bypass the cache and fetch the latest data. Defaults to False. revision (`str`, *optional*): The git revision to list from. Returns: `list[Union[str, dict[str, Any]]]`: List of file paths (if detail=False) or list of file information dictionaries (if detail=True). r)rrANcg|] }|dkr|qSnamer5r|orr5r6r"z#HfFileSystem.ls..rcSsg|]}|dqSrr5rr5r5r6r%s)rr7_ls_treerr/rrlen)r4r/rrrArZroutr5rr6lss$  "    zHfFileSystem.lsc sT|dur|n jdurjnd}j||d}|}|j}|r"|nd}g} |jvr |s j|} | | g} |rdtfdd| D} | r| \} |dusZ|kr| djvri| | dnj| d} | | | fdd| D| sLg}|rt |t rfd d| D}|r| s|r|rt j | |}|d s||ks|t| |vr|d n||durԈt|dd }||8}fd d| D} tjD]}|d rj|dqjd| j|d |||d | St |trj|j|j |d}njj|j|j |||j|jd}|D]z}|d |j }t |t rN||j!d|j"|j#|j$|j%|j&d}n-t |t'rb||j!d|j$|j(|j)d}nt |t*rs|dd|j+|j%d}n|dd|j)d}|d}j,|g||t|dd |dus|kr| |q-| S)NFrrrcs g|] }|ddkr|fqSrc directoryr5r| path_infodepthr5r6rBs z)HfFileSystem._ls_tree..rcs$g|]}|ddkrd|fqS)rcrrr5rrr5r6rMs   cs&g|]}|ddur|dqS) last_commitNr)rrr3r5r6rWs&r1cs"g|] }|dds|qS)rr1) startswithr) common_pathr5r6rhrT)rrrArtr)prefixr)rexpandrAr?filersizercblob_idlfsxet_hashrsecurity)rrrcrmtime uploaded_atrrrrrctree_idr)rrrcr)-rtrr7r.ryextendrpopleftappendrr>osr/ commonprefixendswithrr2rrrlistrr]rrP_list_bucket_tree_with_foldersrQrvZlist_repo_treer@rAr?r&rrrrrrr"rrr'r setdefault)r4r/rrrArtrr root_pathrZcached_path_infosZdirs_not_in_dircacheZ dirs_to_visitdir_infoZdirs_not_expanded common_prefixZcommon_path_depthZ cached_pathtreer cache_pathZcache_path_info parent_pathr5)rrr4r6r's        E      zHfFileSystem._ls_treerc Cs|jj|||d}i}|rd|dnd}g}|D]O}|||s$qtt|jjd| dD]5} tdt | |j d} | j|vrO|| | || j<q3| j durh|| j} | j dusd| j | j krh| j | _ q3q|swt d|d |d |S) z|j||dd}tj|g|Ri|EdHdS)a Return all files below the given path. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.walk). Args: path (`str`): Root path to list files from. Returns: `Iterator[tuple[str, list[str], list[str]]]`: An iterator of (path, list of directory names, list of file names) tuples. rArN)rrKr7rUwalk)r4r/rYrZr[r5r6rs $zHfFileSystem.walkc s0|j||dd}tj|fd|i|S)ah Find files by glob-matching. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.glob). Args: path (`str`): Path pattern to match. Returns: `list[str]`: List of paths matching the pattern. rArr)rrKr7rUglob)r4r/rrZr[r5r6rs zHfFileSystem.globwithdirsc s|dur |dkr td|j||d}|}z|j|fd||d|Wn-tySz|j|fd|i|dd krB|iiniWn tyPiYnwYn(w|s^d d Dn|j|fi|} | dd krr| gnd dDt} |s| Sfdd| DS)a List all files below path. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.find). Args: path (`str`): Root path to list files from. maxdepth (`int`, *optional*): Maximum depth to descend into subdirectories. withdirs (`bool`, *optional*): Include directory paths in the output. Defaults to False. detail (`bool`, *optional*): If True, returns a dict mapping paths to file information. Defaults to False. refresh (`bool`, *optional*): If True, bypass the cache and fetch the latest data. Defaults to False. revision (`str`, *optional*): The git revision to list from. Returns: `Union[list[str], dict[str, dict[str, Any]]]`: List of paths or dict of file information. Nrzmaxdepth must be at least 1rT)rrrrArcrcSsg|] }|ddkr|qSrr5rr5r5r6r-rz%HfFileSystem.find..rcSsi|]}|d|qSrr5rr5r5r6r2rz%HfFileSystem.find..cr{r5r5)r|rrr5r6r7r)rrr7rrinfoFileNotFoundErrorr) r4r/rrrrrArZrrnamesr5rr6finds4     zHfFileSystem.findpath1path2c KsH|j||d}|j||d}t|tst|trtd|j|jko'|j|jk}|rSd|d|}|jj|j|j|j| d|| ddt |j |j |jdgd n?|j |d |jd } | } Wd n1skwYd|d|}|jj| |j |j|j|j|j| d|| dd |j|d |j|d d S)a Copy a file within or between repositories. > [!WARNING] > Note: When possible, use `HfApi.upload_file()` for better performance. Args: path1 (`str`): Source path to copy from. path2 (`str`): Destination path to copy to. revision (`str`, *optional*): The git revision to copy from. rz)Copy from/to buckets is not available yetzCopy z to rrrH)Zsrc_path_in_reporBZ src_revision)r@r?rArrrrNZpath_or_fileobjrBr@rrr?rArrr)rrrPrr?r@rvrrArKrrBopenread upload_filerrrr7) r4rrrArZZresolved_path1Zresolved_path2Z same_reporfcontentr5r5r6cp_file9sN      zHfFileSystem.cp_filecKs|j|fii|ddi}d|vr&|ddur!td|d|djSd|vr2|dr2|dSd|vr>|dr>|dStd |d |d ) ak Get the last modified time of a file. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.modified). Args: path (`str`): Path to the file. Returns: `datetime`: Last modified time of the file. rtTrNz9'modified' is not implemented for repository paths like 'rrrz'Cannot determined 'modified' for path 'z ' (info: ))rrdate)r4r/rZrr5r5r6modifiedus   zHfFileSystem.modifiedc sN|j|d}||d|jdur|jnd}|jsRdddt|tr+dd<t|trQ|rQ|jj|j |j |j d d }idt |j |j|jd d nt|tr{|}|j||d fdd|j|D} | svtd| dnd|}|s||jvr||||jvrfdd|j|D} | std| d|sdus|rrddur|jj|j |j||j |j d} | std| d} t|j |j |j d|jd} t| tr| d| j| jd| j| j| j| j| jdn| d| jdd| j| jd|sfdddDdus%JS)aS Get information about a file or directory. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.info). > [!WARNING] > Note: When possible, use `HfApi.get_paths_info()` or `HfApi.repo_info()` for better performance > (or `HfApi.get_bucket_paths_info()` or `HfApi.bucket_info()` for buckets) Args: path (`str`): Path to get info for. refresh (`bool`, *optional*): If True, bypass the cache and fetch the latest data. Defaults to False. revision (`str`, *optional*): The git revision to get info from. Returns: `dict[str, Any]`: Dictionary containing file information (type, size, commit info, etc.). rrtNFrr)rrrcr)r?rA)oidtitler)rr)rcrrr5rrr5r6rrz%HfFileSystem.info..crrr5rrr5r6rr)rrAr?rH)rBrGr1rrrcr{r5r5)r|krr5r6rrz%HfFileSystem.info..) rr7rKrtr/rr>rvZlist_repo_commitsr@r?rAr% commit_idrZ created_atrPrrryrZget_paths_inforBrGr&rrrrrrr) r4r/rrArZrrtrrZout1Z paths_inforrr5)rr/r6rs           &    zHfFileSystem.infocKsDz|ddr |||j|fi|WdSty!YdSw)a Check if a file exists. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.exists). > [!WARNING] > Note: When possible, use `HfApi.file_exists()` for better performance. Args: path (`str`): Path to check. Returns: `bool`: True if file exists, False otherwise. rFT)rKrrOSError)r4r/rZr5r5r6existss   zHfFileSystem.existscC*z ||ddkWStyYdSw)ai Check if a path is a directory. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.isdir). Args: path (`str`): Path to check. Returns: `bool`: True if path is a directory, False otherwise. rcrFrr r4r/r5r5r6r   zHfFileSystem.isdircCr )a` Check if a path is a file. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.isfile). Args: path (`str`): Path to check. Returns: `bool`: True if path is a file, False otherwise. rcrFr rr5r5r6isfile'rzHfFileSystem.isfilecCsl||}t|tr|jd|jdt|j}nt|j|j |j |j |jd}| |r4| ddd}|S)z Get the HTTP URL of the given path. Args: path (`str`): Path to get URL for. Returns: `str`: HTTP URL to access the file or directory on the Hub. z /buckets/z /resolve/)r?rArqz/tree/r)rrrPrqrQrr/r r@rBr?rArreplace)r4r/rurlr5r5r6r9s   zHfFileSystem.urlc sT|d}t|dh}t|ttfrt|dkr)tj||f||d|St |r0|}n| |r>t j |dddSt|t tfrPt j t j|ddd}|dur]t|d}d}|} |j||d } |j||d d } || z+t|| ||| d|jt|tr|jndd || W|r|dSdS|r|ww) ad Copy single remote file to local. > [!WARNING] > Note: When possible, use `HfApi.hf_hub_download()` or `HfApi.download_bucket_files` for better performance. Args: rpath (`str`): Remote path to download from. lpath (`str`): Local path to download to. callback (`Callback`, *optional*): Optional callback to track download progress. Defaults to no callback. outfile (`IO`, *optional*): Optional file-like object to write to. If provided, `lpath` is ignored. rAr)callbackoutfileT)exist_okNFwbrr)r temp_fileZdisplayed_filename expected_sizeZ resume_sizeheadersZ _tqdm_bar)rKsetkeysrrrrrUget_filerrrmakedirsr<r r/dirnamertellrrZset_sizer!rr7rv_build_hf_headersZtqdmseekclose) r4ZrpathZlpathrrrZrAZunhandled_kwargsZ close_fileZ initial_posZresolve_remote_pathrr[r5r6rSsF        zHfFileSystem.get_filecCtd)zA context within which files are committed together upon exit Requires the file class to implement `.commit()` and `.discard()` for the normal and exception cases. (Transactional commits are not supported.rr3r5r5r6 transactions zHfFileSystem.transactioncCr#)z@Begin write transaction for deferring files, non-context versionr$r%r3r5r5r6start_transactionszHfFileSystem.start_transactioncCstt||j|j|ffSrT) make_instancercrhrirer3r5r5r6 __reduce__s zHfFileSystem.__reduce__cCst|jt|jt|jdS)N)ryrwrx)rryrwrxr3r5r5r6resz HfFileSystem._get_instance_staterT)rNN)FNN)TFN)FFNNN)NFFFN)FNr0N)6r8r9r:r;Z root_markerprotocolrr<rboolintrV classmethodr^tuple Exceptionrrr>rPrrrrrrdictr rrr r"r#rr rrrrrrrr rrrrrpropertyr&r'r)reror5r5r[r6rpsB      j$  3 .  ,($ =<(n>   rp) metaclasscseZdZddededeeffdd ZfddZd ed ed e fd d Z dddZ dde d dfddZ dfdd Zd efddZZS)rNfsr/rAc stz |j||d|_Wnty'}zd|ddvr"t|d|d}~wwtj||jfi||dS)NrwrrHB. Make sure the repository and revision exist before writing data.)rrrrKrUrVr7)r4r4r/rArZrr[r5r6rVszHfFileSystemFile.__init__cst|dsdStSNr)hasattrrU__del__r3r[r5r6r9s  zHfFileSystemFile.__del__startendr0cCsLdd|d|di|jj}|}td||tjd}t||jS)Nrangebytes=-rGETrr) r4rvr rr*rHF_HUB_DOWNLOAD_TIMEOUTr)r)r4r:r;rrrr5r5r6 _fetch_ranges zHfFileSystemFile._fetch_rangecCstjddd|_dS)Nzhffs-F)rr)tempfileNamedTemporaryFilerr3r5r5r6_initiate_uploadsz!HfFileSystemFile._initiate_uploadFfinalc Cs|jd|j}|j||rg|jt|jtr1|j j j |jj |jj |jjfgdn#|j j j|jj |jj|jj|j j|jj|jj|jd|jddt|jj |j j|jddSdS)Nr)addrrrr)bufferr!rrwriter"rrrPr4rvrrQrr/rrBr@rrr?rArZrKrremoverr7)r4rGblockr5r5r6 _upload_chunks0         zHfFileSystemFile._upload_chunkrcs|jdkr;|dus |dkr;|jdkr;|jj|jddd}|}|jt|7_|WdS1s6wYt|S)zRead remote file. If `length` is not provided or is -1, the entire file is downloaded and read. On POSIX systems the file is loaded in memory directly. Otherwise, the file is downloaded to a temporary file and read from there. rNrr)rs)rlocr4rr/rrrU)r4lengthrrr[r5r6rs$ zHfFileSystemFile.readcC|j|jSrTr4rr/r3r5r5r6rzHfFileSystemFile.urlrTr*)Fr)r8r9r:rpr<rrVr9r-bytesrCrFr,rMrrror5r5r[r6rs    rc seZdZ    d!dedededeed ed ef fd d Zd"d edefddZd#defddZ d#de e dede fddZ defddZ fddZddZdd ZZS)$rrNrnoner4r/rrArs cache_typec s|dkr td||dkrtd|d|vr"td|dz |j||d|_Wn"tyN}zd|d d vrDt|d |WYd}~nd}~ww|jdd |_tj||jf|||d |d|_ |t |_ d|_ t |_dS)Nrz:HfFileSystemStreamFile only supports block_size=0 but got rUz?HfFileSystemStreamFile only supports cache_type='none' but got r5z;HfFileSystemStreamFile only supports reading but got mode='rrrrHr6)rrrrsrV)rrrrrKr7detailsrUrVresponser _exit_stack_stream_iterator bytearray_stream_buffer) r4r4r/rrArsrVrZrr[r5r6rVs>    zHfFileSystemStreamFile.__init__rNwhencecCs2|dkr |dkr dS||jkr|dkrdStd)NrrzCannot seek streaming HF file)rNr)r4rNr^r5r5r6r!"s zHfFileSystemStreamFile.seekrrOcCs|jdur |d} z |jdus|jdurWdS||j|}|jt|7_|WStyH|jdur=|j|r@|d}Ynwq )zRead the remote file. If the file is already open, we reuse the connection. Otherwise, open a new connection and read from it. If reading the stream fails, we retry with a new connection. NFT)rY_open_connectionr[_read_from_streamrNrr0r")r4rOZ retried_oncerr5r5r6r)s&    zHfFileSystemStreamFile.readiteratorr0cCs|dkrdS|dkr"t|j}|j|D]}||qt|S|t|jkr:t|jd|}|jd|=|St|j}|j|D]+}|t|}|t|krZ||qF||d||j||dt|St|S)zRead up to `length` bytes from stream buffer and stream. If length < 0, read until EOF. If EOF is reached before length, fewer bytes may be returned. rr_N)r\r]rrrTr)r4rbrObufchunkresultZneedr5r5r6raEs,         z(HfFileSystemStreamFile._read_from_streamcCrPrTrQr3r5r5r6rgrRzHfFileSystemStreamFile.urlcs"t|dsdS|jtSr7)r8rZr"rUr9r3r[r5r6r9js   zHfFileSystemStreamFile.__del__cCst|j|j|j|j|jjffSrT)reopenr4r/r blocksizecacherr3r5r5r6r)qsz!HfFileSystemStreamFile.__reduce__c Cs|jd|_|}|jj}|jdkr d|jd|d<|j t d||t j d|_ zt|j WntyR}z|j jdkrMd|_ WYd}~dSd}~ww|j |_dS) z%Open a connection to the remote file.Nrr=r>Ranger?r@i)r]rr[rr4rvr rNrZ enter_contextr+rrArYr)r status_codeZ iter_bytes)r4rrrr5r5r6r`ts0     z'HfFileSystemStreamFile._open_connection)rNrrU)rrS)r8r9r:rpr<rr-rVr!rr rTrarr9r)r`ror5r5r[r6rs2#" rrAr0cCst|r|St|SrT)rr safe_quoterr5r5r6rMsrMscCs t|ddS)NrH)safe)r)rmr5r5r6rls rlr/rcCsN|}t|tr |d}nt|tr|d}n t|tr"|d}t||)Nz (repository not found)z (revision not found)z (invalid repository id))rrrr(r)r/rmsgr5r5r6rs       rr4rrsrVcCs|j||||dS)NrW)r)r4r/rrsrVr5r5r6rfsrfcCs0||i|}|D] \}}t|||q |SrT)rfrg)rXrYrZrlr4rmrnr5r5r6r(sr()ZrrerDr_ collectionsr contextlibrcopyr dataclassesrrr itertoolsrpathlibr r typingr r r rrr urllib.parserrZfsspecZhttpxZfsspec.callbacksrrrZ fsspec.utilsrrHrZ _commit_apirrerrorsrrrrrZ file_downloadr r!Zhf_apir"r#r$r%r&r'utilsr(r)r*r+Zutils.insecure_hashlibr,compileVERBOSErr-r>rPrcZAbstractFileSystemrRr=rSrpspecZAbstractBufferedFilerrr<rMrlr0rr-rfr(Zhffsr5r5r5r6sl              2(K