o 0iϨ'@s dZddlZddlZddlZddlZddlZddlZddlmZm Z ddl m Z m Z ddl m Z ddlmZmZmZmZmZmZddlmZmZdd lmZdd lmZmZmZmZdd lm Z erhdd l!m"Z"e#e$Z%d Z&dZ'de(de)e(e(ffddZ*eGdddZ+eGdddZ,eGdddZ-eddGdddZ.eGdddZ/eGdd d Z0eGd!d"d"Z1de(de)e(e(ffd#d$Z2de(de3fd%d&Z4eGd'd(d(Z5eGd)d*d*Z6Gd+d,d,Z7d-e(de8e)e(e(ffd.d/Z9d0e(dee)e(e:e;ffd1d2Zdd<de(d=ed>d?e:d@e;dAe:dBe;dCe(dDe(dEe3dFe3dGe3dHeede5fdIdJZ? K K K K K  dkdLe(dMe(d3d4dNe3dFe3dEe3dOe3dGe3dPee7dQeede6fdRdSZ@dTe6ddfdUdVZAdTe6dWe(ddfdXdYZBdWe(de6fdZd[ZCdldTe6d3d4d\e3dQeeddf d]d^ZDdTe6ddfd_d`ZE  dmdKdKdKdKdKddddddKdKdKddadLee(dMee(d3d4dNe3dFe3dEe3dOe3dGe3dbee8e(dcee8e(ddee(dTee(deee(dfe3d\e3dge3dhee3e(dfde6f$didjZFdS)nzzShared logic for bucket operations. This module contains the core buckets logic used by both the CLI and the Python API. N) dataclassfield)datetimetimezone)Path) TYPE_CHECKINGAnyIteratorLiteralOptionalUnion) constantslogging)BucketNotFoundError) XetFileDatadisable_progress_barsenable_progress_barsparse_datetime) StatusLineHfApi hf://buckets/pathreturncCsn|dd}t|dks|dr|dstd|d|dd|d}t|dkr1|dnd}||fS)zSplit 'namespace/name(/optional/prefix)' into ('namespace/name', 'prefix'). Returns (bucket_id, prefix) where prefix may be empty string. Raises ValueError if path doesn't contain at least namespace/name. /rr zInvalid bucket path: 'z)'. Expected format: namespace/bucket_name)splitlen ValueError)rparts bucket_idprefixr%R/home/kim/smarthome/.venv/lib/python3.10/site-packages/huggingface_hub/_buckets.py_split_bucket_id_and_prefix5s r'c@sBeZdZUdZeed<eed<eed<eed<eed<ddZ d S) BucketInfoa Contains information about a bucket on the Hub. This object is returned by [`bucket_info`] and [`list_buckets`]. Attributes: id (`str`): ID of the bucket. private (`bool`): Is the bucket private. created_at (`datetime`): Date of creation of the bucket on the Hub. size (`int`): Size of the bucket in bytes. total_files (`int`): Total number of files in the bucket. idprivate created_atsize total_filescKsV|d|_|d|_t|d|_|d|_|d|_|jjdi|dS)Nr)r*Z createdAtr,Z totalFilesr%) popr)r*rr+r,r-__dict__update)selfkwargsr%r%r&__init__[s    zBucketInfo.__init__N) __name__ __module__ __qualname____doc__str__annotations__boolrintr3r%r%r%r&r(Cs  r(c@seZdZUeeeefed<eed<eddZ e eed<eddZ e e ed<eddZ e ed <eddZe eed <dd d ZdS)_BucketAddFilesource destinationN)defaultxet_hashr,Finitmtime content_typercCsxd|_t|jttfrt|jd|_|jdur"t|jd|_tt|jt s2t j |jdnt d|_dS)Nrr)rD isinstancer=r8r mimetypes guess_typer>r;bytesosrgetmtimetimerC)r1r%r%r& __post_init__ns ( z_BucketAddFile.__post_init__rN)r4r5r6r r8rrHr9rr@r r,r;rCrDrLr%r%r%r&r<ds r<c@seZdZUeed<dS)_BucketDeleteFilerN)r4r5r6r8r9r%r%r%r&rNzs rNT)frozenc@s"eZdZUdZeed<eed<dS)BucketFileMetadataa%Data structure containing information about a file in a bucket. Returned by [`get_bucket_file_metadata`]. Args: size (`int`): Size of the file in bytes. xet_file_data (`XetFileData`): Xet information for the file (hash and refresh route). r,Z xet_file_dataN)r4r5r6r7r;r9rr%r%r%r&rPs  rPc@sfeZdZUdZeed<dZeed<eddZeed<eddZ eed<eddZ eed <dd d Z d S) BucketUrlaDescribes a bucket URL on the Hub. `BucketUrl` is returned by [`create_bucket`]. At initialization, the URL is parsed to populate properties: - endpoint (`str`) - namespace (`str`) - bucket_id (`str`) - url (`str`) - handle (`str`) Args: url (`str`): String value of the bucket url. endpoint (`str`, *optional*): Endpoint of the Hub. Defaults to . urlrendpointFrA namespacer#handlerNcCs|jptj|_|j|jdd}|dr|tdd}t|\}}|r/t d|j| dd|_ ||_ d|j |_ dS)Nrrzbuckets/zUnable to parse bucket URL: rr)rSrZENDPOINTrRreplacestrip startswithr r'r!rrTr#rU)r1Zurl_pathr#r$r%r%r&rLs  zBucketUrl.__post_init__rM) r4r5r6r7r8r9rSrrTr#rUrLr%r%r%r&rQs  rQc@sVeZdZUdZeded<eed<eed<eed<ee ed<ee ed<d d Z d S) BucketFilez Contains information about a file in a bucket on the Hub. This object is returned by [`list_bucket_tree`]. Similar to [`RepoFile`] but for files in buckets. filetyperr,r@rC uploaded_atcKsv|d|_|d|_|d|_|d|_|dd}|r$t|nd|_|dd}|r6t||_dSd|_dS)Nr[rr,ZxetHashrC uploadedAt)r.r[rr,r@rrCr\)r1r2rCr\r%r%r&r3s      zBucketFile.__init__N) r4r5r6r7r r9r8r;r rr3r%r%r%r&rYs     rYc@s:eZdZUdZeded<eed<eeed<ddZ dS) BucketFolderz Contains information about a directory in a bucket on the Hub. This object is returned by [`list_bucket_tree`]. Similar to [`RepoFolder`] but for directories in buckets. directoryr[rr\cKs`|d|_|d|_|ddp|dd}|r+t|tr$||_dSt||_dSd|_dS)Nr[rr]r\)r.r[rrErrr\)r1r2r\r%r%r&r3s  zBucketFolder.__init__N) r4r5r6r7r r9r8r rr3r%r%r%r&r^s    r^cCs,|tstd|dtt|tS)zParse a bucket path like hf://buckets/namespace/bucket_name/prefix into (bucket_id, prefix). Returns: tuple: (bucket_id, prefix) where bucket_id is "namespace/bucket_name" and prefix may be empty string. zInvalid bucket path: z. Must start with )rX BUCKET_PREFIXr!r' removeprefixrr%r%r&_parse_bucket_paths rccCs |tS)z!Check if a path is a bucket path.)rXr`rbr%r%r&_is_bucket_paths rdc@sreZdZUdZeded<eed<dZee ed<dZ eed<dZ eeed <dZ eeed <dZ eeed <dS) SyncOperationz,Represents a sync operation to be performed.)uploaddownloaddeleteskipactionrNr,rreason local_mtime remote_mtime bucket_file)r4r5r6r7r r9r8r,r r;rkrlrmrnrYr%r%r%r&res   rec@s^eZdZUdZeed<eed<eed<eedZee ed<de ee e efffdd Z d S) SyncPlanz Represents a complete sync plan.r=dest timestamp)default_factory operationsrcCsttdd|jD}tdd|jD}tdd|jD}tdd|jD}tdd|jD}|||||dS)Ncs|] }|jdkrdVqdS)rfr Nrj.0opr%r%r& z#SyncPlan.summary..csrt)rgr Nrurvr%r%r&ryrzcsrt)rhr Nrurvr%r%r&ryrzcsrt)rir Nrurvr%r%r&ryrzcss$|] }|jdvr|jp dVqdS)rfrgrN)rjr,rvr%r%r&rys")uploads downloadsdeletesskips total_size)sumrs)r1r|r}r~rrr%r%r&summaryszSyncPlan.summaryN)r4r5r6r7r8r9rlistrsredictr r;rr%r%r%r&ros "roc @s`eZdZdZ   d deeedeeedeeeeeffddZded e fd d Z dS) FilterMatcherz4Matches file paths against include/exclude patterns.Ninclude_patternsexclude_patterns filter_rulescCs"|pg|_|pg|_|p g|_dS)aInitialize the filter matcher. Args: include_patterns: Patterns to include (from --include) exclude_patterns: Patterns to exclude (from --exclude) filter_rules: Rules from filter file as list of ("+"/"-", pattern) tuples Nrrr)r1rrrr%r%r&r31s zFilterMatcher.__init__rrcCsr|jD]\}}t||r|dkSq|jD] }t||r"dSq|jD] }t||r1dSq&|jr7dSdS)zCheck if a path should be included based on the filter rules. Filtering rules: - Filters are evaluated in order, first matching rule decides - If no rules match, include by default (unless include patterns are specified) +FT)rfnmatchrr)r1rsignpatternr%r%r&matchesBs      zFilterMatcher.matches)NNN) r4r5r6r7r rr8tupler3r:rr%r%r%r&r.s   r filter_filecCsg}t|G}|D];}|}|r|drq |dr*|d|ddfq |dr=|d|ddfq |d|fq Wd|S1sPwY|S)zParse a filter file and return a list of (sign, pattern) tuples. Filter file format: - Lines starting with "+" are include patterns - Lines starting with "-" are exclude patterns - Empty lines and lines starting with "#" are ignored #rr N-)openrWrXappend)rrulesfliner%r%r&_parse_filter_file_s       r local_pathc cstj|}tj|std|t|D]3\}}}|D]+}tj||}tj||}|tj d}tj |}tj |d}|||fVq qdS)zsList all files in a local directory. Yields: tuple: (relative_path, size, mtime_ms) for each file z Local path must be a directory: rrN) rIrabspathisdirr!walkjoinrelpathrVsepgetsizerJ) rroot_filesfilename full_pathrel_pathr,mtime_msr%r%r&_list_local_files|s   rapirr#r$ccs|j||pdddD]I}t|trq |j}|r>||dr*|t|dd}n||kr=d|vr:|dddn|}nq |}|jrJ|jdnd}||j ||fVq dS) zList all files in a bucket with a given prefix. Yields: tuple: (relative_path, size, mtime_ms, bucket_file) for each file. bucket_file is the BucketFile object from list_bucket_tree. NT)r$ recursiverr rr) Zlist_bucket_treerEr^rrXr rsplitrCrqr,)rr#r$itemrrrr%r%r&_list_remote_filess rrcCstj|dtjdS)z3Convert mtime in milliseconds to ISO format string.r)tz)r fromtimestamprutc isoformat)rr%r%r& _mtime_to_isosr)rnrjr{ source_size source_mtime dest_size dest_mtimesource_newer_labeldest_newer_label ignore_sizes ignore_timesignore_existingrnc Cst|dkr|n|} t|dkr|n|} ||| | d}| r'td ddd|S||k}||tk}|rV|r@td ||| d|S||tk}|rJ|nd}td d|d|S| ro|retd |d| d|Std dd d|S|ss|r|rwdn|}td ||| d|Std dd d|S) aCompare source and dest files and return the appropriate sync operation. This is a unified helper for both upload and download directions. Args: path: Relative file path action: "upload" or "download" source_size: Size of the source file (bytes) source_mtime: Mtime of the source file (milliseconds) dest_size: Size of the destination file (bytes) dest_mtime: Mtime of the destination file (milliseconds) source_newer_label: Label when source is newer (e.g., "local newer" or "remote newer") dest_newer_label: Label when dest is newer (e.g., "remote newer" or "local newer") ignore_sizes: Only compare mtime ignore_times: Only compare size ignore_existing: Skip files that exist on receiver bucket_file: BucketFile object (for downloads only) Returns: SyncOperation describing the action to take rf)rr,rlrmriz&exists on receiver (--ignore-existing))rjrk)rjrkrnz same mtimez size differsz same sizeZ identicalNr%)rre_SYNC_TIME_WINDOW_MS)rrjrrrrrrrrrrnZlocal_mtime_isoZremote_mtime_isoZ base_kwargsZ size_differsZ source_newerZ dest_newer skip_reasonrkr%r%r&_compare_files_for_syncs2$    rFr=rprhexistingfilter_matcherstatusc "Cs|pt}t| o t|} t|ot| } | s| stdt||ttjd} d} | rt j |}t |\}}t j |sJtd|i}t|D]\}}}||r`||f||<| rm| dt|dqP| r{| dt|di}| rz||j} Wn tyYnwz4t|||D]+\}}}}||r||f||<| r| durd| nd}| d t||dqWntytd |d Ynw| r| d t|dt|t|B}| r| d t|d t|D]}||}||}|rA|sA|r,| jt d||ddt!|ddq| jt d||ddt!|ddq|rd|rd|\}}|\}}| jt"|d||||dd|||d q|s|r|r| jt d||ddt!|ddq| St |\}}t j |}i}i} | rz||j} Wn tyYnwt|||D]3\}}}}!||r||f||<|!| |<| r| durd| nd}| d t||dq| r| d t|di}t j |r t|D] \}}}||r||f||<| r| dt|dq| r.| dt|dt|t|B}| rH| d t|d t|D]}||}||}|r|s|rv| jt d||ddt!|ddqL| jt d||ddt!|d| |dqL|r|r|\}}|\}}| jt"|d||||dd|||| |d qL|s|r|r| jt d||ddt!|ddqL| S)zCompute the sync plan by comparing source and destination. Returns: SyncPlan with all operations to be performed z[One of source or dest must be a bucket path (hf://buckets/...) and the other must be local.r=rprqNzSource must be a directory: zScanning local directory (z files)rrzScanning remote bucket (zBucket 'z' not found, treating as empty.zComparing files (z paths)rirznew file (--existing)r )rjrr,rkrlrfznew filez local newerz remote newer) rrjrrrrrrrrrrhznot in source (--delete))rjrr,rkrmrg)rjrr,rkrmrn) rrjrrrrrrrrrrn)#rrdr!rornowrrrrIrrrcrrrr0r doneZ bucket_infor- Exceptionrrloggerdebugsetkeyssortedgetrsrrerr)"r=rprrhrrrrrr is_upload is_downloadplanZ remote_totalrr#r$Z local_filesrr,rZ remote_filesr total_strZ all_pathsrZ local_infoZ remote_infoZ local_sizerlZ remote_sizermZbucket_file_maprnr%r%r&_compute_sync_plans|                   k               rrcCsd|j|j|j|d}|t|d|jD]4}d|j|j |j d}|j dur0|j |d<|j dur:|j |d<|j durD|j |d <|t|dqdS) z1Write a sync plan as JSONL to a file-like object.header)r[r=rprqr operation)r[rjrrkNr,rlrm)r=rprqrwritejsondumpsrsrjrrkr,rlrm)rrrrxop_dictr%r%r& _write_plans*       r plan_filecCs:t|d}t||WddS1swYdS)z!Save a sync plan to a JSONL file.wN)rr)rrrr%r%r& _save_plans  "rc Cst| }|}Wdn1swY|s!td|t|d}|ddkr3tdt|d|d|d d }|d dD].}t|}|dd krUqF|jt |d |d|d|dd|d|ddqF|S)z#Load a sync plan from a JSONL file.NzEmpty plan file: rr[rz0Invalid plan file: expected header as first liner=rprqrr rrjrr,rkrrlrm)rjrr,rkrlrm) r readlinesr!rloadsrrorsrre)rrlinesrrrrr%r%r& _load_plans8     rverbosec Cst|j o t|j}t|jot|j }|rtj|j}t|j\}}|d}g} g} |jD]n} | j dkretj || j} |rK|d| jn| j} |r]t d| jd| j d| | | fq2| j dkr|rt|d| jn| j} |rt d| jd| j d| | q2| j dkr|rt d | jd| j dq2| s| r|rg}| r| d t| d | r| d t| d |d ||j|| pd| pdddSdS|rt|j\}}|d}tj|j}tj|ddg}g}|jD]} | j dkrStj || j} tjtj| dd|r,t d| jd| j d| jdur<| | j| fq|rG|d| jn| j} | | | fq| j dkrxtj || j} |rqt d| jd| j d| | q| j dkr|rt d | jd| j dqt|dkr|r|dt|d ||||r|r|dt|d|D]9}tj|rt|tj|}||krz t|tj|}Wn tyYnw||ksאqdSdS)zExecute a sync plan.rrfz Uploading: z ()rhz Deleting: riz Skipping: z uploading z filesz deleting z, N)addrhT)exist_okrgz Downloading: rz Downloading z Deleting z local files)rdr=rprIrrrcrstriprsrjrprintrkrr r capitalizeZbatch_bucket_filesmakedirsdirnamernZdownload_bucket_filesexistsremovermdirOSError)rrrrrrrr#r$Z add_filesZ delete_pathsrx local_fileZ remote_pathr"Zdownload_filesZ delete_files file_pathparentr%r%r& _execute_plan9s                  *rcCsl|}td|jd|jtd|dtd|dtd|dtd |d d S) z!Print a summary of the sync plan.z Sync plan: z -> z Uploads: r|z Downloads: r}z Deletes: r~z Skips: rN)rrr=rp)rrr%r%r&_print_plan_summarys r)rhrrrrincludeexclude filter_fromrapplydry_runrquiettokenrrrrrrrc Cs|durddlm}||d}| r|s|rtd| dur!td|r'td|r-td|r3td |r9td | r?td | rEtd |rKtd |rQtd| rWtdt| }t| d}|skt|td|rptzt||||dW|rt n|rt ww|std|S|r|stdt |}t |}|r|rtd|s|std|r|rtd|r|rtd| r| rtd|rt j |rt j |std|n t j |std|d}| rt| }t|| |d}t| o| d}t||||||||||d }| rt|tj|S| r3t|| |s1t|td| |S|s:t||}|dd kr\|d!d kr\|d"d kr\|sZtd#|S|sctd$|ritzt||||dW|ryt n|rt ww|std|S)%aSync files between a local directory and a bucket. This is equivalent to the ``hf buckets sync`` CLI command. One of ``source`` or ``dest`` must be a bucket path (``hf://buckets/...``) and the other must be a local directory path. Args: source (`str`, *optional*): Source path: local directory or ``hf://buckets/namespace/bucket_name(/prefix)``. Required unless using ``apply``. dest (`str`, *optional*): Destination path: local directory or ``hf://buckets/namespace/bucket_name(/prefix)``. Required unless using ``apply``. api ([`HfApi`]): The HfApi instance to use for API calls. delete (`bool`, *optional*, defaults to `False`): Delete destination files not present in source. ignore_times (`bool`, *optional*, defaults to `False`): Skip files only based on size, ignoring modification times. ignore_sizes (`bool`, *optional*, defaults to `False`): Skip files only based on modification times, ignoring sizes. existing (`bool`, *optional*, defaults to `False`): Skip creating new files on receiver (only update existing files). ignore_existing (`bool`, *optional*, defaults to `False`): Skip updating files that exist on receiver (only create new files). include (`list[str]`, *optional*): Include files matching patterns (fnmatch-style). exclude (`list[str]`, *optional*): Exclude files matching patterns (fnmatch-style). filter_from (`str`, *optional*): Path to a filter file with include/exclude rules. plan (`str`, *optional*): Save sync plan to this JSONL file instead of executing. apply (`str`, *optional*): Apply a previously saved plan file. When set, ``source`` and ``dest`` are not needed. dry_run (`bool`, *optional*, defaults to `False`): Print sync plan to stdout as JSONL without executing. verbose (`bool`, *optional*, defaults to `False`): Show detailed per-file operations. quiet (`bool`, *optional*, defaults to `False`): Suppress all output and progress bars. token (Union[bool, str, None], optional): A valid user access token. If not provided, the locally saved token will be used. Returns: [`SyncPlan`]: The computed (or loaded) sync plan. Raises: `ValueError`: If arguments are invalid (e.g., both paths are remote, conflicting options). Example: ```python >>> from huggingface_hub import HfApi >>> api = HfApi() # Upload local directory to bucket >>> api.sync_bucket("./data", "hf://buckets/username/my-bucket") # Download bucket to local directory >>> api.sync_bucket("hf://buckets/username/my-bucket", "./data") # Sync with delete and filtering >>> api.sync_bucket( ... "./data", ... "hf://buckets/username/my-bucket", ... delete=True, ... include=["*.safetensors"], ... ) # Dry run: preview what would be synced >>> plan = api.sync_bucket("./data", "hf://buckets/username/my-bucket", dry_run=True) >>> plan.summary() {'uploads': 3, 'downloads': 0, 'deletes': 0, 'skips': 1, 'total_size': 4096} # Save plan for review, then apply >>> api.sync_bucket("./data", "hf://buckets/username/my-bucket", plan="sync-plan.jsonl") >>> api.sync_bucket(apply="sync-plan.jsonl") ``` Nr r)rz,Cannot specify source/dest when using apply.z#Cannot specify both plan and apply.z'Cannot specify delete when using apply.z-Cannot specify ignore_times when using apply.z-Cannot specify ignore_sizes when using apply.z(Cannot specify include when using apply.z(Cannot specify exclude when using apply.z,Cannot specify filter_from when using apply.z)Cannot specify existing when using apply.z0Cannot specify ignore_existing when using apply.z(Cannot specify dry_run when using apply.)enabledzExecuting plan...)rrzSync completed.z7Both source and dest are required (unless using apply).z?Remote to remote sync is not supported. One path must be local.z?One of source or dest must be a bucket path (hf://buckets/...).z2Cannot specify both ignore_times and ignore_sizes.z1Cannot specify both existing and ignore_existing.z%Cannot specify both dry_run and plan.z!Destination must be a directory: z&Source must be an existing directory: r) r=rprrhrrrrrrzPlan saved to: r|rr}r~zNothing to sync.z Syncing...)hf_apirr!rrrrrrrrdrIrrrrrrrsysstdoutrr)r=rprrhrrrrrrrrrrrrrrZ sync_planrZsource_is_bucketZdest_is_bucketrrrr%r%r&sync_bucket_internalsc      *r)FFFFFNN)FN)NN)Gr7rrrFrIrrK dataclassesrrrrpathlibrtypingrrr r r r rrrerrorsrutilsrrrrZutils._terminalrrrZ get_loggerr4rr`rr8rr'r(r<rNrPrQrYr^rcr:rdrerorrrr;floatrrrrrrrrrrrr%r%r%r&sH       (   1 *"      L   l$+c