o 0iim@sdZddlZddlZddlZddlZddlmZddlmZddl m Z ddl m Z m Z mZmZmZmZmZmZddlZddlmZdd lmZmZmZmZmZmZmZmZdd l m!Z!m"Z"d d l#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*e*d dZ+eddGdddZ,e-dZ.hdZ/hdZ0hdZ1e-dZ2dddddZ3iZ4e5e1D]Z6e6e4e6<e6de4e6d<e6de4e6d<qe de4e7e8d Z9eddGd!d"d"Z:eeeefZ;eeed#eeeZHd?e7d$eIee;geIe d@ffeAffdAdBZJd)edCeKe7d$e,fdDdEZLe+jMdFgdGdHddIde$jNdIddfdJe ee7ejOdKdfdLe eAejOdMdfdNe eeKe7ejOdOdPdQdfdRe e$ejOdSdfdTe eAejOdUdVdWdfdXe ee9ejOdYdfdZe eePejOd[dfd$dfd\d]ZQe+jMgd^dH  I Id|dCe eKe7ejRd_dfdJe ee7ejOdKdfd`e eAejOdadbdcdfdde eAejOdedfd$df dfdgZSe+jMdhdigdH  I Id|dJe ee7ejOdKdfd`e eAejOdadbdcdfdde eAejOdedfd$dfdjdkZTe+jMgdldHe&jUddddIdIdfdme%dne&doe'dJe ee7ejOdpdfdqe ee7ejOdrdfdse eAejOdtdudfdve eAejOdwdxdfdye(d$dfdzd{ZVdS)}zHContains the 'hf cache' command group with cache management subcommands.N) defaultdict) dataclass)Enum) AnnotatedAnyCallableDictListMappingOptionalTuple)CLIError)ANSICachedRepoInfoCachedRevisionInfo CacheNotFound HFCacheInfo _format_sizescan_cache_dirtabulate)parse_duration parse_size) OutputFormat RepoIdArg RepoTypeOpt RevisionOptTokenOpt get_hf_api typer_factoryzManage local cache directory.)helpT)frozenc@s>eZdZUeeed<eeeefed<e edfed<dS)_DeletionResolution revisionsselected.missingN) __name__ __module__ __qualname__ frozensetstr__annotations__dictrrtupler/r/S/home/kim/smarthome/.venv/lib/python3.10/site-packages/huggingface_hub/cli/cache.pyr#:s r#zA^(?P[a-zA-Z_]+)\s*(?P==|!=|>=|<=|>|<|=)\s*(?P.+)$><=<=!=>>=>accessedtyperefsmodifiedsize>r7namer:r;z.^(?P[a-zA-Z_]+)(?::(?Pasc|desc))?$descasc)r7r:r;r<z:ascZ_ascz:descZ_desc SortOptions)r8modulec@s*eZdZUdZeed<eed<eed<dS)CacheDeletionCountsz>Simple counters summarizing cache deletions for CLI messaging. repo_countpartial_revision_counttotal_revision_countN)r'r(r)__doc__intr,r/r/r/r0rAYs  rAselected_by_reporeturncCsdd}d}d}|D]\}}|t|7}t|t|jkr'|d7}|t|7}q ||}t|||S)z(Summarize deletions across repositories.rr)itemslenr$rA)rGrBZtotal_revisionsZrevisions_in_full_reposrepor$rCr/r/r0summarize_deletionsfs   rLc Cst|dddD]M}|jd|j}t||ddd}t|t|jkr1td|dq td|d|D]}d t|jpFd }td |j d |d |j q;q dS)zBPretty-print selected cache revisions during confirmation prompts.cS|j|jfSN repo_typerepo_idlowerrKr/r/r0zz0print_cache_selected_revisions..key/cS|jSrN commit_hashrevr/r/r0rT| - z (entire repo): z (detached)z z [z] N) sortedkeysrPrQrJr$printjoinr9r[size_on_disk_str)rGrKrepo_keyr$revisionr9r/r/r0print_cache_selected_revisionsxs ri hf_cache_infocCsLi}i}|jD]}|j}|||<|jD] }||f||j<qq||fS)zPCreate lookup tables so CLI commands can resolve repo ids and revisions quickly.)reposcache_idrRr$r[)rj repo_lookuprevision_lookuprKrgrhr/r/r0build_cache_indexs   roinclude_revisionscCsg}i}t|jddd}|D]+}tdd|jD||<|r3t|jdddD] }|||fq(q||dfq|rI|jddd||fS|jd dd||fS) z;Flatten cache metadata into rows consumed by `hf cache ls`.cSrMrNrOrSr/r/r0rTrUz'collect_cache_entries..rVcSsh|] }|jD]}|qqSr/)r9).0rhrefr/r/r0 sz(collect_cache_entries..cSrYrNrZr\r/r/r0rTr^NcSs&|dj|ddur|djfSdfS)Nrr)rlr[entryr/r/r0rTs cSs |djS)Nr)rlrur/r/r0rTs )rbrkr*r$appendsort)rjrpentries repo_refs_mapZ sorted_reposrKrhr/r/r0collect_cache_entriess"r{exprrzcst|}|std|d|d|d|dtvr8tdd|dttd tvrLtd d |dttd d kr[t fd dSdvrzt dt dt t dtdtffdd }|Sdkrdkrtdddt dt t dtdtffdd }|Sdkrtdd dt dt t dtdtffdd }|S)zqConvert a `hf cache ls` filter expression into the yes/no test we apply to each cache entry before displaying it.zInvalid filter expression: 'z'.rWopvaluezUnsupported operator 'z ' in filter ''. Must be one of .zUnsupported filter key '' in 'r;cs"t|dur |jS|jSrN)_compare_numeric size_on_diskrKrh_)r}size_thresholdr/r0rTs z&compile_cache_filter..>r7r:rKrhnowrHcs>dkr|jn |dur|jn|j}|durdSt||S)Nr7F) last_accessed last_modifiedr)rKrhr timestamp)rWr}secondsr/r0 _time_filtersz*compile_cache_filter.._time_filterr8r3z/Only '=' is supported for 'type' filters. Got 'rcs|jkSrN)rPrRr)expectedr/r0 _type_filtersz*compile_cache_filter.._type_filterz.Only '=' is supported for 'refs' filters. Got cs2|dur|jn|t}dd|DvS)NcSsg|]}|qSr/)rR)rqrrr/r/r0 z>compile_cache_filter.._refs_filter..)r9getr*rR)rKrhrr9)rz value_rawr/r0 _refs_filtersz*compile_cache_filter.._refs_filter)_FILTER_PATTERNmatchstrip ValueErrorgrouprR_ALLOWED_OPERATORSlist _FILTER_KEYSrrrr rfloatbool)r|rzrrrrr/)rrWr}rzrrrr0compile_cache_filters6    & "$rryc Csg}|D]=\}}|r'|durq|j|j|jt|j|j|j|jt|j d}n|j|j|j|j|jt| |t d}| |q|S)zGNormalize cache entries into serializable records for JSON/CSV exports.N)rQrPrh snapshot_pathrrrr9)rQrPrrrr9) rQrPr[r+rrrrrbr9rr*rw)ryrprzpayloadrKrhrecordr/r/r0_build_cache_export_payloads.   rc s|s|rdnd}t|dS|rgd}dd|D}n gd}fdd|D}tt||d d d |D}t|}|rQtd d |D}tdd |D} ntdd |D}tdd |D} d|d|dt| d} tt| dS)zBRender cache entries as a table and show a human-readable summary.zNo cached revisions found.zNo cached repositories found.N)IDZREVISIONSIZE LAST_MODIFIEDREFSc Ss@g|]\}}|dur|j|j|jd|jdt|jgqS)Nra)rlr[rfrjustlast_modified_strrerbr9)rqrKrhr/r/r0rs z-print_cache_entries_table..)rrZ LAST_ACCESSEDrrc sDg|]\}}|j|jd|jpd|jdt|tgqS)rrtra) rlrfrZlast_accessed_strrrerbrr*rqrKrrzr/r0r+s )headerscSsh|]\}}|qSr/r/rr/r/r0rs8rz,print_cache_entries_table..css |] \}}|durdVqdS)Nrr/rqrrhr/r/r0 ;sz,print_cache_entries_table..css"|] \}}|dur|jVqdSrNrrr/r/r0r< css|]}t|jVqdSrN)rJr$rqrKr/r/r0r>scs|]}|jVqdSrNrrr/r/r0r?z Found z repo(s) for a total of z revision(s) and z on disk.)rdrrJsumrrbold) ryrprzmessagerZ table_rowsZ unique_reposrBZrevision_countZ total_sizesummaryr/rr0print_cache_entries_tables.   rcCs0t|||d}tj|tjddtjddS)z7Dump cache entries as JSON for scripting or automation.rprzr)indent N)rjsondumpsysstdoutwrite)ryrprzrr/r/r0print_cache_entries_jsonEsrleftr}rightcCsT|durdS||k||k||k||k||k||kd}||vr&td|||S)z)Evaluate numeric comparisons for filters.NF)r3r4r5r2r6r1z)Unsupported numeric comparison operator: )r)rr}rZ comparisonsr/r/r0rNs r sort_expr.cst|}|std|d|d|d}tvr3tdd|dttd|r7|nt}|d k}d t d t t d fffd d }||fS)zConvert a `hf cache ls` sort expression into a key function for sorting entries. Returns: A tuple of (key_function, reverse_flag) where reverse_flag indicates whether to sort in descending order (True) or ascending order (False). zInvalid sort expression: 'z5'. Expected format: 'key' or 'key:asc' or 'key:desc'.rWorderzUnsupported sort key 'rrrr=rvrH.cs|\}}dkr|j}|fSdkr$|dur|j}|fS|j}|fSdkr8|jdur3|j}|fSd}|fSdkr]|durP|jdurK|j}|fSd}|fS|jdurX|jnd}|fStd)Nr<r;r7gr:zUnsupported sort key: )rlrRrrrr)rvrKrhr~rVr/r0 _sort_keyws. z%compile_cache_sort.._sort_key) _SORT_PATTERNrrrRrr _SORT_KEYSr_SORT_DEFAULT_ORDER CacheEntryr.r)rrZexplicit_orderrreverserr/rVr0compile_cache_sortbs  rtargetscCst|\}}tt}t}g}|D]V}|}|sq|} td| rE|| } | dur3||q| \} } ||  | | | j q|| } | durT||q| j D]} ||  | | | j qWqdd| D}t t||t|dS)z8Resolve the deletion targets into a deletion resolution.z[0-9a-fA-F]{40}NcSsi|] \}}|t|qSr/)r*)rqrKrevsr/r/r0 sz-_resolve_deletion_targets..r$r%r&)rorsetrrRre fullmatchrrwaddr[r$rIr#r*r.)rjrrmrnr%r$r&Z raw_targettargetZloweredrrKrhZ matched_repoZfrozen_selectedr/r/r0_resolve_deletion_targetss@        rz list | ls)z hf cache lszhf cache ls --revisionsz*hf cache ls --filter "size>1GB" --limit 20zhf cache ls --format json)ZexamplesF cache_dirz9Cache directory to scan (defaults to Hugging Face cache).r$zCInclude revisions in the output instead of aggregated repositories.filterz-fz--filterzZFilter entries (e.g. 'size>1GB', 'type=model', 'accessed>7d'). Can be used multiple times.formatzOutput format.quietz-qz--quietz-Print only IDs (repo IDs or revision hashes).rxaSort entries by key. Supported keys: 'accessed', 'modified', 'name', 'size'. Append ':asc' or ':desc' to explicitly set the order (e.g., 'modified:asc'). Defaults: 'accessed', 'modified', 'size' default to 'desc' (newest/biggest first); 'name' defaults to 'asc' (alphabetical).limitzSLimit the number of results returned. Returns only the top N entries after sorting.c szt|}Wnty}z td|j|d}~ww|pg} t||d\} z fdd| D} WntyG}ztt||d}~wwt | D] fdd| D} qN|rzt |j \} } | j | | dWnty}ztt||d}~ww|dur|dkrtd |d | d|} |r| D]\}}t |dur|jn|jqdStjttjti}||| |d S) z&List cached repositories or revisions.Cache directory not found: N)rpcsg|]}t|qSr/)r)rqr|rr/r0r szls..cs$g|]}|d|dr|qS)rrr/)rqrv)fnrr/r0rs$)rWrrz&Limit must be a positive integer, got rr)rrr rr{rtyperZ BadParameterr+timerr~rxrdr[rlrtablerrr)rr$rrrrxrrjexcfiltersryZ filter_fnsZ sort_key_fnrrKrh formattersr/)rrrzr0lssJ=   r)zhf cache rm model/gpt2zhf cache rm z hf cache rm model/gpt2 --dry-runzhf cache rm model/gpt2 --yeszQOne or more repo IDs (e.g. model/bert-base-uncased) or revision hashes to delete.yesz-yz--yeszSkip confirmation prompt.dry_runz,Preview deletions without removing anything.c Cszt|}Wnty}z td|j|d}~wwt||}|jr5td|jD] }td|q+t|jdkrFtdt j dd|j t |j}t |j} g} | jra| | jd| jrm| | jd | sx| | jd d | } td | d |jd t|j|rtddS|st jdddstddS|t |j} td| jd| jd|jd dS)z(Remove cached repositories or revisions.rNz2Could not find the following targets in the cache:r_rzNothing to delete.codez repo(s)z revision(s)z and About to delete z totalling rDry run: no files were deleted.zProceed with deletion?F)defaultzDeletion cancelled.Deleted z repo(s) and z revision(s); freed )rrr rrr&rdrJr$rExitdelete_revisionsrbrLr%rBrwrCrDreexpected_freed_size_strriconfirmexecute) rrrrrjr resolutionrvstrategycountsZ summary_partsZ summary_textr/r/r0rm,sJ%        rzhf cache prunezhf cache prune --dry-runc Cs8zt|}Wnty}z td|j|d}~wwi}t}|jD]}tdd|jD}|s3q$|||<|dd|Dq$t |dkrNt ddSt t||dd } |j t | j} t|} t d | jd | jd t||r|t d dS|stdst ddS| t d| jd| jddS)z)Remove detached revisions from the cache.rNcss"|] }t|jdkr|VqdS)rN)rJr9rqrhr/r/r0rrzprune..csrrNrZrr/r/r0rrrz2No unreferenced revisions found. Nothing to prune.r/rrz unreferenced revision(s) (z total).rzProceed?zPruning cancelled.rz! unreferenced revision(s); freed r)rrr rrrkr*r$updaterJrdr#rrbrLrDrrirrr) rrrrjrr%r$rKZdetachedrrrr/r/r0prunesH   r)zhf cache verify gpt2z)hf cache verify gpt2 --revision refs/pr/1z.hf cache verify my-dataset --repo-type datasetrQrPrhzXCache directory to use when verifying files from cache (defaults to Hugging Face cache). local_dirz?If set, verify files under this directory instead of the cache.fail_on_missing_filesz--fail-on-missing-filesz?Fail if some files exist on the remote but are missing locally.fail_on_extra_filesz--fail-on-extra-fileszLFail if some files exist locally but are not present on the remote revision.tokenc Cs|dur|durtdtjddt|d}|j|t|dr"|jnt|||||d} d} t| j } | rYtd | j D]} td | d d | d d| dd| dqsZ   ( ( $         A   2  ,6,  #,2 _  K B