o 0iG7@s|dZddlZddlZddlmZddlmZddlmZddl m Z m Z m Z m Z mZddlmZddlmZd d lmZmZmZmZmZmZd d lmZd d lmZmZe r\d d lm Z e!e"Z#e$dZ%dZ&dddZ'eGdddZ(e    dBde e(de e)de)de)de e)de e)de e*e)e)fde e+e)de,e+e*e+e*e e)ffddZ-Gdd d eZ.Gd!d"d"eZ/   dCd#d$d%e*de e)de e*e)e)fde e)ddf d&d'Z0d(e*fd)d*Z1d+e*fd,d-Z2d.e*fd/d0Z3d#d$d1e)ddfd2d3Z4d#d$d4e*d5e5d1e)ddf d6d7Z6d4e*d8e(d5e5de+e)fd9d:Z7d;e+e*dZ8d#d$d?e+e)d5e5de+e*fd@dAZ9dS)Dz.Git LFS related type definitions and utilitiesN) dataclass)ceil)getsize) TYPE_CHECKINGBinaryIOIterableOptional TypedDict)unquote) constants)build_hf_headersfix_hf_endpoint_in_urlhf_raise_for_status http_backoffloggingvalidate_hf_hub_args) SliceFileObj)sha256 sha_fileobj)CommitOperationAddz^[0-9a-f]{40}$zlfs-multipart-uploadzapplication/vnd.git-lfs+json)Acceptz Content-Typec@s`eZdZUdZeed<eed<eed<edefddZ edefd d Z ed e fd d Z dS) UploadInfoad Dataclass holding required information to determine whether a blob should be uploaded to the hub using the LFS protocol or the regular protocol Args: sha256 (`bytes`): SHA256 hash of the blob size (`int`): Size in bytes of the blob sample (`bytes`): First 512 bytes of the blob rsizesamplepathcCs\t|}t|d}|ddd}t|}Wdn1s"wY||||dS)Nrbrrr)rioopenpeekr)clsrrfilershar%M/home/kim/smarthome/.venv/lib/python3.10/site-packages/huggingface_hub/lfs.py from_pathIs  zUploadInfo.from_pathdatacCs&t|}|t||dd|dS)Nr)rrr)rdigestlen)r"r(r$r%r%r& from_bytesQs zUploadInfo.from_bytesfileobjcCsD|d}|dtjt|}|}|dtj||||dS)Nrrr)readseekrSEEK_SETrtell)r"r,rr$rr%r%r& from_fileobjVs zUploadInfo.from_fileobjN) __name__ __module__ __qualname____doc__bytes__annotations__int classmethodstrr'r+rr1r%r%r%r&r6s  r upload_infostoken repo_typerepo_idrevisionendpointheaders transfersreturncCs|dur|ntj}d}|tjvrtj|}|d||d} d|dur&|nddgdd |Dd d } |dur?d t|i| d <itt|d|pJi}td| || d} t| | } | dd} t | t skt d| d}t |t rw|nd}dd | Ddd | D|fS)a Requests the LFS batch endpoint to retrieve upload instructions Learn more: https://github.com/git-lfs/git-lfs/blob/main/docs/api/batch.md Args: upload_infos (`Iterable` of `UploadInfo`): `UploadInfo` for the files that are being uploaded, typically obtained from `CommitOperationAdd.upload_info` repo_type (`str`): Type of the repo to upload to: `"model"`, `"dataset"` or `"space"`. repo_id (`str`): A namespace (user or an organization) and a repo name separated by a `/`. revision (`str`, *optional*): The git revision to upload to. headers (`dict`, *optional*): Additional headers to include in the request transfers (`list`, *optional*): List of transfer methods to use. Defaults to ["basic", "multipart"]. Returns: `LfsBatchInfo`: 3-tuple: - First element is the list of upload instructions from the server - Second element is a list of errors, if any - Third element is the chosen transfer adapter if provided by the server (e.g. "basic", "multipart", "xet") Raises: [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If an argument is invalid or the server response is malformed. [`HfHubHTTPError`] If the server returned an error. N/z.git/info/lfs/objects/batchuploadbasic multipartcSsg|] }|j|jdqS)oidr)rhexr).0rFr%r%r& s z'post_lfs_batch_info..r) operationrBobjectsZ hash_algonameref)r<POSTrAjsonrOzMalformed response from serverZtransfercSsg|] }d|vrt|qSerror)_validate_batch_actionsrLobjr%r%r&rMcSsg|] }d|vrt|qSrU)_validate_batch_errorrXr%r%r&rMrZ)r ZENDPOINTZREPO_TYPES_URL_PREFIXESr LFS_HEADERSr rrrTget isinstancelist ValueErrorr:)r;r<r=r>r?r@rArBZ url_prefixZ batch_urlpayloadrespZ batch_inforOZchosen_transferr%r%r&post_lfs_batch_info`sB,        rcc@seZdZUeed<eed<dS) PayloadPartT partNumberetagN)r2r3r4r8r7r:r%r%r%r&rds  rdc@s&eZdZUdZeed<eeed<dS)CompletionPayloadTz?Payload that will be sent to the Hub when uploading multi-part.rJpartsN)r2r3r4r5r:r7r_rdr%r%r%r&rgs rgrNrlfs_batch_actionc CsFt||d}|durtd|jddS|dd}t||dd}|dur2t||di}|d} t|d |d } | durjzt| } Wntt fy`td | d wt ||| | d nt || d|durt|t|d |} t d| t ||d|jj|jjdd} t| t|jddS)a9 Handles uploading a given object to the Hub with the LFS protocol. Can be a No-op if the content of the file is already present on the hub large file storage. Args: operation (`CommitOperationAdd`): The add operation triggering this upload. lfs_batch_action (`dict`): Upload instructions from the LFS batch endpoint for this object. See [`~utils.lfs.post_lfs_batch_info`] for more details. headers (`dict`, *optional*): Headers to include in the request, including authentication and user agent headers. Raises: [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If `lfs_batch_action` is improperly formatted [`HfHubHTTPError`] If the upload resulted in an error actionsNzContent of file z. is already present upstream - skipping uploadrFverifyheader chunk_sizehref)r@zTMalformed response from LFS batch endpoint: `chunk_size` should be an integer. Got 'z'.)rNrlrm upload_url)rNrorR)r<rArIrSz: Upload successful)rWr]loggerdebugZ path_in_repo_validate_lfs_actionrr8r` TypeError_upload_multi_part_upload_single_partrr upload_inforrKrr) rNrir<rAr@rj upload_action verify_actionrlrmroZ verify_urlZ verify_respr%r%r& lfs_uploadsD        ry lfs_actioncCs:t|dtr|ddust|dtstd|S).validates response from the LFS batch endpointrnrlNz"lfs_action is improperly formatted)r^r]r:dictr`)rzr%r%r&rr s rrlfs_batch_actionscCspt|dtrt|dtstd|did}|did}|dur.t||dur6t||S)r{rJrz)lfs_batch_actions is improperly formattedrjrFrkN)r^r]r:r8r`rr)r}rwrxr%r%r&rWs rWlfs_batch_errorcCsht|dtrt|dtstd|d}t|tr.t|dtr.t|dts2td|S)r{rJrz'lfs_batch_error is improperly formattedrVmessagecode)r^r]r:r8r`r|)r~ error_infor%r%r&r["s  r[rocCsH|jdd}td||d}t|WddS1swYdS)aK Uploads `fileobj` as a single PUT HTTP request (basic LFS transfer protocol) Args: upload_url (`str`): The URL to PUT the file to. fileobj: The file-like object holding the data to upload. Raises: [`HfHubHTTPError`] If the upload resulted in an error. TZ with_tqdmPUTr(N)as_filerr)rNror,responser%r%r&ru0s "rurlrmcCsHt||j|d}t|||d}td|t||jjtd}t|dS)z@ Uploads file using HF multipart LFS transfer protocol. )rlrvrm)rNsorted_parts_urlsrmrR)rTrAN) _get_sorted_parts_urlsrv_upload_parts_iterativelyr_get_completion_payloadrrKr\r)rNrlrmrorresponse_headersZcompletion_resr%r%r&rtDs rtrvcCsNddtdd|DdddD}t|}|t|j|kr%td|S)NcSsg|]\}}|qSr%r%)rL_ror%r%r&rM\sz*_get_sorted_parts_urls..cSs2g|]\}}|rt|dkrt|d|fqS)r )isdigitr*r8)rLZpart_numror%r%r&rM_s  cSs|dS)Nrr%)tr%r%r&dsz(_get_sorted_parts_urls..)keyz0Invalid server response to upload large LFS file)sorteditemsr*rrr`)rlrvrmZsorted_part_upload_urlsZ num_partsr%r%r&r[s rrrJcCsfg}t|D]'\}}|d}|dus|dkr#td|d|d||d|dq||dS)NrfrDzInvalid etag (`z`) returned for part r )rerf)rJrh) enumerater]r`append)rrJrhZ part_numberrlrfr%r%r&rms  rrc Csg}|jdd<}t|D].\}}t||||d}td||d}t|||jWdn1s6wYq Wd|S1sGwY|S)NTr)Z seek_fromZ read_limitrr)rrrrrrrA) rNrrmrAr,Zpart_idxZpart_upload_urlZ fileobj_sliceZpart_upload_resr%r%r&r|s&   r)NNNN)NNN):r5rre dataclassesrmathrZos.pathrtypingrrrrr urllib.parser Zhuggingface_hubr utilsr rrrrrZ utils._lfsrZ utils.sharrZ _commit_apirZ get_loggerr2rpcompileZ OID_REGEXZLFS_MULTIPART_UPLOAD_COMMANDr\rr:r|r_tuplercrdrgryrrrWr[rur8rtrrrr%r%r%r&s          )  V  G