o iǍ @s dZddlZddlZddlZddlZddlZddlZddlmZddl m Z ddl Z ddl mZddlmZedZ d*d d Zd+ddZd,ddZd,ddZd+ddZddZid d d iddgdfddZddZd d!Z " # $ $ %d-d&d'Zid d d d"d#d"d$d$d%ddgf d(d)ZdS).zWeightOnly for onnxrt adaptor.N) numpy_helper)np_dtype_to_tensor_dtype) ONNXModel)simple_progress_barZneural_compressorc Cs.||d} tj|jd| fdd} |jdd|d|} |jd| g} g} i}d}|d kr[|d d d d d f|d d dd d fd >B}|d d d | f| d d d d f<n|dkrb|} n td |d t| d|| f} t|d|f}|jtjks|jtj ksJt j j |jddt |j|j|dd}| |j| ||d urB|dkr|d}n`|d krtj|jddd ddd}t|jd||d}|d d d }|dd d }||d d@||B||d <||d d@||d >B||d <ntd |d t||ddf}t j j |jddd |j|dd}| |j| ||d|d<|d|d<||d<||d<|dkr_||d<t j j | d | j| dd}| |t j j|f| |j|jr|jdt|ndt|dd|}|| fS)aBBuild MatMulNBits node. Args: node: original matmul node weight_shape: original weight shape num_bits (int): num_bits group_size (int): how many elements share one scale/zp k_blocks (int): block number q_weight (array): quantized weight scale (array): scale zero_point (array): zero point accuracy_level (int): accuracy level. Support 0 (unset), 1(fp32), 2(fp16), 3(bf16), or 4(int8). Returns: matmul_weight_only_node: MatMulNBits node new_inits: initializers of the new node ruint8dtyper_QGZ MatMulNBitsNz8MatMulNBits does not have kernel support for num_bits = .Z_scaleTnameZ data_typedimsvalsrawZ_zpKNbits block_sizeaccuracy_levelz com.microsoft)inputsZoutputsrdomain)npzerosshapeinputloggererrorreshaper float32Zfloat16onnxhelper make_tensorrtobytesappendrastypefullarangeZravel ValueErrorZ make_nodeoutputstr)node weight_shapenum_bits group_sizek_blocksq_weightscale zero_pointrZ blob_sizepackedZ q_weight_nameZ input_names new_initskwargsop_typeZq_weight_pairsZ scale_tensorZ packed_zpidxZeven_idxZodd_idxZ zp_tensorq_weight_tensorZmatmul_weight_only_noderAp/home/kim/smarthome/.venv/lib/python3.10/site-packages/onnxruntime/quantization/neural_compressor/weight_only.pymake_matmul_weight_only_node,s 0&      $*      $rCr asymint?c st|d|f}|dks|dkrd|ddn|dkr8|dkr)d|ddnd|dkr6d|d ndtj|ddd |}tj|ddd |}|dkrtt|t|}t|j} |dk} || d tj | | <|d krt | jn tj|jd d d|d>} nHt|j} t fdd||||k D| ||k<|d krt | j|| ntdtt | j|| d } tj|| jd } tj|| | dtj| | | dtj| | dtj| | d| | | fS)a Quantize tensor per group. Args: data : input weight num_bits (int, optional): num_bits. Defaults to 4. group_size (int, optional): how many elements share one scale/zp. Defaults to 4. scheme (str, optional): quantization scheme. Defaults to "asym". dtype (str, optional): data type. Defaults to "int". ratio (float, optional): percentile of clip. Defaults to 1.0. Returns: output: quantized weight scale: scale zero_point: zero point rrEuintrrrsymTaxisZkeepdimsg@rFrr csg|] }t|qSrA)float.0imaxqminqrArB z quant_tensor..out)r r&minmaxmaximumabsonesr"r-float64r!arrayflattentolistroundminimum empty_liker divideaddclip) datar5r6schemer ratiorminrmaxZ max_ranger9maskr:r8rArPrB quant_tensors<   . & , rlc&Cs@t|d|ftj}d|d}d}tj|dddd}t||}t|t|}tj|ddd}tj |ddd} tj|ddd} tj||ddd} tj | j |j d} || k} ||| | || | | <d| }t t| ||||}||||}tj||dddd}d}d }d}t|D]}tj | j |j d}t|||||g|j d}|| k} || | || || <t t|||||}||}tj|ddd}tj||ddd}tj||ddd}t| ||d}| || ||}|| |||}||||}tj||dddd} t| }!t|}"t|!|"kd}#||#d d f||#d d f<| |#||#<||#||#<||#||#<qt | |d|d }$|tj}tj||j d}%tj|||%d tj|%|$|%d tj|%|%d tj |%|||%d |%||$fS) aQuantize tensor per group based on k quant. Ref: https://github.com/ggml-org/llama.cpp/blob/64eda5deb9859e87a020e56bab5d2f9ca956f1de/ggml/src/ggml-quants.c Args: data : input weight num_bits (int, optional): num_bits. Defaults to 4. group_size (int, optional): how many elements share one scale/zp. Defaults to 32. Returns: output: quantized weight scale: scale zero_point: zero point rrrrTrJr 皙?NrrU)r r&r-r'sumsqrtrdrZrWrXr[r"r rer`ranger]subtractwherer\rbrc)&rfr5r6rQrRsum_x2av_xweightsrirjsum_wsum_xiscalerkr9 quant_datadiffbest_madnsteprdeltarrminis_ iscale_newfactorquant_data_newmul_weights_quant_data_newsum_lsum_l2sum_xlD this_scalethis_minmadmad_1 best_mad_1idx_to_replacer:r8rArArBquant_tensor_k_quant_cpusd  (      rc(Cszddl}ddl}|jr||}|d|f|j}d|d}d}|j|dddd}| ||}| || |} |j |ddd} |j |ddd} |j| ddd} |j| |ddd} |j| j|jd}| | k}||| || |||<d|}||||| ||}||| |}|j| |dddd}d }d }d}t|D]}|j| j|jd}||||||g|jd}| | k}|| || |||<||||| ||}| |}|j|ddd}|j||ddd}|j||ddd}|| ||d}| || ||} || |||}!| ||!|}|j| |dddd}"||"}#||}$||#|$kd}%||%ddf||%ddf<|"|%||%<| |%||%<|!|%| |%<q|| |d|d }&||j}|j||jd}'|j|||'d |j |'|&|'d |j|'|'d |j|'|||'d |'||&fWStd t|||WStytdt|||YSw)aQuantize tensor per group based on k quant. Ref: https://github.com/ggml-org/llama.cpp/blob/64eda5deb9859e87a020e56bab5d2f9ca956f1de/ggml/src/ggml-quants.c Args: data : input weight num_bits (int, optional): num_bits. Defaults to 4. group_size (int, optional): how many elements share one scale/zp. Defaults to 4. Returns: output: quantized weight scale: scale zero_point: zero point rNrrrTrJr rmrnrrUzqTry to use k-quant quantization on CUDA. However, CUDA is not available.Fall back to k-quant quantization on CPU.zNow we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.) ZcupytorchcudaZ is_availableZasarrayr&r-r'rorprdrZrWrXr[r"r rer`rqr]rrrsr\rbrcgetr$warningr ImportErrorinfo)(rfr5r6cprrQrRrtrurvrirjrwrxryrkr9rzr{r|r}r~rrrrrrrrrrrrrrrrr:r8rArArBquant_tensor_k_quant_cudas    (     rc Cs2|j}t||||||\}}} t||| |S)aQuant dequant tensor per group. Args: data : input weight num_bits (int, optional): num_bits. Defaults to 4. group_size (int, optional): how many elements share one scale/zp. Defaults to 4. scheme (str, optional): quantization scheme. Defaults to "asym". dtype (str, optional): data type. Defaults to "int". ratio (float, optional): percentile of clip. Defaults to 1.0. Returns: output: quant-dequant weight )r"rlr r&) rfr5r6rgr rh org_shapeweightr9zprArArB qdq_tensorysrcCsH|dkr|S|j}||}||d}|dkr"t|d|fdfd}|S)aPad tensor rowi so that it can be is divisible by group_size. Args: weight (array): weight group_size (int): how many elements share one scale/zp k_blocks (int): the number of block Returns: weight: paded weight rr)rrZconstant)r"r pad)rr6r7 org_w_shapeZ padded_rowsZpad_lenrArArB pad_tensors  rZCPUExecutionProviderk_quantc  Cst|}|jdurtj|jnd} g} g} tdd|D} d} |D]B}|jdvr8| d7} t| | |jdvrj| |j ddurj| |j idkrj| |j d}t j|| d }t|jd krlq'|j}|j |vr||j d }||j d }||j d }|j}|dkr|n|d}|dd|d}||j d}t|||}|dkp|dk}|r |dkrt|j||\}}}nt|j|||d| |j dd\}}}t||||||d|||dks|dkr|nd|d \}}||| || |nUt|j|||d| |j dd}t||ddf}t|}|d|dddf|}tj j!|j dd|d|t"||j|#dd}|$||j |j d<|dkrj|%|q'|&| |'| |(|S)aQuant the model with round to nearst method. Args: model (ModelProto or ONNXModel): onnx model weight_config (dict): quantization config For example, weight_config = { 'fc2': { 'bits': 4, 'group_size': 32, 'scheme': 'sym', 'algorithm': 'RTN' } } num_bits (int, optional): num_bits. Default is 4. group_size (int, optional): how many elements share one scale/zp. Default is 32. scheme (str, optional): sym or asym. Defaults to "asym". ratios (dict, optional): percentile of clip. Defaults to {}. accuracy_level (int): accuracy level. Support 0 (unset),1(fp32), 2(fp16), 3(bf16), or 4(int8). providers (list): providers to use Returns: model: fake quantized ONNXModel NcSsg|] }|jdvr|qS)ZMatMul)r>rMrArArBrSsz rtn_quantize..rrrfp32)base_dirrrr6rgrr rrrHrrE r3r4r5r6r7r8r9r:rrFr r Tr))r model_pathospathdirnamelennodesr>rget_initializerr#rrrto_arraycopyr"r get_initializer_share_numrrTrlrCr-add_initializersr,rr r&Z transposer(r)r*rr+add_initializerremove_initializerZ add_nodes remove_nodestopological_sort)model weight_configr5r6rgratiosr providers algorithmrZ new_nodesrZ total_numZcurr_idr3 weight_tensorrr rr7init_share_numsatisfy_MatMulNBits_conditionr8r9r q_matmul_noder<r@rArArB rtn_quantizes$        "       rc CsX|j}|dkrt|d|fn|}tjtt|tjt|ddd|dd}|S)zGet the scale of weight.rrTrJrrK)r"r r&meanrZrX)rr6rr9rArArBget_weight_scales2rc sddlm}ddlmt}tjdkr%|dr%ddlm }| ||j r6t j |j|jdd d d d |j sDtj|j||d n tj|jd||d }d d|D}~g} t|D]\} } |dkru| d|j|kru| |fSt|dkst| dtrt| dt|ksJdt|dt| dt| dtr| tfdd| dDq_t| dtjr| tddt|| dgd dDq_| tfddt|| dd dDq_| |fS)asPrepare inputs for weight only quantization. Args: model (ModelProto or ONNXModel): onnx model n_samples (int, optional): calibration sample number. -1 means all samples. dataloader (object): dataloader for calibration. providers (list): providers to use Returns: inputs: prepared inputs. so: session options r) find_specrto_numpy) onnxruntime_extensions)get_library_path _augment.onnxTFZsave_as_external_dataZall_tensors_to_one_fileZconvert_attributercSg|]}|jqSrArrMrArArBrSGz"prepare_inputs..rzInput number mismatch, require z but get cg|] \}}||fqSrArA)rNrZinp_datarrArBrSTrTcSsg|]\}}||fqSrArArNrinprArArBrSVsstrictcrrArArrrArBrSXrT)importlib.utilrutilrortZSessionOptionssys version_inforrZregister_custom_ops_libraryis_large_modelr( save_modelrrInferenceSessionSerializeToStringZ get_inputs enumerateZ batch_sizer isinstancedictr,itemsr Zndarrayzip) r n_samples dataloaderrrsorsessionZ inputs_namesrrOrfrArrBprepare_inputs#sF     &*,r{Gz?FTc # sZd|ddddfdd} |j} | |\} } t|dk}d|||f<d||d d f<|r[tt|d d d }||d d f}||d d fd d |f}t|}t|}|tt|}t| d}|||f|7<tjtj |j }|}t d| d|D]}t ||| d}||}t |||d d f}t|}t|}t|}|||||f}t |D]}||d d f}|||f}|d kr|||dkr| ||||||d d f\} } | tt|d d tjf| | d| } | ||d d f<|| d|d||d d f<|| |}!||d d d fttj||d |fdd tj|!dd 8<|!||d d f<q||||d d f<|d|||d d f<||d d d ft||d ||f|8<q|rt|}"||"d d f}t||j}~|S) aQuant the weight with GPTQ method. Args: W (array): weight. H (array): Hessian matrix. num_bits (int, optional): num_bits. Default is 4. group_size (int, optional): how many elements share one scale/zp. Default is 32. scheme (str, optional): sym or asym. Defaults to "asym". blocksize (int, optional): blocksize to quantize weight. percdamp (float, optional): percent of the average Hessian diagonal to use for dampening. actorder (bool, optional): whether rearrange Hessian matrix considering the diag's value. mse (bool, optional): whether get scale and zero point with mse error. perchannel (bool, optional): whether quantize weight per-channel. Returns: Q: fake quantized weight rrdg?g333333@cs>|j}stj|dd}t|jd}ttj|dd|}ttj|dd|}dkrItt ||}|dk}t |rI|| ||<|dk|dk@}d||<d||<||}dkrpt |jdd}nt | |}rt |jdgt d}ttD]`}d|} | |} | |} | | } dkrt | | n|} tt || | d}||8}tt |}t|d}||k}t |r||||<| |||<| |||<qs|d}t||}t||}dgdgt|d}t||}t||}||fS)NrrrrIrrinf)r"r expand_dimsr^r!rarWrYrXrZanyr[r`rLrqrFrepowerrorepeatrr&)rrtmpZxminZxmaxr9zerobestrOpZxmin1Zxmax1Zscale1Zzero1qerrr"gridrQZ maxshrinkmseZnorm perchannelrgrArB find_params~sX             zgptq..find_paramsrNrr)r"r diagZargsortZ zeros_likerr/ZlinalgZcholeskyinvrrqrWrdeepcopyrer`Znewaxisr^matmulrr&)#WHr5r6rg blocksizepercdampactorderrrrr"r9rZdeadpermZLossesQZdamprZHinvi1i2countZW1ZQ1ZErr1ZLosses1ZHinv1rOwdrZerr1ZinvpermrArrBgptq\sd 0         (6  D6 r c*st|}|jdurtj|jnd}t|||| \}}~t|jj j }| dd|Dg}| D]%}|j dvrY||jidkrY||jidddkrY||jd q4tt|}|||jrvtj|j|jd d d d d |jstj|j|| dn tj|jd || d}t|D]\}}tt||dg}g}|j|D]O}|j dvr||jidkr||jidddkr||jddurt !||"|jjd|}t|j#dkrq||||"|jqt|d krqdd|D}d |D]<}|$|g|d j#d t%&dj#dffdd|D}7t%'dfdd|D}q t(|||d dD]\}}}|j|vrr||jd}||jd}||jd}|dkry|n|j#d }|j)}t*|||||||| | | d }||jd}|+|jd} |dk}!|!r|j#}"|"d |d|}#t,|||#}t-|j.|||d\}}$}%t/||"|||#|0d|$0||dkr|%nd| d \}&}'|1|'|2||3|&n)tj4j5|jdd|d |t6||j#|0|7d d!}(|8|(|(j|jd<| dkr$|9|qRq| ||jj j :||;|jrOd d"l|jd |S)#aQuant the model with GPTQ method. Args: model (ModelProto or ONNXModel): onnx model dataloader (object): dataloader for calibration. weight_config (dict): quantization config For example, weight_config = { 'fc2': { 'bits': 4, 'group_size': 32, 'scheme': 'sym', 'algorithm': 'GPTQ' } } num_bits (int, optional): num_bits. Default is 4. group_size (int, optional): how many elements share one scale/zp. Default is 32. scheme (str, optional): sym or asym. Defaults to "asym". n_samples (int, optional): calibration sample number. percdamp (float, optional): percent of the average Hessian diagonal to use for dampening. blocksize (int, optional): blocksize to quantize weight. actorder (bool, optional): whether rearrange Hessian matrix considering the diag's value. mse (bool, optional): whether get scale and zero point with mse error. perchannel (bool, optional): whether quantize weight per-channel. accuracy_level (int): accuracy level. Support 0 (unset), 1(fp32), 2(fp16), 3(bf16), or 4(int8). providers (list): providers to use Returns: model: fake quantized ONNXModel NrcSrrArrMrArArBrSrz!gptq_quantize..rrrZGPTQrrTFrrrrcSs&g|]}t|jd|jdfqSr)r r!r"rMrArArBrSOs&rcsg|] }|qSrArArM)nsamplesrrArBrSUrTcsg|] }|tjqSrA)r rrrM)rrArBrSXsrrr6rg)r5r6rgrrrrrr rHrrErr r r)load_external_data_for_model)?rrrrrrrrrgraphr1Zremove_tensors_from_outputsrr>rrr,r#listsetZadd_tensors_to_outputsrr(rrrrrrrZinput_name_to_nodesrrrZget_noder"runr r&rprr r rrrlrrCr-r remove_nodeadd_noder)r*rr+rrZ MergeFromrZonnx.external_data_helperr split)*rrrr5r6rgrrrrrrrrrrrZ org_outputZ output_namesr3rr?Z input_nameZ node_listrvrZHsrfrr r8rrrrr7r9rrr<r@r rA)rr rrB gptq_quantizes/                   ? rr )r rDrErFrG)r rD)r rDrErrFFT)__doc__rloggingrrnumpyr r(rZ onnx.helperrZ onnxruntimerZ onnx_modelrrr getLoggerr$rCrlrrrrrrrr rrArArArBsl      s 6 I [ w<