o i@#@s(ddlZddlZddlZddlZddlmZddlZddl m Z m Z m Z m Z ddlmZddlmZddlmZeeZGdddZd d Zed kreZejrXeejejZejZ ej!"e rue#d e d e$d e d e%eZ&ee&ej'ej(ej)dZ*e*+e*j&,e ddSdS)N) GraphProto ModelProto NodeProto TensorProto)quantize_matmul_bnb4) ONNXModel)attribute_to_kwargc@seZdZdZdZdZddededefdd Ze d e e d e e e ffd d Zdejd ejfddZdede e d efddZde e fddZddZdS)MatMulBnb4QuantizerzMPerform 4b quantization of constant MatMul weights using FP4 or NF4 data typerrNmodel quant_type block_sizecCs@|pg}|tjtjfvsJt||_||_||_t||_dSN) r FP4NF4rr r r setnodes_to_exclude)selfr r r rrh/home/kim/smarthome/.venv/lib/python3.10/site-packages/onnxruntime/quantization/matmul_bnb4_quantizer.py__init__%s  zMatMulBnb4Quantizer.__init__ graph_pathreturncCsLtt|dddD]}||}|jD]}|j|kr"||fSqq dS)Nr)NN)rangelen initializername)rrgidgraphZtensorrrrZ__get_initializer-s  z%MatMulBnb4Quantizer.__get_initializerfpweightc Cst|jdkr td|}|j\}}||}|j}||d|}|dd}tj|dd} tj||jd} t | || ||j ||| | fS)z4b quantize fp32/fp16 weightz9Current bnb4 block quantization only supports 2D tensors!rZuint8)dtype) rshape ValueErrorZ transposecopyr npZzerosr"rr ) rr Z fpweight_trowscolsZnumelr Z num_blocksZquantized_numelpackedabsmaxrrrbnb4_block_quant6s   z$MatMulBnb4Quantizer.bnb4_block_quantnode graph_stackcCs|jdkr|Std|jd|j|jvr#td|jd|S|jd}t||\}}|dur;td|Stj |}t |j d krOtd |S| |\}}tj |} |jd | _|jD]} | j|krt|j| nqetj |} |jd | _|j| | gi} |j \} }| | d <|| d<|j| d<|j| d<tjj d|jd| j| jg|jdg|jr|jd nddd| }td|jd|S)zdIf the node is MatMul with fp32 const weight, quantize the weight with int4, and return the new nodeZMatMulzstart to quantize z ...zexclude to quantize z$ as specified by nodes_to_exclude...rNz2MatMul doesn't have const weight. Skip to quantizer!z)MatMul weight is not 2D. Skip to quantizeZ_Bnb4Z_absmaxKNr r MatMulBnb4r com.microsoft)ZinputsZoutputsrdomainzcomplete quantization of )r0)op_typeloggerdebugrrinputr %_MatMulBnb4Quantizer__get_initializeronnxZ numpy_helperZto_arrayrr#r+Z from_arrayremoverextendr r helper make_nodeoutput)rr,r-ZinputBBZBs_graphZB_arrayr)r*ZB_quantr7Z absmax_tensorkwargsr'r(Zmatmul_bnb4_noderrr_bnb4_matmul_node_weightLsX                  z,MatMulBnb4Quantizer._bnb4_matmul_node_weightc Csg}|d}|jD]q}dd|jD}|rqi}|jD]D}|jtjjkr2||j|j| |i}n'|jtjj krUg} |j D]} || | | |gq>|j| i}nt |}||qtjj|j|j|jfd|ji|}||||q |d|j |||S)NrcSs,g|]}|jtjjks|jtjjkr|qSr)typer9AttributeProtoGRAPHGRAPHS).0attrrrr s z9MatMulBnb4Quantizer._process_subgraph..rr,)r, attributerBr9rCrDappendgr_process_subgraphrEZgraphsr;r updater<r=r4r7r>rAZ ClearFieldpop) rr-Z new_nodesrr,Z graph_attrsr@rGkvvalueZsubgraphrrrrLs@          z%MatMulBnb4Quantizer._process_subgraphcCsd|jg}|j}d}|D] }|jdkrd}q|s&|tjddg|||j dS)NFr2Tr) r r opset_importr3r;r9r<Z make_opsetidrLZclean_initializers)rr-rQZ has_ms_domainZopsetrrrprocesss    zMatMulBnb4Quantizer.processr)__name__ __module__ __qualname____doc__rrrintr staticmethodlistrtuplerr8nptZ ArrayLiker&Zndarrayr+rrArLrRrrrrr s 7 &r cCstjdd}|jdddd|jdddd|jd d d tjtjgd d |jdd ddd|jddd dd|jd d|jddtd gdd|S)NaBlockwise FP4/NF4 quantization for MatMul 2D weight matrices. A weight matrix is partitioned into blocks, where each block is a contiguous subset inside the flattened transposed weight matrix. Each block is quantized into a set of 4b integers with an absolute value scaling factor. ) descriptionz --input_modelTzPath to the input model file)requiredhelpz--output_modelzPath to the output model filez --quant_typeFrz&Quantization data type. 0: FP4, 1: NF4)r]defaultchoicesr^z --block_size@zVBlock size for blockwise quantization. Note: bnb.nn.Linear4bit only uses block_size=64)r]r_r^z-vz --verbose store_true)r]action)verbosez--nodes_to_exclude+zBSpecify the nodes to be excluded from quantization with node names)nargsrBr]r_r^) argparseArgumentParser add_argumentr rr set_defaultsstr parse_args)parserrrrrls:    rl__main__zfile z already exists)rT)-rgloggingosnumpyr&Z numpy.typingtypingr[r9Z onnx.onnx_pbrrrrZonnxruntime.capi._pybind_staterZ onnx_modelrZ quant_utilsr getLoggerrSr5r rlargsrdsetLevelDEBUGZ input_modelZinput_model_pathZ output_modelZoutput_model_pathpathexistserror Exceptionloadr r r rZquantrRZsave_model_to_filerrrrs8     "'