o il@s~ddlZddlZddlZddlZddlZddlZddlZddlmZddl Z ddl Z ddl Z ddl m Z ddlmZddlmZddlmZddlmZddlmZdd lmZdd lmZmZmZmZee Z!gd Z"e j#d e j$d e j%diZ&GdddeZ'GdddeZ(GdddeZ)GdddeZ*GdddeZ+e*ddfe+ddfe)ddfdZ,GdddZ-Gd d!d!Z.dS)"N)Path) Precision)float_to_float16_max_diff) FusionOptions)IOBindingHelper) OnnxModel)optimize_model)torch_onnx_export) GPT2ConfigGPT2LMHeadModel GPT2Model TFGPT2Model)Z distilgpt2gpt2z gpt2-mediumz gpt2-largezgpt2-xlMb@?g?g@c,eZdZdZfddZfddZZS)GPT2ModelNoPastState2Here we wrap a class to disable past state output.ct|dSNsuper__init__selfconfig __class__j/home/kim/smarthome/.venv/lib/python3.10/site-packages/onnxruntime/transformers/models/gpt2/gpt2_helper.pyr*zGPT2ModelNoPastState.__init__cstj|dddS)NF) use_cache return_dict)rforwardr input_idsrrrr"-szGPT2ModelNoPastState.forward__name__ __module__ __qualname____doc__rr" __classcell__rrrrr' rcr)TFGPT2ModelNoPastStatercsd|_t|dS)NF)r rrrrrrr4szTFGPT2ModelNoPastState.__init__cstj|ddS)NF)r )rcallr#rrrr"8rzTFGPT2ModelNoPastState.forwardr%rrrrr,1s r,cs8eZdZdZfddZeddZfddZZS) MyGPT2ModelzMHere we wrap a class for Onnx model conversion for GPT2Model with past state.crrrrrrrr?rzMyGPT2Model.__init__cCst|ddttfrNt|d|krt|dddksJg}t|D] }|tj|d|dd|d|ddfddq%|dt|fS|S)Nr)dim) isinstancetuplelistlenrangeappendtorchcatZ unsqueeze)result num_layerZpresentirrr post_processBs( *zMyGPT2Model.post_processc&tj||||dd}t||jjSNF) position_idsattention_maskpast_key_valuesr!rr"r.r=rn_layerrr$r@rApastr:rrrr"UszMyGPT2Model.forward) r&r'r(r)r staticmethodr=r"r*rrrrr.<s   r.cr)MyGPT2LMHeadModelzSHere we wrap a class for Onnx model conversion for GPT2LMHeadModel with past state.crrrrrrrrcrzMyGPT2LMHeadModel.__init__cr>r?rCrErrrr"fszMyGPT2LMHeadModel.forwardr%rrrrrH`r+rHcr)MyGPT2LMHeadModel_NoPaddingaHere we wrap a class for Onnx model conversion for GPT2LMHeadModel with past state and no padding. When you always use batch_size=1 in inference, there is no padding in inputs. In such case, position_ids and attention_mask need no be in inputs. crrrrrrrrxrz$MyGPT2LMHeadModel_NoPadding.__init__cs"tj||dd}t||jjS)NF)rBr!rC)rr$rFr:rrrr"{sz#MyGPT2LMHeadModel_NoPadding.forwardr%rrrrrIrs rIlogitsTF last_state)r ZGPT2LMHeadModel_NoPaddingr c@s8eZdZddZdefddZdefddZdd Zd S) Gpt2InputscCs||_||_||_||_dSr)r$r@rArF)rr$r@rArFrrrrs zGpt2Inputs.__init__returncCs0dd|j|j|jfD}|jr||j|S)NcSsg|]}|dur|qSrr.0vrrr z&Gpt2Inputs.to_list..)r$r@rArFextend)r input_listrrrto_lists zGpt2Inputs.to_listcCs"tdd|j|j|j|jfDS)Ncss|] }|dur|VqdSrrrNrrr sz&Gpt2Inputs.to_tuple..)r3r$r@rArF)rrrrto_tuples"zGpt2Inputs.to_tuplecCsTd}|jdur|jjtjkr|jjtjdn|j}dd|jD}t|j|j ||S)NdtypecSsg|] }|jtjdqS)rX)tor8float32rOprrrrQsz&Gpt2Inputs.to_fp32..) rArYr8float16rZr[rFrLr$r@)rrArFrrrto_fp32s zGpt2Inputs.to_fp32N) r&r'r(rr4rUr3rWr_rrrrrLs  rLc"@seZdZdZedddejejejdfdededededed ed ed ejd e d e de dej dej dej de de f ddZ e dXdededede dedeeeeff ddZeddZedYddZedYdd ZedZd"d#Zed[d%d&Zeddddejejejfd'ed(e d)e d e de dej dej dej fd*d+Ze   ,d\d-d.Zegd/fd0ed1eefd2d3Zed]d4e d5efd6d7Zed]d4e d5efd8d9Zed:d;Zed^deeejfd?eeeefd5ed@e dAe f dBdCZ edDdEZ!edFdGZ"eddHdHdIdJddddejejejd,ddfdKdLZ#eddMddddejejejdNdJdOf dPdQZ$ed`dRdSZ%edddgdTfdefdUdVZ&dWS)a Gpt2HelperzEA helper class for Gpt2 model conversion, inference and verification.FT batch_sizepast_sequence_lengthsequence_lengthnum_attention_heads hidden_sizer; vocab_sizedevicer^has_position_idshas_attention_maskinput_ids_dtypeposition_ids_dtypeattention_mask_dtypeleft_side_paddingrMcs"|rtjntjd|||t||gfddt|D}tjd|d||f| d}d}| rh||}tj||g| d}|dkrht|D]}td|d}|r]d||d|f<qHd||||df<qHd}| r| d d}| |dkd|dd|df | }t ||||S) zCreate random inputs for GPT2 model. Returns torch tensors of input_ids, position_ids, attention_mask and a list of past state tensors. r0cs$g|]}tjdddqS)rYrgg@?)r8Zrand)rO_rgZ float_typeZ past_shaperrrQs$z/Gpt2Helper.get_dummy_inputs..rr/)lowhighsizerYrgNrn) r8r^r[intr6randintZonesrandomlongZcumsumZ masked_fill_rZrL)rarbrcrdrer;rfrgr^rhrirjrkrlrmrFr$rAZtotal_sequence_lengthr<Zpadding_lengthr@rrqrget_dummy_inputssF  zGpt2Helper.get_dummy_inputsr r model_classcCs~|j}|j}|j}|j}t|d} ||| dkr|n|g} d||||t||g} | | i} t|D] } | | dt| <q2| S)zAReturns a dictionary with output name as key, and shape as value.r/rJr0present_)rdreZnum_hidden_layersrf MODEL_CLASSESrvr6str)rarbrcrr{rdrer;rfZ output_nameZlast_state_shapeZpresent_state_shape output_shapesr<rrrget_output_shapess&    zGpt2Helper.get_output_shapescCsZ|D](}||vs J||}t|||kr*tjt|||j|jd||<qdS)Nrn)numpyprodZnelementr8emptyrYrg)output_buffersrkeybufferrrrauto_increase_buffer_sizes   z$Gpt2Helper.auto_increase_buffer_sizecCsD|rtjntj}i}|D]\}}tjt|||d||<q|S)zpReturns a dictionary of output name as key, and 1D tensor as value. The tensor has enough space for given shape.rn)r8r^r[itemsrrr)rrg is_float16Z data_typernameshaperrrget_output_bufferss zGpt2Helper.get_output_bufferscCsH|d}t||d}|rt|t|dSt|S)zGReturns the maximum difference between PyTorch and OnnxRuntime outputs.rư>)cpurabsamax) torch_outputs ort_outputsrelativeZexpected_outputsdiffrrr diff_outputs%s  zGpt2Helper.diff_outputsMbP?c Kstj|d|d||d}td||}t|d}t|D])}tj|d||d|||d}td|d|d||oM|}q%|s`t||} t d| d |S) zReturns True if torch and ORT outputs are close for given thresholds, and False otherwise. Note: need kwargs since Gpt2BeamSearchHelper.compare_outputs has an extra parameter model_class r)rtolatolz9PyTorch and OnnxRuntime output 0 (last_state) are close: r/zPyTorch and OnnxRuntime layer z state (present_z ) are close:z@PyTorch and OnnxRuntime results are not all close: max_abs_diff=.5f) rallcloserloggerdebugr5r6r`rinfo) rrrrkwargsis_close is_all_closeZ num_layerslayer max_abs_diffrrrcompare_outputs/s""     zGpt2Helper.compare_outputsrc Csd}d}g}g}tt|D]}||}|dkr|dn|d|d} tj|| |dd} |tt| ||oA| }t|  rRt d|dt |  rbt d|dt| rrt d |dt | rt d |dt || } t| | j} |d | | d d | d || d dt| | d |dkrttj|dd|j} ttj| dd| j}t| |}q|t|}|t||||fS)aCompare outputs from PyTorch and OnnxRuntime Args: torch_outputs (Tuple[Torch.Tensor]): PyTorch model output ort_outputs (List[numpy.ndarray]): OnnxRuntime output atol (float, optional): Absolute tollerance. Defaults to 1e-06. Returns: is_all_close(bool): whether all elements are close. max_abs_diff(float): maximum absolute difference. messages(str): a list of debug message for each output TFrr/)rrzPyTorch output z has nanz has infz ORT output zdiff=z.9fz index=z ort=z torch=N)Zaxis)r6r5rrrr7rrisnananyrrisinffabsZ unravel_indexZargmaxrfloatZ array_equalindexmax)rrrris_top1_matchedZ max_diffsmessagesr<Z ort_outputZ torch_outputrridxZ ort_max_indexZtorch_max_indexmax_diff_output_indexrrrcompare_outputs_v2JsF(0 zGpt2Helper.compare_outputs_v2onnx_model_pathverboseuse_external_data_formatc Cs|j} | j} tjddd| j| j| | j|d||||| d} | } t || }Wdn1s3wYddt | D}ddt | D}|dj d | jks`|dj d | jks`J|dj d | jkrld nd g|}d d dd|dd ddi}|D] }d dd||<q|D] }d dd||<qd g}|rd dd|d<| d|rd dd|d<| d| |t|d krt|d| ksJtd| jj d| jdj d|dj d|ddj t|jjddd|rAt;}tj|d}t|jjdddt|t| |d|||ddd|d tj|dd}tj ||dddWddS1s:wYdSt|t| |d|||ddd|d dS) z1Export GPT-2 model with past state to ONNX model.r/F)rarbrcrdrer;rfrgr^rhrirjrkrlNcSg|]}d|qS)past_rrOr<rrrrQz*Gpt2Helper.export_onnx..cSr)r|rrrrrrQrrr0rJrKr$raZseq_len)rr/Z past_seq_len)r/Z total_seq_lenr@rAzShapes: input_ids=z past=z output=z present=T)parentsexist_okz gpt2.onnx ) argsfZ export_params input_names output_names dynamic_axesZ opset_versionZdo_constant_foldingrr)Zload_external_data)Zsave_as_external_dataZall_tensors_to_one_file)!rrDr`rzrdrerfrUr8no_gradr6rr7rSr5rrr$rFrparentmkdirtempfileTemporaryDirectoryospathjoinr r3onnxZ load_modelrsave)modelrgrrrrhrirjrkrlrr; dummy_inputsrToutputsZ past_namesZ present_namesrrrrZ tmp_dir_nameZtemp_onnx_model_pathrrr export_onnxs  ,"     6 $ zGpt2Helper.export_onnxrc Ksftd} t|d||d| dd} |r+|rt| nd|vr!d|d<| jd ddi|| ||| S) zHOptimize ONNX model with an option to convert it to use mixed precision.rrF) model_typeZ num_headsre opt_leveloptimization_optionsZuse_gpu keep_io_typesuse_symbolic_shape_inferTNr)rrr`auto_mixed_precisionconvert_float_to_float16Zsave_model_to_file) rZoptimized_model_pathrrdrerrstagerrmrrr optimize_onnxs$    zGpt2Helper.optimize_onnx)AddZLayerNormalizationZSkipLayerNormalizationZFastGeluZEmbedLayerNormalization onnx_model op_block_listcCsPdd|D}t|}||}td|d||jdj}d}|}||vs1J||}d} |j dkro|} td |jd} |j D] } | | } | durYnqLt | } t d |jd | | d k}n td |j d|jg} g}|s| dur|g} | jg}| |||d}td||jdddi||S)a?Convert GPT-2 model to mixed precision. It detects whether original model has fp16 weights, and set parameters for float16 conversion automatically. Args: onnx_model (OnnxModel): optimized ONNX model op_block_list (List[str], optional): operators to compute in fp32. Defaults to ["Add", "LayerNormalization", "SkipLayerNormalization", "FastGelu", "EmbedLayerNormalization"] Returns: parameters(dict): a dictionary of parameters used in float16 conversion cSsh|]}|jqSr)op_type)rOnoderrr 0sz2Gpt2Helper.auto_mixed_precision..z fp32 op: z fp16 op: rFNZMatMulz#Found last MatMul node for logits: z3max diff of converting weights in last MatMul node : rz-Failed to find MatMul node for logits. Found z of node )rrnode_block_listZforce_fp16_initializersz!auto_mixed_precision parameters: rTr)nodesset differencerrgraphoutputroutput_name_to_noderinputZget_initializerrrwarningr)rrZ op_full_setZ fp32_op_setZ fp16_op_setZlogits_output_nameZis_weight_fp16_precisionrrZlast_matmul_nodeZ initializerrZmax_diffrr parametersrrrrsH       zGpt2Helper.auto_mixed_precisioninputs total_runsc Cstd|}t ||}Wdn1swY|dkr)|Sg}t t|D]}t}||}|t|q4Wdn1sRwYt |dt |}td t |d||fS)zfRun inference of PyTorch model, and returns average latency in ms when total_runs > 0 besides outputs.zstart pytorch_inferenceNrzPyTorch inference time = {} ms.2f) rrr_rUr8rr6timer7sumr5format) rrrrTrlatencyrpstartaverage_latencyrrrpytorch_inferencebs$      zGpt2Helper.pytorch_inferencec Cs"tddt|ji}|jdur.t|jD]\}}t||d|<q|jdur?t|j|d<|j durPt|j |d<| d|}|dkr\|Sg}t |D]}t } | d|}| t | qbt|dt|} td t| d || fS) zcRun inference of ONNX model, and returns average latency in ms when total_runs > 0 besides outputs.zstart onnxruntime_inferencer$NrrAr@rrz"OnnxRuntime Inference time = {} msr)rrrZascontiguousarrayr$rrF enumeraterAr@runr6rr7rr5r) ort_sessionrrZ ort_inputsr<Zpast_irrrprrrrronnxruntime_inference|s(       z Gpt2Helper.onnxruntime_inferencec Cst|||||||S)z)Returnas IO binding object for a session.)rprepare_io_binding)rr$r@rArFrrrrrrs zGpt2Helper.prepare_io_bindingcCst||||S)z3Copy results to cpu. Returns a list of numpy array.)r"get_outputs_from_io_binding_buffer)rrr return_numpyrrrrsz-Gpt2Helper.get_outputs_from_io_binding_bufferrrrinclude_copy_output_latencyc Cstdt||j|j|j|j||}||t ||||}|dkr'|Sg} t |D]} t } |||rBt ||||} | t | q-t | dt| } td| || fS)zUInference with IO binding. Returns outputs, and optional latency when total_runs > 0.z*start onnxruntime_inference_with_binded_iorrz4OnnxRuntime with IO binding inference time = %.2f ms)rrr`rr$r@rArFZrun_with_iobindingrr6rr7rr5) rrrrrrrZ io_bindingrrrprrrrr$onnxruntime_inference_with_binded_ios8     z/Gpt2Helper.onnxruntime_inference_with_binded_iocCstd|dd}t||Wdn1swYtd|dtd|dd}t||Wdn1sBwYtd|ddS)NZ ort_outputs_.picklewbz$ORT output are saved to ort_outputs_Ztorch_outputs_z(Torch output are saved to torch_outputs_openpickledumprr)r<rrrrrr save_outputsszGpt2Helper.save_outputscCsTtd|dd}t||Wdn1swYtd|ddS)NZ dummy_inputs_rrz!inputs are saved to dummy_inputs_r)r<rrrrrrr save_inputsszGpt2Helper.save_inputsri'r/c,s(|j}td|dd|d|d| d|d d}d }d }d }|r5t||||| }t|||}d }d }gd g|}|}t|D]}t|}t d |}|d kr\d nt d |}t d |} t d| d|dtj | |||j |j |j|j||| | | | |dd}!t||!}"|rt||!}#nt| |||| }$t||!||$}#tj|"|#|d\}%}&}'}(})t|&sÈ|&|%r|d 7}|)r|d 7}||d 7<|r |%s td|d| d|d|d|& t|(D]\}}*td|d||jd|*q|r*t|&s|&d|kr*t||!t||#|"qHr8fdddD}+ndddD}+|d ||+d!<fd"d#|D|+d$<|d ||+d%<|td ||+d&<td'|d(|d)|td*||d+|krtd,t|d|d-d.|+S)/zKGenerate random inputs and compare the results of PyTorch and Onnx Runtime.zRunning parity test (atol=z , test_cases=z, runs=z, use_io_binding=z, model_class=z , is_float16=z) ...r0Nrr/z#Running parity test for batch_size=z past_sequence_length=z...T)rjrkrlrm)rz test_case=z batch_size=z sequence_length=z MaxDiff= z: Name=z, dcs$i|]}d|t|dqS)max_diff_percentile_r)rZ percentiler\)max_abs_diff_listrr osz*Gpt2Helper.test_parity..)2Z_ccSsi|]}d|dqS)r nanrr\rrrr srRroZtop1_match_ratecsg|]}|dqS)ror)rOx)test_cases_per_runrrrQvrRz*Gpt2Helper.test_parity..Ztop1_match_rate_per_runZdiff_pass_rateZnan_ratezParity Test Cases=z ; Passed=z; Nan=z; Top1_Matched=gffffff?zParity is good: passed rate=z.0f%)rrrr`rrr6rvrxrwrrzrdrerDrfrrrrrrr7rZ get_outputsrrrr5),rrrgrrrrruse_io_bindingr{rhrirjrkrlrrZenable_pickle_outputrZmax_batch_sizeZmax_past_seq_lenZ max_seq_lenrZmax_output_shapesZpassed_test_casesZtop1_matched_casesZtop1_matched_cases_per_runZtotal_test_casesr<Zrun_idrcrbrarrrrrrrrrmessager:r)r rr test_paritys(            (   " zGpt2Helper.test_parityrr cCs|j}d}|rt| || ||}t|||}tj| || |j|j|j|j||||| | | d}|r;t |||\}}|St |||||\}}|S)zCGenerate random inputs and measure average latency of Onnx Runtime.N)rjrkrl) rr`rrrzrdrerDrfrr)rrrgrrrr{rhrirjrkrlrarcrbrrrrrprrrrtest_performances<  zGpt2Helper.test_performancec Cs:tjddd|j|j|j|j|d||d }tj ||S)zJIT trace for TorchScript.r/F) rarbrcrdrer;rfrgr^rhri) r`rzrdrerDrfrUr8Zjittrace)rrrgrhrirTrrr torchscripts   zGpt2Helper.torchscriptrawZfp32Zfp16Zint8c Cs|}tj|rt|jd}n|dd|dkr!|d|7}|r'|d7}|rdddd d }d D]P}tj||||} tj| r||vrwzt | t d | Wq2t yv} zt d | d | j WYd} ~ q2d} ~ wwt d|d| q2tjtj|||dtjtj||d|dtjtj||d|dtjtj||d |dd Stj||dtj||dtj||dtj||dd S)z=Build a path name for given model based on given attributes.ru/r rpZ_pastZ_fp32Z_fp16Z_int8rzRemoved the existed directory: zFailed to remove the directory rNzDirectory for z existed: z.onnxz _fp32.onnxz _fp16.onnxz _int8.onnx)rrisdirrpartssplitrexistsshutilrmtreerrOSErrorstrerror) output_dirZmodel_name_or_pathr{Zhas_pastZ new_folderZremove_existingZ model_namesuffixrZnew_direrrrget_onnx_pathssT    $zGpt2Helper.get_onnx_pathsN)r )F)rr)r)FFr)r)T)rTF)TT)'r&r'r(r)rGr8Zint32rvrgboolrYrLrzr r~dictr4rrrrrrrrrrrrrrZTensorrrrrrrr*rrrrr`s`       @ "     5  w # E    2    6 r`)/loggingrrrxr#rrpathlibrrrr8Zbenchmark_helperrr^rZfusion_optionsrZio_binding_helperrrrZ optimizerrZtorch_onnx_export_helperr Z transformersr r r r getLoggerr&rZPRETRAINED_GPT2_MODELSZFLOAT32ZFLOAT16ZINT8ZDEFAULT_TOLERANCErr,r.rHrIr}rLr`rrrrsH           $