o i-`@sddlZddlZddlmZddlZddlZddlmZddlm Z m Z m Z ddl m Z ddlmZddlmZddlmZmZmZmZmZdd lmZmZmZd ejd <eeZd ej iZ!dCd dZ"ddZ#ddZ$ej%fddZ&ddZ'ddZ(ddZ)ddZ* dDddZ+de,d e,d!e-d"e.d#e.d$e d%e.d&e.fd'd(Z/d)e,d*e,d+e,fd,d-Z0d.d/Z1 dDd0d1Z2d2d3Z3dEd5d6Z4d7d8Z5d9d:Z6d;d<Z7d=d>Z8d?d@Z9dAdBZ:dS)FN)Path)AffinitySetting) OptimizerInfo Precisioncreate_onnxruntime_session) MODEL_CLASSES)QuantizeHelper)torch_onnx_export) AutoConfigAutoFeatureExtractor AutoTokenizer LxmertConfigTransfoXLConfig)PRETRAINED_GPT2_MODELSGPT2ModelNoPastStateTFGPT2ModelNoPastState2ZTF_CPP_MIN_LOG_LEVELtriucCs|dusJt|jdkr|d|dksJtd}|tjdtjd|}|d|dd|df}t||t |S)Nrr)rdtype) lenshapesize torch_functorchonesuint8whereboolZ zeros_like)xZdiagonaloutZ torch_triutemplatemaskr&`/home/kim/smarthome/.venv/lib/python3.10/site-packages/onnxruntime/transformers/onnx_exporter.py triu_onnx#s & r(cCs tt_dSN)r(rrr&r&r&r'replace_torch_functions-s r*cCstdt_dS)Nr)rrrr&r&r&r'restore_torch_functions1sr+c Cs|jdvrtj|d|j|jtj}d|i}|Stjjd|d||f|d}d|i}d|vr=tj||g|d }||d<d |vrNtj ||g|d } | |d <|j rU||d <t |t rxtj dd|jtj|d <tj dd|jtj|d <t |trtj |jgtjd |d<|S)NZvitswin pixel_valuesrrlowhighrr input_idsattention_maskrZtoken_type_idsdecoder_input_ids visual_feats visual_posz@tf_transfo_xl_model/transformer/pos_emb/einsum/Einsum/inputs_1:0) model_typenumpyrandomZrand image_sizeZastypeZfloat32randintrZzerosis_encoder_decoder isinstancer Zrandnvisual_feat_dimvisual_pos_dimr hidden_size) vocab_size batch_sizesequence_length input_namesconfigZ data_typer3inputsr4Z segment_idsr&r&r'create_onnxruntime_input5s,     rHcCs&i}|D] }||vr||||<q|Sr)r&)rGrEZremaining_model_inputsZ input_namer&r&r' filter_inputsSs  rIcCs$t|ttfrdd|DgS|gS)NcSsg|]}t|qSr&)flatten.0ir&r&r' \szflatten..)r>listtuple)rGr&r&r'rJ[s$rJcCs0|D]}t|ttfs||nt||q|Sr))r>rOrPappendupdate_flatten_list)rGZres_listrMr&r&r'rR_s rRc Cs|djd}dd|D}ddtt|D}t|D]%\}}ddi||<||j}t|D]\}} | |krA|||d iq0q||fS) Nr3cSsi|]}|dddqS)rCseq_len)rrr&rLkeyr&r&r' hsz&build_dynamic_axes..cSsg|] }dt|dqS)Zoutput_r)strrKr&r&r'rNjsz&build_dynamic_axes..rrCrT)rranger enumerateupdate) example_inputsZoutputs_flattenrD dynamic_axes output_namesrMZ output_namedimsjdimr&r&r'build_dynamic_axeses  rbc CsNt||dd}|durt|ddSt|ddd|D}|||}t|t|krEtdt|dt|dStt|D]Q} t t || ||  } | d krntd | d | |rrd nd } |rxd nd } tj || ||  | | dstd| d| d| dSqKtd|dS)NF)Zenable_all_optimizationz is an invalid ONNX modelz is a valid ONNX modelcSsi|] \}}||qSr&)r9)rLktr&r&r'rWsz'validate_onnx_model..z"Number of output tensors expected z, got g-C6?zMax absolute diff=z for output tensor g?g?)rtolatolzOutput tensor z is not close: rtol=z, atol=z0inference result of onnxruntime is validated on T) rloggererrorinfoitemsrunrrYr9ZamaxabscpuZallclose) onnx_model_pathr\example_outputs_flattenuse_gpuZfp16r^Z test_sessionZexample_ort_inputsZexample_ort_outputsrMZabs_diffrerfr&r&r'validate_onnx_modelts: $   rqonnx_dir model_name input_countoptimized_by_scriptrp precisionoptimized_by_onnxruntimeuse_external_datac Csddlm}|dd|} |s| d|} n|rdnd} | d|d|d| } |r/| d7} |} |rG|sGtj|| } tj| sGt| tj| | dS) Nr)subz [^a-zA-Z0-9_]_Zgpurm_ortz.onnx)reryospathjoinexistsmakedirs) rrrsrtrurprvrwrxryZnormalized_model_namefilenameZdevice directoryr&r&r'get_onnx_file_paths    r file_pathsuffixreturncCs&t|}t|j|j||jS)a Append a suffix at the filename (before the extension). Args: path: pathlib.Path The actual path object we would like to add a suffix suffix: The suffix to add Returns: path with suffix appended at the end of the filename and before extension )rrXparentjoinpathstem with_suffixr)rrr~r&r&r'add_filename_suffixsrcCsh|stj|s*t|jjdddddlm}m}||||dd}||||<dSt d|dS)NTparentsexist_okr)get_fusion_statisticsoptimize_by_onnxruntimec)rpoptimized_model_path opt_level'Skip optimization since model existed: ) r}r~rrrmkdir optimizerrrrgri)rnort_model_pathrp overwritemodel_fusion_statisticsrrrzr&r&r'optimize_onnx_model_by_ortsrc  Cs|stj|stt|jjdddddlm} ddlm } | dur&| |} | ||t j kr3d| _ |t jkr;d| _|dkrCd}d}| ||||d| |dd}|d ksW|d kr[||| |<|t j krl|jdd ||| dStd |dS) NTrr) FusionOptions)optimize_modelFr-)Z num_headsrAroptimization_optionsrpZonly_onnxruntimeZ bert_kerasZbert_tf)Z keep_io_typesr)r}r~rrrrfusion_optionsrrruse_raw_attention_maskrFLOAT16Zenable_gelu_approximationINT8Zenable_embed_layer_normZuse_dynamic_axesZget_fused_operator_statisticsZconvert_float_to_float16Zsave_model_to_filergri)rnrr8num_attention_headsrArprvrrruse_external_data_formatrrrZ opt_modelr&r&r'optimize_onnx_models>         rcCsz|dur|tvr |Stddt|tvrdSddl}|d|dur'dS|d|dur1dS|d |dur;d Sd S) NzValid model class:  rrz-squad$ZAutoModelForQuestionAnsweringz-mprc$Z"AutoModelForSequenceClassificationZgpt2ZAutoModelWithLMHeadZ AutoModel)r Exceptionrrr|search)rscustom_model_classr|r&r&r'modelclass_dispatchersrFcCszt||}|dkr|rtj|||dStj|||dS|r!d|}td|gd}td|t||}|j|||dS)Nr)rF cache_dirZTF transformers)fromlistzModel class name: )rrfrom_pretrainedr __import__rgrigetattr)rsrFrr is_tf_modelZmodel_class_nameZtransformers_module model_classr&r&r'load_pretrained_model/s  rcCs@tj||d}t|drd|_||t||||d}||fS)Nr return_dictF)rFrr)r rhasattrrmodifyr)rsrrconfig_modifierrFmodelr&r&r' load_pt_modelBs   rcCsHtj||d}||t}|t||||dd}|||fS)NrT)rFrrr)r rrrZ get_affinityrZ set_affinity)rsrrrrFZaffinity_settingrr&r&r' load_tf_modelNs rcCs ddlm}||\}}||fS)Nr)tf2pt_pipeline)Zconvert_tf_models_to_pytorchr)rsrrFrr&r&r'load_pt_model_from_tfcs  rc Cs2d}|r t| |||d|}|jtjjkr| || jfS|jtjjks+|tjks+|tjkrut ||t |d||d|}t | ||| j | j ||| | | || |} |rYt| ||||tjk|}|tjkrutd| t| | |td| |jtjjkr|rt| d}t| ||| | | ||dvr| jfS| jfS)NTFzQuantizing model: zFinished quantizing model: r{r,)rqnamerZNOOPTrBZBYSCRIPTrrrrrrrrArgrirZquantize_onnx_modelZBYORTrrZ num_labels)rsrr8rrrErprvZ optimize_info validate_onnxrrrFrrnr\ror^ris_valid_onnx_modelrrr&r&r'validate_and_optimize_onnxms        rc Cst||||\}}|d}d}|dvr;tj||d}tjjdd|j|jdtjd |j|jd}||dd}nt j||d}|j }|j d dd}t ||}|di|}t|ttfshJd t|t|}t|g}t||t|d | | d |}|stj|std |t|jjd d dd}d}|dvrdd|Ddg}}nt||\}}tt |t|!|t|"||d ||d t#ntd|t$|||||| | | | | ||||||d|\}}}||||fS)Nr,rrr.r0pt)return_tensorsThis is a sample inputz%type of output is not list or tuple: FExporting ONNX model to TrcSsi|]}|ddiqS)rr/r&rUr&r&r'rWsz-export_onnx_model_from_pt..Zlogits) rargsfrEr^r]Zdo_constant_folding opset_versionr!Skip export since model existed: r&)%rrmr rr9r:r<r;rZreshaper model_max_length encode_plusrIr>rOrPtyperJrRrrr}r~rrgrirrrrbr*r valueskeysr+r) rsrrr8rrrrrrErprvoptimizer_inforrrrrrFrr\max_input_sizeZimage_processordata tokenizerexample_outputsrornr]r^Zonnx_model_filerrBr&r&r'export_onnx_model_from_pts        rc(Csddl}|jgdtj||d}|jdur|ddi|j}t||||\}}| t ||j dd|dd d }t ||}|j rP|j dd|dd d j|d <|d krl|jd d |jg|d<|jd d |jg|d<z|jrsd|_Wn ty}Ynw||dd}d}|dks|dkrdg}|d}ddlm}||}t||t |d| | d|}|r|ddn|}|stj|sUtd||st|jj d d dddl!}ddl"}|j#$|j#j%g}|&D]\} }!dgt |!j'}"|(|j)t*|"|!j+| dq|j,j-|t*||||d\}#}#|rT|.|d}$|$/tj0|Wdn 1s2wYtj1tj0|d}tj|rNt2|t3||ntd||d}t4|||||| | | | | ||||||||\}%}&}'|%|&|'|fS)NrZGPUr pad_tokenz[PAD]rtf max_lengthT)rrpaddingZ truncationr5zunc-nlp/lxmert-base-uncasedrr6r7F)Ztrainingzxlnet-base-casedzxlnet-large-casedZlast_hidden_state)nestrr)r)Zinput_signatureZopsetZ large_modelZ output_pathrz__MODEL_PROTO.onnxrZ_tf)5Z tensorflowrFZset_visible_devicesr rrZadd_special_tokensrrZresize_token_embeddingsrrrIr=r3r:normalr?r@Z use_cacherZtensorflow.python.utilrrJrr}r~rrgrirrrzipfiletf2onnxloggingZ set_levelERRORrjrrQZ TensorSpecrPrconvertZ from_kerasZipFile extractalldirnamerremoverenamer)(rsrrr8rrrrrrErprvrrrrrrrrrrFrr\rr^rrornZtf_internal_model_pathrrspecsrvaluer_rzzZoptimized_onnx_pathrrBr&r&r'export_onnx_model_from_tf=s           r)rNr))F);rr}pathlibrr9rZaffinity_helperrZbenchmark_helperrrrZhuggingface_modelsrZquantize_helperrZtorch_onnx_export_helperr rr r r r rZ0onnxruntime.transformers.models.gpt2.gpt2_helperrrrenviron getLogger__name__rgrrr(r*r+Zint64rHrIrJrRrbrqrXintr!rrrrrrrrrrrrr&r&r&r'sl           , !  ;   a o