o i/@sddlZddlZddlmZddlZddlmZddlmZddl m Z ddl m Z m Z ddlmZmZddlmZmZdd lmZeeZgd Zgd ZGd d d ZdS)N)Path)float_to_float16_max_diff) OnnxModel)optimize_model) T5DecoderT5DecoderHelper)T5EncoderDecoderInitT5EncoderDecoderInitHelper)MT5ForConditionalGenerationT5ForConditionalGeneration)InferenceSession)zt5-smallzt5-basezt5-largezt5-3bzt5-11b)zgoogle/mt5-smallzgoogle/mt5-basezgoogle/mt5-largez google/mt5-xlzgoogle/mt5-xxlc@s<eZdZe  d.dededededef dd Ze  d/ded ed ejd ededede ee e BffddZ e    d0de e Bd ejdededededefddZ e   d1dedeedBdedefd d!Ze    d2ded"ed#ed$ed%eded&ed'ed(efd)d*Zede e Bd+ed ejdefd,d-ZdS)3T5HelperF output_dirmodel_name_or_pathsuffix new_folderreturncCs^|}tj|rt|jd}n|dd||7}|r$tj||n|}tj||dS)aBuild onnx path Args: output_dir (str): output directory model_name_or_path (str): pretrained model name, or path to the model checkpoint suffix (str, optional): suffix like "_encoder" or "_decoder_fp16" will be appended to file name. Defaults to None. new_folder (bool, optional): create a new directory for the model. Defaults to False. Returns: str: path of onnx model /z.onnx)ospathisdirrpartssplitjoin)rrrrZ model_name directoryrf/home/kim/smarthome/.venv/lib/python3.10/site-packages/onnxruntime/transformers/models/t5/t5_helper.py get_onnx_path!s zT5Helper.get_onnx_patht5 cache_dirdevice model_typestate_dict_pathencoder_decoder_initc Cs|dkr tj||d}n|dkrtj||d}ntd|r&|t|t|j|j |j }| |t |j|j|j |j d| d}|rIdnd} | |d |iS) a{Load model given a pretrained name or path, then build models for ONNX conversion. Args: model_name_or_path (str): pretrained model name or path cache_dir (str): cache directory device (torch.device): device to run the model model_type (str, optional): model type "t5" or "mt5" state_dict_path(str, optional): state dictionary path encoder_decoder_init (bool, optional): combine encoder and decoder kv cache initialization into one model. Returns: Dict[str, torch.nn.Module]: mapping from name to modules for ONNX conversion. r )r!Zmt5z only support mode_type=t5 or mt5N)Zdecoder_start_token_idZoutput_cross_onlyr%encoderdecoder)r Zfrom_pretrainedr ValueErrorZload_state_dicttorchloadrr'Zlm_headconfigevaltorr&) rr!r"r#r$r%modelr'r&Z encoder_namerrr load_model>s& zT5Helper.load_modelTr.onnx_model_pathverboseuse_external_data_formatuse_decoder_input_idsuse_int32_inputsc Cs<t|trt|||||||dSt||||||dS)N) isinstancerr export_onnxr)r.r"r0r1r2r3r4rrrr6ls$  zT5Helper.export_onnxN onnx_model op_block_listforce_fp16_logitsuse_symbolic_shape_infercspdurgddd|D}t}||}td|d||jdj}d}|} || vs9J| |} d} | j d krw| } td | jd} | j D] } | | } | duranqTt | }t d | jd ||d k}n td| j d| jg}g|s| dur|s|g}| jgdvr|}d}d}|d}|rd}|D]=} | jvr|| | }|| |}fdd|D}fdd|D}t|t|dkr| j|d7}d}q|st||}td|d|td||d}td||r%|jdddi||Sddlm}||jfddi||S)aConvert model to mixed precision. It detects whether original model has fp16 precision weights, and set parameters for float16 conversion automatically. Args: onnx_model (OnnxModel): optimized ONNX model op_block_list (List[str], optional): operators need to run in fp32. force_fp16_logits (bool, optional): force logits and last MatMul node to be in float16. Defaults to False. use_symbolic_shape_infer (bool, optional): use symbolic shape inference to convert float to float16. Defaults to True. Returns: parameters(dict): a dictionary of parameters used in float16 conversion N)ZSimplifiedLayerNormalizationZ SkipSimplifiedLayerNormalizationZReluAddcSsh|]}|jqSrop_type).0noderrr sz0T5Helper.auto_mixed_precision..z fp32 op: z fp16 op: rFZMatMulz#Found last MatMul node for logits: z3max diff of converting weights in last MatMul node z: gư>z-Failed to find MatMul node for logits. Found z of node r;Tc"g|] }|jvs |vr|qSrr<)r>childnode_block_listr8rr z1T5Helper.auto_mixed_precision..crArr<)r>parentrCrrrErFz#node counter of Add operator: fp32=z fp16=znode_block_list: ) keep_io_typesr8rDZforce_fp16_initializersz!auto_mixed_precision parameters: r:)convert_float_to_float16Zdisable_shape_inferr)nodesset differenceloggerinfographoutputnameoutput_name_to_noder=inputZget_initializerrdebugwarninginput_name_to_nodesZget_nodes_by_op_typeZ get_parentsZ get_childrenlenappendrJfloat16r.)r7r8r9r:Z op_full_setZ fp32_op_setZ fp16_op_setZlogits_output_nameZis_weight_fp16_precisionrSr?Zlast_matmul_nodeZ initializerrTZmax_diffrIrWZfp32_addchangedZ add_nodesparentschildrenZblocked_childrenZblocked_parentsZfp16_add parametersrJrrCrauto_mixed_precisions               zT5Helper.auto_mixed_precisionoptimized_model_path is_float16num_attention_heads hidden_sizer_use_gpu force_fp16_ioc Csrddlm} d} |r| d} | | _t|d||d| |d} |r/|r)tj| |dn| j|d| j||dd dS) zHOptimize ONNX model with an option to convert it to use mixed precision.r) FusionOptionsNr )r#Z num_headsrc opt_leveloptimization_optionsrd)r9)Zcast_input_outputT)Zall_tensors_to_one_file)Zfusion_optionsrfZenable_skip_layer_normrr r_Z convert_model_float32_to_float16Zsave_model_to_file) r0r`rarbrcr2r_rdrerfrhmrrr optimize_onnxs&   zT5Helper.optimize_onnx ort_sessioncCs*t|tr t||||St||||S)zQCompare the result from PyTorch and OnnxRuntime to verify the ONNX model is good.)r5rr verify_onnxr)r.rkr"r4rrrrl#s zT5Helper.verify_onnx)rF)r rF)TFTF)NFT)FTFF)__name__ __module__ __qualname__ staticmethodstrboolrr)r"dictrrr/r6rlistr_intrjr rlrrrrr s  -   p 'r )loggingrpathlibrr)rZrr7rZ optimizerrZ t5_decoderrrZt5_encoder_decoder_initrr Z transformersr r Z onnxruntimer getLoggerrmrNZPRETRAINED_T5_MODELSZPRETRAINED_MT5_MODELSr rrrrs