o i=b@slddlmZddlZddlmZddlmZddlm Z m Z m Z m Z ddl mZeeZGdddeZdS) ) getLoggerN)Fusion) FusionUtils) NodeProto TensorProtohelper numpy_helper) OnnxModelc s.eZdZdZdeffdd Zd.dedefdd Zd ed e defd d Z de de de fddZ dede dBfddZ dede dBfddZdede fddZdedee efde fddZdede dBfdd Zdede dBfd!d"Zd#e de dBfd$d%Zd#e d&e d'e d(e d)edef d*d+Zd,d-ZZS)/FusionMultiHeadAttentionMMDitzO Fuse MultiHeadAttention for Multimodal Diffusion Transformer (MMDiT). modelcstj|ddgdi|_dS)NMultiHeadAttentionSoftmax)Z fused_op_typeZsearch_op_types)super__init__unsqueeze_update_map)selfr  __class__c/home/kim/smarthome/.venv/lib/python3.10/site-packages/onnxruntime/transformers/fusion_mha_mmdit.pyrs z&FusionMultiHeadAttentionMMDit.__init__r start_nodereturncCs|jj|gd|ddg|d}|durdS|d}t|jdkr"dS|j|jd}|dur1dSt|jdkr:dSt|dS) a Detect num_heads from Reshape & Transpose of q/k/v for both Stable Diffusion 3.x and Flux 1.x: MatMul .. [-1] [24] .. | | | / / Add Concat(axis=0) | / Reshape | Transpose(perm=0,1,3,2) | (start_node) ) TransposeReshapeConcatroutput_name_to_nodeN)r match_parent_pathleninputZget_constant_valueshapeint)rrr input_indexnodesZ concat_shapevaluerrr get_num_headss z+FusionMultiHeadAttentionMMDit.get_num_heads transpose_kconcat_before_transposecCsh|r|jj|ddgddg|d}|r||d|SdS|jj|dgdg|d}|r2||d|SdS)a Detect num_heads from subgraph like the following (num_heads=24 in this example): MatMu .. [-1] [24] .. | | | / / Add Concat | / Reshape | Transpose(perm=0,2,1,3) | SimplifiedLayerNormalization | Transpose(perm=0,1,3,2) Another variant is to an extra Concat node to join two symmetrical subgraphs: | | MatMul MatMul .. [-1] [24] .. | | | | / / Add Concat Add Concat | / | / Reshape Reshape | | Transpose Transpose(perm=0,2,1,3) | | SimplifiedLayerNormalization SimplifiedLayerNormalization | / Concat | Transpose(perm=0,1,3,2) Both patterns are used in stable diffusion 3.5 model. rSimplifiedLayerNormalizationrrr)r r!r))rr*rr+r'rrrget_num_heads_from_k:s" z2FusionMultiHeadAttentionMMDit.get_num_heads_from_k input_name output_namecCsd}|j|}|dur"tjtjgddd|d}|j||jtj d||g|g|j dd}|j ||j|j |j<|jd S) a+Add a Reshape node to convert 4D BxSxNxH to 3D BxSxD. Args: input_name (str): input name for the 4D tensor of shape BxSxNxH. output_name (str): output name for the 3D tensor of shape BxSxD, where D = N * H. Returns: str: the output name Zbsnh_to_bsd_reshape_dimsN)rrrZint64)Zdtype)namerinputsoutputsr0r)r get_initializerrZ from_arraynparrayadd_initializerthis_graph_namer make_nodecreate_node_name nodes_to_addappendnode_name_to_graph_namer0output)rr.r/Z new_dims_nameZnew_dimsZ reshape_qrrr reshape_to_3dks    z+FusionMultiHeadAttentionMMDit.reshape_to_3dmul_qNcCs|j|ddgddg}|durdS|\}}t|dgds!dS|jd|jd<|jd}|d|jd<||jd|dS) a MultiHeadAttenion requires query in BSD format. This function adjusts query from BNSH to BSD format. Before: MatMul | Add Concat | / Reshape | Transpose(perm=0,2,1,3) | SimplifiedLayerNorm | Mul After: MatMul | Add Concat | / Reshape | SimplifiedLayerNorm | Reshape (shape=[0, 0, -1]) r,rrNpermrr r_BSNH_BSD)r r!rcheck_node_attributer#r>r?)rr@rpathsln_a transpose_aZ sln_outputrrr'adjust_query_from_bnsh_to_bsd_no_concats zEFusionMultiHeadAttentionMMDit.adjust_query_from_bnsh_to_bsd_no_concatc Cs8|j|gdgd}|durdS|\}}}t|jdkr dS|j|ddgddg}|dur2dS|\}}t|d gd sAdSt|d gd sLdSt|d dsUdS|jd|jd<|jd|jd<tjd |jd|jdg|jdd g|j d dd} |j | |j |j | j<|| jd|jddS)a MultiHeadAttenion requires query in BSD format. This function adjusts query from BNSH to BSD format. Before: MatMul MatMul | | Add Concat Add Concat | / | / Reshape Reshape | | Transpose(perm=0,2,1,3) Transpose(perm=0,2,1,3) | | SimplifiedLayerNorm SimplifiedLayerNorm | / Concat(axis=2) | Mul After: MatMul MatMul | | Add Concat Add Concat | / | / Reshape Reshape | | SimplifiedLayerNorm SimplifiedLayerNorm | / Concat(axis=1) | Reshape (shape=[0, 0, -1]) )rr,r)rrrNr r,rrrrArBaxisrrDr2r3r0rKrE)r r!r"r#rrFrr9r>r:r;r<r8r=r0r?) rr@rrGconcatrHrIsln_b transpose_bnew_concat_noderrradjust_query_from_bnsh_to_bsdsH!   z;FusionMultiHeadAttentionMMDit.adjust_query_from_bnsh_to_bsd unsqueezecCs|j|j}|durwt|jdkr)tjd|j|jddg|j ddgd}n6d}|j |durGtj |t j dgdgd}|j||jtjd|jd|g|jddg|j dd }|j||j|j|j<|jd}||j|j<|S) Nr UnsqueezerrDr )r2r3r0Zaxesunsqueeze_axes_2)r0Z data_typedimsvalsr1)rgetr0r"r#rr9r>r r:r4Z make_tensorrZINT64r7r8r;r<r=)rrRZupdated_unsqueeze_outputnew_nodeZinitializer_namerTrrrupdate_unsqueeze_axes_1_to_2s<      z:FusionMultiHeadAttentionMMDit.update_unsqueeze_axes_1_to_2addrcCs,t|jdkr dS|j|gdgd|}|durdSt|j}||d}|dus1|dgkr3dS||d}|dusC|dgkrEdS|j|gdgd|}|durXdS||d}|dush|dgkrjdS||d}|dusz|dgkr|dS||d|djd<||d|djd<d S) a Update axes of Unsqueeze from [1] to [2] in the following pattern: Unsqueeze Unsqueeze (axes=[0]) (axes=[0]) | | Unsqueeze Unsqueeze ... (axes=[1]) ... (axes=[1]) | / | / Mul Mul | / Add Args: add (NodeProto): the Add node output_name_to_node (Dict[str, NodeProto]): mapping from output name to node Returns: bool: True if the pattern is matched and updated successfully, False otherwise. r F)MulrSrS)rrrNrr)rrrT)r"r#r r!rZget_squeeze_or_unsqueeze_axesrY)rrZrZnodes_b fusion_utilsZaxes_1Zaxes_0Znodes_arrrupdate_unsqueeze_axes(s0 z3FusionMultiHeadAttentionMMDit.update_unsqueeze_axesc Csf|j|gdgd}|durdS|\}}}}}t|jdkr"dS|j|ddgddg}|dur4dS|\} } t|d gd sCdSt| d gd sNdSt|d dsWdS|||s_dS|jd|jd<| jd| jd<tjd |j d| j dg|j dd g|j d dd} |j | |j |j| j<|j|j d| j d||j d|j ddS)a3 Adjust graph to change query format from BNSH to BSD for Flux model. Note that the graph pattern is complex, and we only do a shallow match here. Before: | | Transpose(perm=0,2,1,3) Transpose(perm=0,2,1,3) | | SimplifiedLayerNorm SimplifiedLayerNorm | / Concat(axis=2) | Mul Mul | / Add | Mul After (Transpose nods are removed, and a Reshape is added): | | SimplifiedLayerNorm SimplifiedLayerNorm | / Concat(axis=1) | Mul Mul | / Add | Reshape (shape=[0, 0, -1]) )Addr[rr,r)rrrrrNr r,rrrrArBrKrrDrLrE)r r!r"r#rrFr]rr9r>r:r;r<r8r=r0Zreplace_input_of_all_nodesr?) rr@rrGrZ_mul_arMrHrIrNrOrPrrr"adjust_flux_query_from_bnsh_to_bsd]sN!   z@FusionMultiHeadAttentionMMDit.adjust_flux_query_from_bnsh_to_bsdcCs|j|gdgd}|durdS|\}}}}t|dgds#dS|||s+dS|jd|jd<|jdd|jd<||jd|jddS) a0 Adjust graph to change query format from BNSH to BSD for Flux model. Note that the graph pattern is complex, and we only do a shallow match here. Before: | Transpose(perm=0,2,1,3) | SimplifiedLayerNorm | Mul Mul | / Add | Mul After (Transpose is removed, and a Reshape is added): | SimplifiedLayerNorm | Mul Mul | / Add | Reshape (shape=[0, 0, -1]) )r^r[r,r)rrrrNrArBrrDrE)r r!rrFr]r#r>r?)rr@rrGrZr_rHrIrrr)adjust_flux_single_query_from_bnsh_to_bsds  zGFusionMultiHeadAttentionMMDit.adjust_flux_single_query_from_bnsh_to_bsdqcCsZtjd|g|dg|jjdddgdd}|j||j|j|j<| |d|dS)NrrDZTranspose_BNSH_to_BSNH)Z name_prefixrB)r0rArE) rr9r r:r;r<r8r=r0r?)rrbrZ transpose_qrrrtranspose_reshape_bnsh_to_bsds z;FusionMultiHeadAttentionMMDit.transpose_reshape_bnsh_to_bsdkvr> num_headsc CsV|dksJ|||g}|g}tjd|||jdd}d|_|jtd|g|S)a~ Create a MultiHeadAttention node. Args: q (str): name of q k (str): name of k v (str): name of v output (str): output name of MHA num_heads (int): number of attention heads. If a model is pruned, it is the number of heads after pruning. Returns: NodeProto: the node created. rr r1z com.microsoftrf)rr9r r:domain attributeextendZmake_attribute) rrbrdrer>rfZ mha_inputsZ mha_outputsZmha_noderrrcreate_multihead_attention_nodes   z=FusionMultiHeadAttentionMMDit.create_multihead_attention_nodecCs|jdksJ|}|j|jdrdS|j|gdgd|}|dur'dS|\}}}t|dgds7dS|j|gdgd} | durIdS| \} } } } }}}}| jd}||jdkradS|j| d d gd dg}|dursdS|\}}|jd}t|dgd sdS|j|d dgd dg}|durdS|djd| jdkrdS|jd }|jj |dd |d}|dur|jj |d d|d}|durdSt|dgdsdS|jj |d d |d}|durdSt|dgdsdSn|jj |d d |d}|durdSt|dgdsdS|r| ||n|j ||d d}|dkr4| |||du}|dkr4dS|dur@| | |}n| | |}|durg|| |}|durg|| |}|durg|||}|j||||jd|d}|j||j|j|j<|j|||gd|_dS)Nr r)MatMulrr)rrrlrlrArB)rkr[SqrtDivrmZCastSliceZShape)rrrrrrrrr[rr)rrrCr rmrnr)r&r)r&)rbrdrer>rfT)Zop_typer Zfind_graph_outputr>Zmatch_child_pathrrFr!r#Z match_parentr)r-rQrJr`rarcrjr;r<r8r=r0Znodes_to_removeriZ prune_graph)rnodeZinput_name_to_nodesrZsoftmaxr'Z matmul_s_vZ transpose_outZ reshape_outZq_nodesZ matmul_qkr@Zsqrt_q_2Zdiv_qZsqrt_q_Zshape_qZq_bnshZk_nodesZmul_kr*rdZ k_scale_nodesreZconcat_vZ transpose_1Z transpose_2rfqueryrXrrrfuses                 z"FusionMultiHeadAttentionMMDit.fuse)r)__name__ __module__ __qualname____doc__r rrr%r)boolr-strr?rJrQrYdictr]r`rarcrjrs __classcell__rrrrr s6!10O$5T3  +r )loggingrnumpyr5Z fusion_baserr\rZonnxrrrrZ onnx_modelr rtloggerr rrrrs