L i'ddlZddlmZddlmZddlmZddlmZddl m Z m Z ddl m Z mZdd lmZmZd d lmZer ddlZddlmZd Zd ZdZdZdZ ddZGddZy)N)Image)create_causal_mask)_get_model_class) AutoConfig)MODEL_FOR_PRETRAINING_MAPPING MODEL_MAPPING)PROCESSOR_MAPPING_NAMES AutoProcessor)TOKENIZER_MAPPING_NAMES AutoTokenizer)is_torch_availablezzzu■u⬚c ~ jjdk(r dddddfjdk(r ddddddft}tdD}d}g} t D]C\ } | k(r |s }d f<|dkDs| k7s |dz k(s* |dz k(r dz d| | f<d}Ed|Et |D cgc].} t |D cgc]} d| | z cxkr|krnndndc} 0c} } dj fd t |D} ||dk(}tj|jd d z|zj|jd dtjd|dz|}tj| tttd t ttd}| j#d|zd|d zzdj%t| dzz}||dz }| j#|g}t D]\}||fdk(ra|j#t't)|j+tt)|D cgc]} t | tc} q|j#t't)|j+tt)|t't-t&t/|}|D]D}| j#|/|d zdzdj |zdzdj |zndFt D]\ t1j%|}vrt |tn|}dj fdt |D}d}|(dj fdt |D}| j#|dt) j+dd|d|dj | Scc} wcc} } wcc} w)z Generates an attention matrix from a given attention mask. Optionally applies a sliding window mask (e.g., for Gemma2/3) and marks regions where image tokens occur based on the specified `img_token`. rNc3DK|]}tt|ywN)lenrepr).0words m/mnt/ssd/data/python-lab/Trading/venv/lib/python3.12/site-packages/transformers/utils/attention_visualizer.py z6generate_attention_matrix_from_mask..8s.Ksj   1: (<.( 6w|nUG , 1:    sAA) boundariesz: i == j (diagonal) z: token_type_idszAttention MatrixzSliding Window Maskz | c3K|]V}|vr|frvrtttn+|k(rtttn|frtntXywrr)rr$i img_tokenr%rwordss rrz6generate_attention_matrix_from_mask..xs|   E!H$ad T8Ih|nUG ,Av7<.0AqDz    sAAc3K|]_}|vr'vr#dfd|fk(rtttn,|k(rtttn|rtntaywrr)rr$r+r,sliding_window_masktoken_type_bucketsrr-s rrz6generate_attention_matrix_from_mask..s * a(Y$->CUVWYZVZC[_qrsuvrv_wCw(<.06w|nUG4'q)!,"! " *sA%A(z:  )intndimrmax enumeraterangejointorchwherecumsumboolarange bucketizer"r r!rappendljustliststrrjustmapzipr)r-r%r,sliding_windowtoken_type_idsimage_seq_lengthnmax_word_length first_img_idxoutputkr+r$ row_dummy is_specialr(legendf_stringvertical_headeridxrow word_repr colored_word row_displaysliding_window_rowr/r0rs``` ` @@@r#generate_attention_matrix_from_maskrX(sP 88:D yyA~Aq!G} yyA~Aq!QJ E A-MDAJ 1 !y.AQJAEzQ56Dq-/1 2M!fklmfnoabSXYZS[\aa1q5&A>&A I\o q  I!#q( "[[  " "2 & *Z 7 = = ?AVAVWYAZ\] \\!%5%9;KL "__-?JWw|nUG+A&,X]W^^n oF MM#,o)*-?-E-Ec)nXYFY-ZZH!)) MM(Ou%F T S>Q   " "4CWZ[^_`[aWbHcCd#eavhqc%$9#e f  " "4Cs3q6{(C#D E F 3tS/%:;rEz Model name z- is not supported for attention visualizationceZdZdZy)7AttentionMaskVisualizer.__init__.._ModelWrapperctjj|tjdd|_||_y)Nr)nnModule__init__Linear dummy_moduleconfig)selfrfr[s rrcz@AttentionMaskVisualizer.__init__.._ModelWrapper.__init__s- ""4($&IIaO!$ N)__name__ __module__ __qualname__rcrhr _ModelWrapperr_s %rhrm)rfrom_pretrained image_tokenhasattrget_text_configgetattrrErr Exceptionr ValueError mapped_clsrarbmodeltodtyperepo_idrf)rgr[rfrurms rrcz AttentionMaskVisualizer.__init__s++J7" 6))+-= >")&*@*@*BDTVZ"[D  Q)&-@J  {:,6cde e$ %J  % #6:6  fll#!  ! Q)&2OPJ QsCC87C8input_sentencec*|j||y)N)suffix)visualize_attention_mask)rgrzr|s r__call__z AttentionMaskVisualizer.__call__s %%nV%Drhc z|j}i}d}|jjtvrd}t j t j|dj}d}tj|j|}t|dr |j}n)|jj|j gd}|r|j#d|}||||d } |jj|j gd|_ | d } d | vr| d |d <|jj| d d} n}|jjt$vr>t'j|j} | j)|} | |d d } n#t+d|jjdd|j_|j/| j0\} }t3j4| ||jj6f|jj8}t3j:|}t=|j|| |d}||j?} n2| jAdjAdjC| d||} dtEd|jjd|jFdzz}d}tId|tIdd|jjd|jd|jFjJjMtE|zdz|ztI|tO| | |jtQ|jdd|jd |}tI|tI|y) Nzchttps://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg?download=trueT)streamr')rGrorr]pt)imagestextr|return_tensorsattention_maskrF input_ids)rz Model type z) does not support attention visualizationeager)rx)rf input_embedsrcache_positionpast_key_valuesrz##zAttention visualization for z | rr1z" Attention visualization for :z z rE)r,rErFrG))rvrf model_typer ropenrequestsgetrawr rnryrpro tokenizerconvert_ids_to_tokensimage_token_idreplacer r tokenizert_attn_implementationtrainshaper8zeros hidden_sizerxr<rr; unsqueezeexpandrruprintricenterrXrr)rgrzr|rvkwargsrGimg processorroinputsrtokensr batch_size seq_lengthrr causal_masktop_bottom_border side_borderrPs rr}z0AttentionMaskVisualizer.visualize_attention_masks  ;; ! !%< <wC**X\\#d;??@C %55dllUefIy-0'33 '11GGIaIaHbcdef !/!7!7!Mcv^bcF(22HH)JbJbIcdefgD #$45N6)+12B+C'(((>>vk?RST?UVF [[ # #'> >%55dllCI''7F&~dKL\]N{5<<+B+B*CClmn n,3 ) !/!5!5 J{{J ELLrse .77TPR,    `diXg&g&rh