L i^dZddlZddlZddlZddlZddlmZddlmZddl m Z m Z m Z ddl Z ddlZddlmZddlmZddlmZd d lmZd d lmZmZmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&d d l'm(Z(d d l)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1eeeeee e!e"e$e%eiee#e&edZ2e1jfe4Z5GddZ6dZ7dZ8idddddddddddddd d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/d0d1d2Z9gd3Z:d4Z;d5Zd8Z?d9Z@eGd:d;ZAd<ZBd=ZCd>ZDdEd?ZEd@ZFdAZGdBZHgdCZIdDZJy)Fz'Configuration base class and utilities.N) dataclass)Path)AnyOptionalUnion) model_info)OfflineModeIsEnabled)HFValidationError) __version__),MODEL_FOR_AUDIO_CLASSIFICATION_MAPPING_NAMES!MODEL_FOR_CAUSAL_LM_MAPPING_NAMESMODEL_FOR_CTC_MAPPING_NAMES,MODEL_FOR_IMAGE_CLASSIFICATION_MAPPING_NAMES*MODEL_FOR_IMAGE_SEGMENTATION_MAPPING_NAMES*MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES!MODEL_FOR_MASKED_LM_MAPPING_NAMES(MODEL_FOR_OBJECT_DETECTION_MAPPING_NAMES*MODEL_FOR_QUESTION_ANSWERING_MAPPING_NAMES,MODEL_FOR_SEQ_TO_SEQ_CAUSAL_LM_MAPPING_NAMES/MODEL_FOR_SEQUENCE_CLASSIFICATION_MAPPING_NAMES(MODEL_FOR_SPEECH_SEQ_2_SEQ_MAPPING_NAMES0MODEL_FOR_TABLE_QUESTION_ANSWERING_MAPPING_NAMES,MODEL_FOR_TOKEN_CLASSIFICATION_MAPPING_NAMES6MODEL_FOR_ZERO_SHOT_IMAGE_CLASSIFICATION_MAPPING_NAMES) ParallelMode)MODEL_CARD_NAME cached_fileis_datasets_availableis_offline_modeis_tf_availableis_tokenizers_availableis_torch_availablelogging)text-generationimage-classificationimage-segmentation fill-maskobject-detectionquestion-answeringtext2text-generationtext-classificationtable-question-answeringtoken-classificationaudio-classificationautomatic-speech-recognitionzzero-shot-image-classificationzimage-text-to-textcjeZdZdZdZdZedZedZedZ dZ dZ d Z d Z d Zy ) ModelCarda Structured Model Card class. Store model card as well as methods for loading/downloading/saving model cards. Please read the following paper for details and explanation on the sections: "Model Cards for Model Reporting" by Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji and Timnit Gebru for the proposal behind model cards. Link: https://huggingface.co/papers/1810.03993 Note: A model card can be loaded and saved to disk. c tjdt|jdi|_|jdi|_|jdi|_|jdi|_|jdi|_|jdi|_ |jdi|_ |jd i|_ |jd i|_ |jD]\}} t|||y#t$r%}t j#d |d |d ||d}~wwxYw)NzTThe class `ModelCard` is deprecated and will be removed in version 5 of Transformers model_details intended_usefactorsmetricsevaluation_data training_dataquantitative_analysesethical_considerationscaveats_and_recommendationsz Can't set z with value z for )warningswarn FutureWarningpopr4r5r6r7r8r9r:r;r<itemssetattrAttributeErrorloggererror)selfkwargskeyvalueerrs \/mnt/ssd/data/python-lab/Trading/venv/lib/python3.12/site-packages/transformers/modelcard.py__init__zModelCard.__init__\s& bdq $ZZ<"JJ~r:zz)R0 zz)R0 %zz*;R@#ZZ<%+ZZ0G%L"&,jj1I2&N#+1::6SUW+X(!,,. JC c5) "  z#l5'tfMN  s D D? D::D?ctjj|r%tjj|t}n|}|j |t jd|y)zKSave a model card object to the directory or file `save_directory_or_file`.zModel card saved in N)ospathisdirjoinr to_json_filerDinfo)rFsave_directory_or_fileoutput_model_card_files rKsave_pretrainedzModelCard.save_pretrainedssT 77==/ 0%'WW\\2H/%Z "%; " 01 *+A*BCDc 6|jdd}|jdd}|jdd}|jdd}ddi}|||d <tjj|}tjj |r|} d }n` t |t ||| } |rtjd | ntjd t d | |j| } g} |jD]0\} } t | st| | | | j!| 2| D]} |j| dtjd |r| |fS| S#ttjf$r |} YwxYw)a Instantiate a [`ModelCard`] from a pre-trained model model card. Parameters: pretrained_model_name_or_path: either: - a string, the *model id* of a pretrained model card hosted inside a model repo on huggingface.co. - a path to a *directory* containing a model card file saved using the [`~ModelCard.save_pretrained`] method, e.g.: `./my_model_directory/`. - a path or url to a saved model card JSON *file*, e.g.: `./my_model_directory/modelcard.json`. cache_dir: (*optional*) string: Path to a directory in which a downloaded pre-trained model card should be cached if the standard cache should not be used. kwargs: (*optional*) dict: key/value pairs with which to update the ModelCard object after loading. - The values in kwargs of any keys which are model card attributes will be used to override the loaded values. - Behavior concerning key/value pairs whose keys are *not* model card attributes is controlled by the *return_unused_kwargs* keyword parameter. proxies: (*optional*) dict, default None: A dictionary of proxy servers to use by protocol or endpoint, e.g.: {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request. return_unused_kwargs: (*optional*) bool: - If False, then this function returns just the final model card object. - If True, then this functions returns a tuple *(model card, unused_kwargs)* where *unused_kwargs* is a dictionary consisting of the key/value pairs whose keys are not model card attributes: ie the part of kwargs which has not been used to update *ModelCard* and is otherwise ignored. Examples: ```python # Download model card from huggingface.co and cache. modelcard = ModelCard.from_pretrained("google-bert/bert-base-uncased") # Model card was saved using *save_pretrained('./test/saved_model/')* modelcard = ModelCard.from_pretrained("./test/saved_model/") modelcard = ModelCard.from_pretrained("./test/saved_model/modelcard.json") modelcard = ModelCard.from_pretrained("google-bert/bert-base-uncased", output_attentions=True, foo=False) ``` cache_dirNproxiesreturn_unused_kwargsF_from_pipeline file_type model_cardusing_pipelineT)filenamerYrZ user_agentzloading model card file z from cache at z Model card: )r@rNrOrPisfilerrrDrSfrom_json_fileOSErrorjsonJSONDecodeErrorrAhasattrrBappend)clspretrained_model_name_or_pathrGrYrZr[ from_pipelinerais_localresolved_model_card_file modelcard to_removerHrIs rKfrom_pretrainedzModelCard.from_pretrained~sZJJ{D1 **Y-%zz*@%H #3T: !<0  $+8J' (77==!>? 77>>7 8'D $H "+61,'#) ,(KK":;S:T UVKK":?:K?[sZt uv../GH   ,,. &JCy#& 3.  % & "C JJsD ! "  l9+./ f$ $ #T112 "E  "sAE55 FFc|di|S)z@Constructs a `ModelCard` from a Python dictionary of parameters.rr)ri json_objects rK from_dictzModelCard.from_dicts![!!rWct|d5}|j}dddtj}|di|S#1swY&xYw)z8Constructs a `ModelCard` from a json file of parameters.utf-8encodingNrr)openreadreloads)ri json_filereadertextdict_objs rKrczModelCard.from_json_filesJ)g . !&;;=D !::d#X ! !s AA c4|j|jk(SN)__dict__)rFothers rK__eq__zModelCard.__eq__s}}..rWc4t|jSr)strto_json_stringrFs rK__repr__zModelCard.__repr__s4&&())rWcDtj|j}|S)z0Serializes this instance to a Python dictionary.)copydeepcopyr)rFoutputs rKto_dictzModelCard.to_dictst}}- rWcTtj|jdddzS)z*Serializes this instance to a JSON string.T)indent sort_keys )redumpsrrs rKrzModelCard.to_json_strings zz$,,.dCdJJrWct|dd5}|j|jdddy#1swYyxYw)z"Save this instance to a json file.wrvrwN)rywriter)rFjson_file_pathwriters rKrRzModelCard.to_json_files: .# 8 0F LL,,. / 0 0 0s 8AN)__name__ __module__ __qualname____doc__rLrV classmethodrprtrcrrrrrRrrrWrKr2r2Qsk. E[[z""/* K0rWr2z z r(zMasked Language Modelingr&zImage Classificationr'zImage Segmentationzmultiple-choicezMultiple Choicer)zObject Detectionr*zQuestion Answering summarization Summarizationr-zTable Question Answeringr,zText Classificationr%zCausal Language Modelingr+z&Sequence-to-sequence Language Modelingr.zToken Classification translation Translationzzero-shot-classificationzZero Shot Classificationr0zAutomatic Speech Recognitionr/zAudio Classification) accuracybleuf1matthews_correlationpearsonr precisionrecallrouge sacrebleu spearmanrwerc4|gSt|tr|gS|Sr) isinstancer)objs rK_listifyr,s" { C u  rWc||St|tr|g}|Dcgc]}|| }}t|dk(r|S|||<|Scc}w)Nr)rrlen)metadatanamevaluesvs rK_insert_values_as_listr5sX ~&# 1A1=a 1F 1 6{aHTN O 2s AAc|iSi}|D]e}|jjddtvr$|||jjdd<M|jdk(sa||d<g|S)N _rouge1r)lowerreplace METRIC_TAGS) eval_resultsresultrHs rK#infer_metric_tags_from_eval_resultsrAsw F" 99;  sC (K 747F399;&&sC0 1 YY[H $!F7O " MrWc||S|||<|Srrr)rrrIs rK _insert_valuerMs }HTN OrWcDtsyddlm}m}t |||fS)NFr)DatasetIterableDataset)rdatasetsrrr)datasetrrs rK is_hf_datasetrTs "1 g9 ::rWcg}|jD]8}t|ttfr|t|z }(|j |:|Sr)rrtuplelistrh)mappingrrs rK_get_mapping_valuesr]sL F ^^  a% ' d1g F MM!   MrWceZdZUeed<dZeeeeefed<dZ eeed<dZ eeeeefed<dZ eeed<dZ eeeeefed<dZ eeeeefed<dZeeeeefed <dZeeeeefed <dZeeeefed <dZeeeefed <dZeeeed <dZeeeefed<dZeeed<dZdZdZdZe ddZe ddZy)TrainingSummary model_nameNlanguagelicensetagsfinetuned_fromtasksr dataset_tags dataset_argsdataset_metadatar eval_lineshyperparameterstrainersourcec|jutsj|j]t|jdkDrD t |j}|j D]}|j ds|dd|_ yyyyy#tjjtjjttf$rYywxYw)Nrzlicense:) rr rrrr startswithrequests exceptions HTTPErrorConnectionErrorr r )rFrStags rK __post_init__zTrainingSummary.__post_init__xs LL #%##/D''(1, !$"5"5699/C~~j1'*12w / -0& !##--##33!$     s5B2 BACCcd|ji}t|j}t|j}t|j}t|j }t |t |kr|dgt |t |z zz}tt||}tt||}tt||} t|jD cic]} | tvs | t| } } g|d<t | dk(rt |dk(r|gSt | dk(rddi} t |dk(rddi}| D cgc] } |D]} | | f}} } |D]\} } i}| | | | d|d<| /| j| i}|| | d||d<||  || |dd<t |dkDrBg|d<|jD]*\}}|dj|||j|d ,d|vrd|vrd|vr|dj|tj!d ||gScc} wcc} } w) Nrresultsr)rtypetaskrargsr7)rrrIzLDropping the following result as it does not have all the necessary fields: )rrrrrrrdictziprTASK_TAG_TO_NAME_MAPPINGgetrArhrrDrS)rFmetric_mapping model_index dataset_namesrrrdataset_mappingdataset_arg_mappingdataset_metadata_mappingr task_mappingtask_tagds_tagall_possibilitiesrr metric_tag metric_names rKcreate_model_indexz"TrainingSummary.create_model_indexst/ !.  1 12  1 12 #D$9$9: | s<0 0'4&C 4ELHY4Y*ZZLs<?@"3|\#BC#'L:J(K#L >Fdjj=Q 59UY]uUuD*40 0  "$ I |  !c/&:a&?= |  ! $"Q&$&y!/=/C/C/E+J 9%,,$/$.%)%6%6{%CI$7I>J)(J N 9dllC    *z$:M:Ms/SX[\`\o\oXpstXt$X|T=P=PQH)(FDIIF)(J@Q@QR)(ItNDWDWDY?Z["&"9"9."IrWc d}tj|jd}t|dkDrd|d}|jdk(r |t z }n |t z }|d|jdz }|j|d z }n |d |jd |jd z }|j2t|jtrt|jdk(r|d z }nt|jtr|d|jdz }nt|jttfr/t|jdk(r|d|jddz }nF|dj|jddDcgc]}d| c}d|jddzz }|jS|dz }|dj|jj!Dcgc]\}}d|dt#|c}}z }|dz }|dz }|dz }|dz }|dz }|dz }|j$P|dz }|dj|j$j!Dcgc] \}}d|d|c}}z }|dz }n|dz }|j&"|d z }|t)|j&z }|dz }|d!z }|d"t*dz }|jdk(r"t-rddl}|d#|j*dz }n0|jd$k(r!t1rddl}|d%|j*dz }t5rddl}|d&|j*dz }t9rddl} |d'| j*dz }|Scc}wcc}}wcc}}w)(NF)rrz--- rz # z z'This model was trained from scratch on z'This model is a fine-tuned version of [z](https://huggingface.co/z) on zan unknown dataset.zthe z dataset.r z, z and the z datasets.z: It achieves the following results on the evaluation set: rz- z: z/ ## Model description More information needed z9 ## Intended uses & limitations More information needed z: ## Training and evaluation data More information needed z ## Training procedure z ### Training hyperparameters z: The following hyperparameters were used during training: z More information needed z ### Training results z ### Framework versions z- Transformers z - Pytorch kerasz - TensorFlow z - Datasets z - Tokenizers )yamldumprrrAUTOGENERATED_TRAINER_COMMENTAUTOGENERATED_KERAS_COMMENTrrrrrrrrQrrA _maybe_roundrrmake_markdown_tabler r#torchr! tensorflowrrr" tokenizers) rFr^rdsrrIrtfrrs rK to_model_cardzTrainingSummary.to_model_cards 99T113uE x=1   %0J ;;) # 7 7J 5 5JT__-T22    & C CJ (())B4CVCVBWW\^ J << Jt||T$Bs4< >J $ J44  }B77 ;;) #(:(<  Ju'8'8&9< 3J3JUXYdYjYjUk(l'm$'$/=L'$/$;$;#GGMM'--"6"6"D"DE$]]11??N =&}}66?? !-!3!3!5 ! g#':7'CC E !  gll556;;J z?a 'J <,-D c "t/G'G23D %T 1 KK0 1&7 8Q8Q&R#:|>wG!)%%-%!+  rWc | 3t| r(| | $| j} | dvr| | g} | | jg} | | | } |_t|jdrIt j j|jjs|jj}|B|jj} tjD]\}}| t|vs|}|dg}n/t|tr |dk7r|dg}nd|vr|j!d|t#|\}}}ng}i}t%|}|||||||| | | |||d S)Nrrgenerated_from_keras_callbackr) rrrrrrrrrrrrr)rrr rgrrNrOrPrr"rr#rArrrrhparse_keras_history"extract_hyperparameters_from_keras)rir!r keras_historyrrrrrrrrr*r+rrrrrrs rK from_keraszTrainingSummary.from_keraszs  W%<+?> 1HCQh;k#.>> !8{3// /#C I EM 3Z! [^ #'N62M +a. (!!n))+G L$/AKK.E;;vt,D ND1A 5tA'4udSF  %1 #01F,-WWS\F886!":$N4T__%6$NOD#$F4L  % LL )!, k Q C ({+c*:: q ({+c*:: Qw %c*002 6JC""7+C``"%((#))TW.+Y$DOO,=+Y"Z05 _-  6 %--%%%#%O,Zs <I$ 5I) cddlm}i}t|dr*|j|jj |d<nd|d<|j j j|d<|S)Nr )r optimizertraining_precision)modeling_tf_utilsrrgr^ get_configmixed_precision global_policyr)r!rrs rKr0r0&sd(Ouk"u'B','A'A'C $'+ $,1,A,A,O,O,Q,V,VO() rWct|trWtt|j ddkDr1tt|j dd|kDr|d|dSt|S)N.r f)rr4rrr)rdecimalss rKr r 3sb!UCFLL$5 6 :s3q6<$::__ 77$ % --`s$A cd|Dcgc] }dd|zzdz}}dj|dzScc}w)Nz|:-:rri)rQ)rjrrs rK_second_table_linerp>s:,6 7qdS1Wns" 7F 7 776?U ""8s-c |t|dk(ry|dDcic]}|tt|}}|D]L}|jD]7\}}||tt|ks!tt|||<9Nt t |dj t |j}|tt |jz }|D]M}|t |jDcgc] }t|c}t |jz }O|Scc}wcc}w)zC Create a nice Markdown table from the results in `lines`. rr) rrrAr rlrrrrp)rArHrjlinerItablers rKr r Cs+ }E a05a9#s3s8}$9J9;**, ;JC#\%%8!99"%l5&9": 3 ;; U1X]]_ 5tJE ) learning_ratetrain_batch_sizeeval_batch_sizeseedc dtDcic]}|t|j|}}|jjtj tj fvrL|jjtjk(rdn|jjj|d<|jjdkDr|jj|d<|jjdkDr|jj|d<|jj|jjz|jjz}||dk7r||d<|jj|jjz}||dk7r||d <|jjr|jj}|jjr|jjnd }d |jvrQd |d |jj d|jj"d|jj$d| |d<n d |d||d<|jj&j|d<|jj(dk7r|jj(|d<|jj*dk7r|jj*|d<|jj,dk7r|jj,|d<n|jj.|d<|jj0r.|j2rd|jj4|d<nd|d<|jj6dk7r|jj6|d<|Scc}w)Nz multi-GPUdistributed_typer num_devicesgradient_accumulation_stepsrutotal_train_batch_sizervtotal_eval_batch_sizez!No additional optimizer argumentsadamzUse z with betas=(,z) and epsilon=z and optimizer_args=r^z and the args are: lr_scheduler_typeglr_scheduler_warmup_ratiolr_scheduler_warmup_stepsrtraining_steps num_epochszApex, opt level mixed_precision_trainingz Native AMPlabel_smoothing_factor)_TRAINING_ARGS_KEYSgetattrr parallel_moder NOT_PARALLELNOT_DISTRIBUTED DISTRIBUTEDrI world_sizer{rurvoptim optim_argsr adam_beta1 adam_beta2 adam_epsilonr warmup_ratio warmup_steps max_stepsnum_train_epochsfp16use_apexfp16_opt_levelr)rrErr|r}optimizer_nameoptimizer_argss rKr(r(^s,JZ\ K ( .2.1AAUVdUe+fOK (+2<<+I+I+O+OO'(||  C'7>||7P7P34||  C'7>||7P7P34||#,3LL,B,B()(/ (E(E %||   rs. !'' &73"(   9HD2@DHJ PHH$o'B$oFn$o&\D "   H %j0j0Z!  +2.(  *  . _ :01D2= :#$B 2!(    ; [ [  [ | %F:&z  . # &4rW