L i_ z d Z ddlZddlZddlZddlmZ ddlmZmZ ddlZ ddl
m Z ddl m
Z
mZmZmZ ddlmZ dd lmZ ej* e Ze G d
d Z G d d
Ze G d de Z G d d Z G d de Z G d de Z G d de Z G d de Z G d dee Z y) zJ
Callbacks to use with the Trainer class and customize the training loop.
N) dataclass)OptionalUnion)tqdm )HPSearchBackendIntervalStrategy SaveStrategy
has_length)TrainingArguments)loggingc e Zd ZU dZdZee ed< dZe ed< dZ
e ed< dZ e ed< dZ e ed < dZ
e ed
< dZee ed < dZe ed < dZe ed
< dZeed< dZeeeef ed< dZee ed< dZee ed< dZee ed< dZeed< dZeed< dZ eed< dZ ee ed< dZ eeee eee ef f ed< dZ eed ed< d Z!d efd Z"e#d efd Z$d Z%d! Z&y)" TrainerStatea
A class containing the [`Trainer`] inner state that will be saved along the model and optimizer when checkpointing
and passed to the [`TrainerCallback`].
In all this class, one step is to be understood as one update step. When using gradient accumulation, one update
step may require several forward and backward passes: if you use `gradient_accumulation_steps=n`, then one update
step requires going through *n* batches.
Args:
epoch (`float`, *optional*):
Only set during training, will represent the epoch the training is at (the decimal part being the
percentage of the current epoch completed).
global_step (`int`, *optional*, defaults to 0):
During training, represents the number of update steps completed.
max_steps (`int`, *optional*, defaults to 0):
The number of update steps to do during the current training.
logging_steps (`int`, *optional*, defaults to 500):
Log every X updates steps
eval_steps (`int`, *optional*):
Run an evaluation every X steps.
save_steps (`int`, *optional*, defaults to 500):
Save checkpoint every X updates steps.
train_batch_size (`int`, *optional*):
The batch size for the training dataloader. Only needed when
`auto_find_batch_size` has been used.
num_input_tokens_seen (`int`, *optional*, defaults to 0):
When tracking the inputs tokens, the number of tokens seen during training (number of input tokens, not the
number of prediction tokens).
total_flos (`float`, *optional*, defaults to 0):
The total number of floating operations done by the model since the beginning of training (stored as floats
to avoid overflow).
log_history (`list[dict[str, float]]`, *optional*):
The list of logs done since the beginning of training.
best_metric (`float`, *optional*):
When tracking the best model, the value of the best metric encountered so far.
best_global_step (`int`, *optional*):
When tracking the best model, the step at which the best metric was encountered.
Used for setting `best_model_checkpoint`.
best_model_checkpoint (`str`, *optional*):
When tracking the best model, the value of the name of the checkpoint for the best model encountered so
far.
is_local_process_zero (`bool`, *optional*, defaults to `True`):
Whether or not this process is the local (e.g., on one machine if training in a distributed fashion on
several machines) main process.
is_world_process_zero (`bool`, *optional*, defaults to `True`):
Whether or not this process is the global main process (when training in a distributed fashion on several
machines, this is only going to be `True` for one process).
is_hyper_param_search (`bool`, *optional*, defaults to `False`):
Whether we are in the process of a hyper parameter search using Trainer.hyperparameter_search. This will
impact the way data will be logged in TensorBoard.
stateful_callbacks (`list[StatefulTrainerCallback]`, *optional*):
Callbacks attached to the `Trainer` that should have their states be saved or restored.
Relevant callbacks should implement a `state` and `from_state` function.
Nepochr global_step max_stepsi
logging_steps
eval_steps
save_stepstrain_batch_sizenum_train_epochsnum_input_tokens_seen
total_flos log_history best_metricbest_global_stepbest_model_checkpointTis_local_process_zerois_world_process_zeroFis_hyper_param_search
trial_name trial_paramsTrainerCallbackstateful_callbacksc | j g | _ | j i | _ y t | j t ry i }| j D ] }t |t st dt
| |j j }||v r?t || t s || g||< || j |j |j ||< || _ y )NzNAll callbacks passed to be saved must inherit `ExportableState`, but received ) r r$
isinstancedictExportableState TypeErrortype __class____name__listappendstate)selfr$ callbacknames c/mnt/ssd/data/python-lab/Trading/venv/lib/python3.12/site-packages/transformers/trainer_callback.py
__post_init__zTrainerState.__post_init__u s # !D " " *&(D #
//
6 "$ 3 3
@!(_>#himnviwhxy ))22-- &&8&>E4Ft4L3M*40&t,33HNN4DE/7~~/?&t,
@ '9D # json_pathc t j t j | dd dz }t |dd 5 }|j | ddd y# 1 sw Y yxY w) zDSave the content of this instance in JSON format inside `json_path`. T)indent sort_keys
wutf-8encodingN)jsondumps dataclassesasdictopenwrite)r0 r6 json_stringfs r3 save_to_jsonzTrainerState.save_to_json sT jj!3!3D!9!tTW[[
)S7
3 !q
GGK ! ! !s AA!c t |d 5 }|j }ddd | di t j S # 1 sw Y $xY w)z3Create an instance from the content of `json_path`.r= r> N )rD readr@ loads)clsr6 rG texts r3 load_from_jsonzTrainerState.load_from_json sG )g
. !668D &TZZ%&& s AA c dD ]A }t || d }||dk rt j ||z }t | | d| C y)z
Calculates and stores the absolute value for logging,
eval, and save steps based on if it was a proportion
or not.
)r
evalsave_stepsNr )getattrmathceilsetattr)r0 argsr step_kind num_stepss r3
compute_stepszTrainerState.compute_steps sY 5 ?I 6&:;I$ q= $ )i*? @I 6 2I> ?r5 c r |j ,|j |j |j | _ d| _ |>ddlm} |j t j k( r |j n|} || | _ || _
|| _ |j | _ |j | _
y)zI
Stores the initial training references needed in `self`
Nr ) hp_params)hp_name_trialr! r" transformers.integrationsr] hp_search_backendr SIGOPT assignmentsr r r r )r0 trainerr r trialr] rc s r3 init_training_referencesz%TrainerState.init_training_references s ?? &7>>+E &oognn =DO ;/6/H/HOLbLb/b%++hmK )+ 6D " 0 %,%B%B%D"%,%B%B%D"r5 )'r,
__module__ __qualname____doc__r r float__annotations__r intr r r r r r r r r r- r' strr r r r boolr r r! r" r r$ r4 rH classmethodrO r[ rf rJ r5 r3 r r # s` 9v "E8E?!KIs M3 JJ&*hsm* c !"3"J*.Kd3:&'.#'K% '&*hsm*+/8C=/"&4&"&4&"'4' $J
$FJL(4U3sD+@%A ABCJ<@&7!89@96!c ! 's ' ' ?Er5 r c , e Zd ZdZdefdZed Zy)r( aj
A class for objects that include the ability to have its state
be saved during `Trainer._save_checkpoint` and loaded back in during
`Trainer._load_from_checkpoint`.
These must implement a `state` function that gets called during the respective
Trainer function call. It should only include parameters and attributes needed to
recreate the state at a particular time, to avoid utilizing pickle/maintain standard
file IO writing.
Example:
```python
class EarlyStoppingCallback(TrainerCallback, ExportableState):
def __init__(self, early_stopping_patience: int = 1, early_stopping_threshold: Optional[float] = 0.0):
self.early_stopping_patience = early_stopping_patience
self.early_stopping_threshold = early_stopping_threshold
# early_stopping_patience_counter denotes the number of times validation metrics failed to improve.
self.early_stopping_patience_counter = 0
def state(self) -> dict:
return {
"args": {
"early_stopping_patience": self.early_stopping_patience,
"early_stopping_threshold": self.early_stopping_threshold,
},
"attributes": {
"early_stopping_patience_counter": self.early_stopping_patience_counter,
}
}
```returnc t d )NzInternal class that just calls the list of callbacks in order.c g | _ |D ] }| j | || _ || _ || _ || _ d | _ d | _ t d | j D s#t j d| j z y y )Nc 3 < K | ] }t |t y wN)r& DefaultFlowCallback.0cbs r3 z+CallbackHandler.__init__.. s P2:b"56Ps zThe Trainer will not work properly if you don't have a `DefaultFlowCallback` in its callbacks. You
should add one before training with `trainer.add_callback(DefaultFlowCallback). The current list ofcallbacks is
:) callbacks add_callbackmodelprocessing_class optimizer lr_schedulertrain_dataloadereval_dataloaderanyloggerwarning
callback_list)r0 r r r r r r s r3 __init__zCallbackHandler.__init__ s "B b ! "
0 " ( $ # P PP NN$ $$%
Qr5 c P t |t r | n|}t |t r|n |j }|| j D cg c] }|j c}v r)t j d| ddz | j z | j j | y c c}w )NzYou are adding a zH to the callbacks of this Trainer, but there is already one. The currentzlist of callbacks is
:)r& r* r+ r r r r r. )r0 r1 r cb_classcs r3 r z CallbackHandler.add_callback s %h5XZ8 )(D 98x?Q?Q T^^< < < NN#H:-uv+,$$%
b!
=s B#c t |t r=| j D ]- }t || s| j j | |c S y | j D ]& }||k( s | j j | |c S y r r& r* r remover0 r1 r s r3 pop_callbackz CallbackHandler.pop_callback ss h %nn
b(+NN))"- I
nn
>NN))"- I
r5 c t |t r;| j D ]+ }t || s| j j | y y | j j | y r r r s r3 remove_callbackz CallbackHandler.remove_callback sR h %nn
b(+NN))"-
NN ! !( +r5 c F dj d | j D S )Nr; c 3 H K | ] }|j j y wr )r+ r, r s r3 r z0CallbackHandler.callback_list.. s H2 ..Hs ")joinr rt s r3 r z CallbackHandler.callback_list s yyH HHHr5 rX r/ r c * | j d||| S )Nr
call_eventr0 rX r/ r s r3 r zCallbackHandler.on_init_end }dE7CCr5 c 8 d|_ | j d||| S )NFr )r~ r r s r3 r z CallbackHandler.on_train_begin s ',$ /ugFFr5 c * | j d||| S )Nr r r s r3 r z CallbackHandler.on_train_end ~tUGDDr5 c 8 d|_ | j d||| S )NFr )r r r s r3 r z CallbackHandler.on_epoch_begin s $)! /ugFFr5 c * | j d||| S )Nr r r s r3 r z CallbackHandler.on_epoch_end r r5 c T d|_ d|_ d|_ | j d||| S )NFr )r r r r r s r3 r z CallbackHandler.on_step_begin s/ ""' #eWEEr5 c * | j d||| S )Nr r r s r3 r z%CallbackHandler.on_pre_optimizer_step s 6eWMMr5 c * | j d||| S )Nr r r s r3 r z!CallbackHandler.on_optimizer_step s 2D%IIr5 c * | j d||| S )Nr r r s r3 r z CallbackHandler.on_substep_end s /ugFFr5 c * | j d||| S )Nr r r s r3 r zCallbackHandler.on_step_end r r5 c < d|_ | j d|||| S )NFr r )r r r0 rX r/ r r s r3 r zCallbackHandler.on_evaluate s# "' }dE7GTTr5 c . | j d|||| S )Nr r r r s r3 r zCallbackHandler.on_predict s |T5'7SSr5 c 8 d|_ | j d||| S )NFr )r r r s r3 r zCallbackHandler.on_save s #y$w??r5 c < d|_ | j d|||| S )NFr )logs)r r )r0 rX r/ r r s r3 r zCallbackHandler.on_log# s# "xugDIIr5 c * | j d||| S )Nr r r s r3 r z"CallbackHandler.on_prediction_step' s 3T5'JJr5 c | j D ]_ } t || |||f| j | j | j | j
| j | j d|}|^|}a |S )N)r r r r r r )r rT r r r r r r )r0 eventrX r/ r r r1 results r3 r zCallbackHandler.call_event* s !H-WXu- jj!%!6!6 .. ! . .!%!6!6 $ 4 4 F ! ! r5 N) r, rg rh ri r r r r propertyr r r r} r r r r r r r r r r r r r r r r rJ r5 r3 r r s H& "
, I ID 1 D, DQ_ DG#4 G\ GTb GE!2 E< ER` EG#4 G\ GTb GE!2 E< ER` EF"3 FL FSa F N*; NL N[i NJ&7 J JWe JG#4 G\ GTb GD 1 D, DQ_ DU 1 U, UQ_ UT 0 T TP^ T@ - @l @^ @J, J\ JN JK'8 K KXf Kr5 r c 8 e Zd ZdZdededefdZdededefdZy)r zx
A [`TrainerCallback`] that handles the default flow of the training loop for logs, evaluation and checkpoints.
rX r/ r c |j dk( r|j rd|_ |j t j
k( r#|j |j z dk( rd|_ |j t j
k( r<|j |j z dk( r |j |j k rd|_
|j t j
k( r2|j dkD r#|j |j z dk( rd|_ |j |j k\ r+d|_ |j t j
k( rd|_ |S )Nr Tr )r logging_first_stepr logging_strategyr STEPSr
eval_strategyr
eval_delayr
save_strategyr
r r r r~ r s r3 r z DefaultFlowCallback.on_step_endC s# !d&=&=!%G $4$:$: :u?P?PSXSfSf?fjk?k!%G
"2"8"8 8!!E$4$4495#4#44&*G #
,"4"4 4 1$!!E$4$449"&G /+/G (!!\%7%77&*#r5 c |j t j k( rd|_ |j t j k( r |j
|j k rd|_ |j t j k( rd|_
|S )NT) r r EPOCHr r r r r r r
r r s r3 r z DefaultFlowCallback.on_epoch_endc sp $4$:$: :!%G !1!7!7 7DOOu{{ s@ 1 , Q_ @
!2
<
R`
r5 r c L e Zd ZdZd defdZd Zd Zd
dZd Z d Z
d
d
Z d Z y)ProgressCallbackz
A [`TrainerCallback`] that displays the progress of training or evaluation.
You can modify `max_str_len` to control how long strings are truncated when logging.
max_str_lenc . d| _ d| _ || _ y)a!
Initialize the callback with optional max_str_len parameter to control string truncation length.
Args:
max_str_len (`int`):
Maximum length of strings to display in logs.
Longer strings will be truncated with a message.
N) training_barprediction_barr )r0 r s r3 r zProgressCallback.__init__y s ! "&r5 c b |j r t |j d | _ d| _ y )NT)total
dynamic_ncolsr )r r r r current_stepr s r3 r z ProgressCallback.on_train_begin s& & & $5??$ OD r5 c |j rD| j j |j | j z
|j | _ y y r )r r updater r r s r3 r z ProgressCallback.on_step_end sC & & $ $U%6%69J9J%J K % 1 1D 'r5 Nc |j r\t | rP| j (t t | | j
d u d | _ | j j
d y y y )NT)r leaver r )r r r r lenr r )r0 rX r/ r r r s r3 r z#ProgressCallback.on_prediction_step sb & &:o+F""*&* o.d6G6G46O_c'#
& &q ) ,G &r5 c x |j r.| j | j j d | _ y y r r r closer s r3 r z ProgressCallback.on_evaluate 6 & &"".##))+"&D 'r5 c x |j r.| j | j j d | _ y y r r
r s r3 r zProgressCallback.on_predict r r5 c |j r| j i }|j D ]R \ }}t |t r8t | | j kD r dt | d| j d||< N|||< T |j dd } d|v rt |d d |d< | j j t | y y y )Nz%[String too long to display, length: z > z/. Consider increasing `max_str_len` if needed.]r r r8 )
r r rw r& rm r r poproundrE )
r0 rX r/ r r r shallow_logsry rz _s
r3 r zProgressCallback.on_log s & &4+<+<+H L
(1 a%#a&43C3C*C?Axs4K[K[J\ ]H H !O
'(LO
( t4A,&(-l7.CQ(G W% # #C $5 6! ,I &r5 c ` |j r"| j j d | _ y y r )r r r r s r3 r z ProgressCallback.on_train_end s* & & # # % $D 'r5 )d r )
r, rg rh ri rl r r r r r r r r rJ r5 r3 r r s s6
'C '
2
*' ' 7&%r5 r c e Zd ZdZddZy)PrinterCallbackz?
A bare [`TrainerCallback`] that just prints the logs.
Nc X |j dd }|j r t | y y )Nr )r r print)r0 rX r/ r r r r s r3 r zPrinterCallback.on_log s' HH\4 ( & & $K 'r5 r )r, rg rh ri r rJ r5 r3 r r s r5 r c F e Zd ZdZd dedee fdZd Zd Z d Z
de fd Z y
) EarlyStoppingCallbacka1
A [`TrainerCallback`] that handles early stopping.
Args:
early_stopping_patience (`int`):
Use with `metric_for_best_model` to stop training when the specified metric worsens for
`early_stopping_patience` evaluation calls.
early_stopping_threshold(`float`, *optional*):
Use with TrainingArguments `metric_for_best_model` and `early_stopping_patience` to denote how much the
specified metric must improve to satisfy early stopping conditions. `
This callback depends on [`TrainingArguments`] argument *load_best_model_at_end* functionality to set best_metric
in [`TrainerState`]. Note that if the [`TrainingArguments`] argument *save_steps* differs from *eval_steps*, the
early stopping will not occur until the next save step.
early_stopping_patienceearly_stopping_thresholdc . || _ || _ d| _ y )Nr r r early_stopping_patience_counter)r0 r r s r3 r z EarlyStoppingCallback.__init__ s '>$(@%/0,r5 c |j rt j nt j }|j 8 |||j r-t ||j z
| j kD rd| _ y | xj dz
c_ y )Nr r )greater_is_betternpgreaterlessr absr r )r0 rX r/ r metric_valueoperators r3 check_metric_valuez(EarlyStoppingCallback.check_metric_value sl !%!7!72::RWW $ \5#4#4 5L5#4#4458U8UU34D 0 0 0A 5 0r5 c |j st j d |j J d |j t
j k7 sJ d y )NzUsing EarlyStoppingCallback without load_best_model_at_end=True. Once training is finished, the best model will not be loaded automatically.zBEarlyStoppingCallback requires metric_for_best_model to be definedzAEarlyStoppingCallback requires IntervalStrategy of steps or epoch)load_best_model_at_endr r metric_for_best_modelr r NOr s r3 r z$EarlyStoppingCallback.on_train_begin sb ** NN^
))5
P
5 !!%5%8%88
O
8r5 c |j }|j d sd| }|j | }|t j d| d y | j |||| | j | j k\ rd|_ y y )Neval_z@early stopping required metric_for_best_model, but did not find z so early stopping is disabledT) r+
startswithgetr r r( r r r~ )r0 rX r/ r r r metric_to_checkr& s r3 r z!EarlyStoppingCallback.on_evaluate s 44 ))'2 %o%6 7O {{?3 NNRSbRc d
eWlC / /43O3O O+/G ( Pr5 rq c R | j | j dd| j idS )N)r r r r r rt s r3 r/ zEarlyStoppingCallback.state s7 ,0+G+G,0,I,I
243W3W
r5 N)r g )
r, rg rh ri rl r rj r r( r r r' r/ rJ r5 r3 r r s< 1 1S[\aSb 1 6
0"
t
r5 r )!ri rB r@ rU r typingr r numpyr" tqdm.autor
trainer_utilsr r r
r
training_argsr utilsr
get_loggerr, r r r( r} r# r r r r r rJ r5 r3 r: s ! " V V ,
H % XE XE XEv) )X :
_ :
:
zO
O
d@o @F2/ 2jG% G%To I
O_ I
r5