L iVddlZddlmZddlZddlmcmZddlmZddl m Z m Z m Z ddl mZddlmZddlmZgd ZGd d eZGd d eZGddeZGddeZGddeZGddeZGddeZGddeZGddeZGddeZGddeZGd d!eZGd"d#eZ Gd$d%eZ!Gd&d'eZ"Gd(d)eZ#Gd*d+eZ$Gd,d-eZ%Gd.d/eZ&Gd0d1eZ'Gd2d3eZ(d4eejd5e)fd6Z*d4eejd5e)fd7Z+d8Z,Gd9d:eZ-Gd;deZ/Gd?d@eZ0GdAdBeZ1GdCdDeZ2GdEdFeZ3GdGdHeZ4y)IN)Optional)Tensor) constant_xavier_normal_xavier_uniform_) Parameter)NonDynamicallyQuantizableLinear)Module) ThresholdReLURReLUHardtanhReLU6Sigmoid HardsigmoidTanhSiLUMish HardswishELUCELUSELUGLUGELU Hardshrink LeakyReLU LogSigmoidSoftplus SoftshrinkMultiheadAttentionPReLUSoftsign TanhshrinkSoftminSoftmax Softmax2d LogSoftmaxc zeZdZUdZgdZeed<eed<eed<d dedededdffd Zd e de fd Z de fd Z xZ S) r aThresholds each element of the input Tensor. Threshold is defined as: .. math:: y = \begin{cases} x, &\text{ if } x > \text{threshold} \\ \text{value}, &\text{ otherwise } \end{cases} Args: threshold: The value to threshold at value: The value to replace with inplace: can optionally do the operation in-place. Default: ``False`` Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/Threshold.png Examples:: >>> m = nn.Threshold(0, 0.5) >>> input = torch.arange(-3, 3) >>> output = m(input) ) thresholdvalueinplacer*r+r,returnNcLt|||_||_||_yN)super__init__r*r+r,)selfr*r+r, __class__s a/mnt/ssd/data/python-lab/Trading/venv/lib/python3.12/site-packages/torch/nn/modules/activation.pyr1zThreshold.__init__Ts# "  inputcntj||j|j|jS( Runs the forward pass. )Fr*r+r,r2r6s r4forwardzThreshold.forward[s%{{5$..$**dllKKr5c^|jrdnd}d|jd|j|S)@ Return the extra representation of the module. , inplace=Truez threshold=z, value=)r,r*r+r2 inplace_strs r4 extra_reprzThreshold.extra_repras2+/,,&B DNN+8DJJ< }MMr5F__name__ __module__ __qualname____doc__ __constants__float__annotations__boolr1rr<strrC __classcell__r3s@r4r r 0se:6M L M%QULVLL NCNr5r c\eZdZUdZdgZeed<d deddffd ZdedefdZ de fdZ xZ S) r aApplies the rectified linear unit function element-wise. :math:`\text{ReLU}(x) = (x)^+ = \max(0, x)` Args: inplace: can optionally do the operation in-place. Default: ``False`` Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/ReLU.png Examples:: >>> m = nn.ReLU() >>> input = torch.randn(2) >>> output = m(input) An implementation of CReLU - https://arxiv.org/abs/1603.05201 >>> m = nn.ReLU() >>> input = torch.randn(2).unsqueeze(0) >>> output = torch.cat((m(input), m(-input))) r,r-Nc0t|||_yr/r0r1r,r2r,r3s r4r1z ReLU.__init__  r5r6cDtj||jSr9r,)r:relur,r;s r4r<z ReLU.forwardvveT\\22r5c*|jrd}|Sd}|Sr>z inplace=Truer@rXrAs r4rCzReLU.extra_repr")- n ;= r5rD rFrGrHrIrJrMrLr1rr<rNrCrOrPs@r4r r isH6KM M3V33 Cr5r c |eZdZUdZgdZeed<eed<eed< d dedededdffd Zd e de fd Z de fd Z xZ S) raApplies the randomized leaky rectified linear unit function, element-wise. Method described in the paper: `Empirical Evaluation of Rectified Activations in Convolutional Network `_. The function is defined as: .. math:: \text{RReLU}(x) = \begin{cases} x & \text{if } x \geq 0 \\ ax & \text{ otherwise } \end{cases} where :math:`a` is randomly sampled from uniform distribution :math:`\mathcal{U}(\text{lower}, \text{upper})` during training while during evaluation :math:`a` is fixed with :math:`a = \frac{\text{lower} + \text{upper}}{2}`. Args: lower: lower bound of the uniform distribution. Default: :math:`\frac{1}{8}` upper: upper bound of the uniform distribution. Default: :math:`\frac{1}{3}` inplace: can optionally do the operation in-place. Default: ``False`` Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/RReLU.png Examples:: >>> m = nn.RReLU(0.1, 0.3) >>> input = torch.randn(2) >>> output = m(input) )lowerupperr,r`rar,r-NcLt|||_||_||_yr/)r0r1r`rar,)r2r`rar,r3s r4r1zRReLU.__init__s%    r5r6ctj||j|j|j|j Sr8)r:rrelur`ratrainingr,r;s r4r<z RReLU.forwards+wwudjj$**dmmT\\RRr5c^|jrdnd}d|jd|j|S)r>r?r@zlower=z, upper=)r,r`rarAs r4rCzRReLU.extra_reprs1+/,,&B  |8DJJ< }EEr5)g?gUUUUUU?FrErPs@r4rrso#J2M L L MOT-2GK SVSS FCFr5rc eZdZUdZgdZeed<eed<eed< ddedededeedeed df fd Z d e d e fd Z d e fd Z xZS)raApplies the HardTanh function element-wise. HardTanh is defined as: .. math:: \text{HardTanh}(x) = \begin{cases} \text{max\_val} & \text{ if } x > \text{ max\_val } \\ \text{min\_val} & \text{ if } x < \text{ min\_val } \\ x & \text{ otherwise } \\ \end{cases} Args: min_val: minimum value of the linear region range. Default: -1 max_val: maximum value of the linear region range. Default: 1 inplace: can optionally do the operation in-place. Default: ``False`` Keyword arguments :attr:`min_value` and :attr:`max_value` have been deprecated in favor of :attr:`min_val` and :attr:`max_val`. Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/Hardtanh.png Examples:: >>> m = nn.Hardtanh(-2, 2) >>> input = torch.randn(2) >>> output = m(input) )min_valmax_valr,rhrir,N min_value max_valuer-ct||tjdtd|}|tjdtd|}||_||_||_|j |j kDsJy)NzBkeyword argument `min_value` is deprecated and rename to `min_val`) stacklevelzBkeyword argument `max_value` is deprecated and rename to `max_val`)r0r1warningswarn FutureWarningrhrir,)r2rhrir,rjrkr3s r4r1zHardtanh.__init__s   MMT   G  MMT   G   ||dll***r5r6cntj||j|j|jSr8)r:hardtanhrhrir,r;s r4r<zHardtanh.forward s%zz%t||T\\JJr5c^|jrdnd}d|jd|j|S)r>r?r@zmin_val=z , max_val=)r,rhrirAs r4rCzHardtanh.extra_repr&s1+/,,&B $,,z$,, }MMr5)g?FNN)rFrGrHrIrJrKrLrMrr1rr<rNrCrOrPs@r4rrs@6M N N M%)%) +++ + E? + E? + +:KVKK NCNr5rc:eZdZdZddeddffd ZdefdZxZS)raApplies the ReLU6 function element-wise. .. math:: \text{ReLU6}(x) = \min(\max(0,x), 6) Args: inplace: can optionally do the operation in-place. Default: ``False`` Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/ReLU6.png Examples:: >>> m = nn.ReLU6() >>> input = torch.randn(2) >>> output = m(input) r,r-Nc(t|dd|y)Ng@)r0r1rTs r4r1zReLU6.__init__Ds c7+r5c*|jrd}|Sd}|Sr\rXrAs r4rCzReLU6.extra_reprGr]r5rD) rFrGrHrIrMr1rNrCrOrPs@r4rr.s'*,,,Cr5rc eZdZdZdedefdZy)raApplies the Sigmoid function element-wise. .. math:: \text{Sigmoid}(x) = \sigma(x) = \frac{1}{1 + \exp(-x)} Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/Sigmoid.png Examples:: >>> m = nn.Sigmoid() >>> input = torch.randn(2) >>> output = m(input) r6r-c,tj|Sr8)torchsigmoidr;s r4r<zSigmoid.forwardcs}}U##r5NrFrGrHrIrr<r5r4rrOs&$V$$r5rcPeZdZUdZdgZeed<ddeddffd ZdedefdZ xZ S) raApplies the Hardsigmoid function element-wise. Hardsigmoid is defined as: .. math:: \text{Hardsigmoid}(x) = \begin{cases} 0 & \text{if~} x \le -3, \\ 1 & \text{if~} x \ge +3, \\ x / 6 + 1 / 2 & \text{otherwise} \end{cases} Args: inplace: can optionally do the operation in-place. Default: ``False`` Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/Hardsigmoid.png Examples:: >>> m = nn.Hardsigmoid() >>> input = torch.randn(2) >>> output = m(input) r,r-Nc0t|||_yr/rSrTs r4r1zHardsigmoid.__init__rUr5r6cBtj||jSr8)r: hardsigmoidr,r;s r4r<zHardsigmoid.forwards}}UDLL11r5rD rFrGrHrIrJrMrLr1rr<rOrPs@r4rrjs<6KM M2V22r5rc eZdZdZdedefdZy)raApplies the Hyperbolic Tangent (Tanh) function element-wise. Tanh is defined as: .. math:: \text{Tanh}(x) = \tanh(x) = \frac{\exp(x) - \exp(-x)} {\exp(x) + \exp(-x)} Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/Tanh.png Examples:: >>> m = nn.Tanh() >>> input = torch.randn(2) >>> output = m(input) r6r-c,tj|Sr8)r|tanhr;s r4r<z Tanh.forwardszz%  r5Nr~rr5r4rrs(!V!!r5rc\eZdZUdZdgZeed<d deddffd ZdedefdZ de fdZ xZ S) raApplies the Sigmoid Linear Unit (SiLU) function, element-wise. The SiLU function is also known as the swish function. .. math:: \text{silu}(x) = x * \sigma(x), \text{where } \sigma(x) \text{ is the logistic sigmoid.} .. note:: See `Gaussian Error Linear Units (GELUs) `_ where the SiLU (Sigmoid Linear Unit) was originally coined, and see `Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning `_ and `Swish: a Self-Gated Activation Function `_ where the SiLU was experimented with later. Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/SiLU.png Examples:: >>> m = nn.SiLU() >>> input = torch.randn(2) >>> output = m(input) r,r-Nc0t|||_yr/rSrTs r4r1z SiLU.__init__rUr5r6cDtj||jSrW)r:silur,r;s r4r<z SiLU.forwardrZr5c*|jrd}|Sd}|Sr\rXrAs r4rCzSiLU.extra_reprr]r5rDr^rPs@r4rrsH8KM M3V33 Cr5rc\eZdZUdZdgZeed<d deddffd ZdedefdZ de fdZ xZ S) raxApplies the Mish function, element-wise. Mish: A Self Regularized Non-Monotonic Neural Activation Function. .. math:: \text{Mish}(x) = x * \text{Tanh}(\text{Softplus}(x)) .. note:: See `Mish: A Self Regularized Non-Monotonic Neural Activation Function `_ Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/Mish.png Examples:: >>> m = nn.Mish() >>> input = torch.randn(2) >>> output = m(input) r,r-Nc0t|||_yr/rSrTs r4r1z Mish.__init__rUr5r6cDtj||jSrW)r:mishr,r;s r4r<z Mish.forwardrZr5c*|jrd}|Sd}|Sr\rXrAs r4rCzMish.extra_reprr]r5rDr^rPs@r4rrsH.KM M3V33 Cr5rcPeZdZUdZdgZeed<ddeddffd ZdedefdZ xZ S) raApplies the Hardswish function, element-wise. Method described in the paper: `Searching for MobileNetV3 `_. Hardswish is defined as: .. math:: \text{Hardswish}(x) = \begin{cases} 0 & \text{if~} x \le -3, \\ x & \text{if~} x \ge +3, \\ x \cdot (x + 3) /6 & \text{otherwise} \end{cases} Args: inplace: can optionally do the operation in-place. Default: ``False`` Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/Hardswish.png Examples:: >>> m = nn.Hardswish() >>> input = torch.randn(2) >>> output = m(input) r,r-Nc0t|||_yr/rSrTs r4r1zHardswish.__init__2rUr5r6cBtj||jSr8)r: hardswishr,r;s r4r<zHardswish.forward6s{{5$,,//r5rDrrPs@r4rrs<:KM M0V00r5rcleZdZUdZddgZeed<eed<d dededdffd Zde de fdZ de fd Z xZ S) rasApplies the Exponential Linear Unit (ELU) function, element-wise. Method described in the paper: `Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) `__. ELU is defined as: .. math:: \text{ELU}(x) = \begin{cases} x, & \text{ if } x > 0\\ \alpha * (\exp(x) - 1), & \text{ if } x \leq 0 \end{cases} Args: alpha: the :math:`\alpha` value for the ELU formulation. Default: 1.0 inplace: can optionally do the operation in-place. Default: ``False`` Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/ELU.png Examples:: >>> m = nn.ELU() >>> input = torch.randn(2) >>> output = m(input) alphar,r-Nc>t|||_||_yr/r0r1rr,r2rr,r3s r4r1z ELU.__init__`   r5r6cXtj||j|jSr8)r:elurr,r;s r4r<z ELU.forwardesuuUDJJ 55r5cD|jrdnd}d|j|Sr>r?r@zalpha=r,rrAs r4rCzELU.extra_reprk'+/,,&B  |K=11r5ruFrErPs@r4rr=sW<i(M L MeDT 6V66 2C2r5rcleZdZUdZddgZeed<eed<d dededdffd Zde de fdZ de fd Z xZ S) ra3Applies the CELU function element-wise. .. math:: \text{CELU}(x) = \max(0,x) + \min(0, \alpha * (\exp(x/\alpha) - 1)) More details can be found in the paper `Continuously Differentiable Exponential Linear Units`_ . Args: alpha: the :math:`\alpha` value for the CELU formulation. Default: 1.0 inplace: can optionally do the operation in-place. Default: ``False`` Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/CELU.png Examples:: >>> m = nn.CELU() >>> input = torch.randn(2) >>> output = m(input) .. _`Continuously Differentiable Exponential Linear Units`: https://arxiv.org/abs/1704.07483 rr,r-Nc>t|||_||_yr/rrs r4r1z CELU.__init__rr5r6cXtj||j|jSr8)r:celurr,r;s r4r<z CELU.forwardsvveTZZ66r5cD|jrdnd}d|j|SrrrAs r4rCzCELU.extra_reprrr5rrErPs@r4rrssW6i(M L MeDT 7V77 2C2r5rc\eZdZUdZdgZeed<d deddffd ZdedefdZ de fdZ xZ S) raApplies the SELU function element-wise. .. math:: \text{SELU}(x) = \text{scale} * (\max(0,x) + \min(0, \alpha * (\exp(x) - 1))) with :math:`\alpha = 1.6732632423543772848170429916717` and :math:`\text{scale} = 1.0507009873554804934193349852946`. .. warning:: When using ``kaiming_normal`` or ``kaiming_normal_`` for initialisation, ``nonlinearity='linear'`` should be used instead of ``nonlinearity='selu'`` in order to get `Self-Normalizing Neural Networks`_. See :func:`torch.nn.init.calculate_gain` for more information. More details can be found in the paper `Self-Normalizing Neural Networks`_ . Args: inplace (bool, optional): can optionally do the operation in-place. Default: ``False`` Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/SELU.png Examples:: >>> m = nn.SELU() >>> input = torch.randn(2) >>> output = m(input) .. _Self-Normalizing Neural Networks: https://arxiv.org/abs/1706.02515 r,r-Nc0t|||_yr/rSrTs r4r1z SELU.__init__rUr5r6cBtj||jSr8)r:selur,r;s r4r<z SELU.forwardsvveT\\**r5c*|jrd}|Sd}|Sr\rXrAs r4rCzSELU.extra_reprr]r5rDr^rPs@r4rrsI DKM M+V++ Cr5rc\eZdZUdZdgZeed<d deddffd ZdedefdZ de fdZ xZ S) rajApplies the gated linear unit function. :math:`{GLU}(a, b)= a \otimes \sigma(b)` where :math:`a` is the first half of the input matrices and :math:`b` is the second half. Args: dim (int): the dimension on which to split the input. Default: -1 Shape: - Input: :math:`(\ast_1, N, \ast_2)` where `*` means, any number of additional dimensions - Output: :math:`(\ast_1, M, \ast_2)` where :math:`M=N/2` .. image:: ../scripts/activation_images/GLU.png Examples:: >>> m = nn.GLU() >>> input = torch.randn(4, 2) >>> output = m(input) dimr-Nc0t|||_yr/r0r1rr2rr3s r4r1z GLU.__init__ r5r6cBtj||jSr8)r:glurr;s r4r<z GLU.forwardsuuUDHH%%r5c d|jSr>zdim=rr2s r4rCzGLU.extra_reprdhhZ  r5)) rFrGrHrIrJintrLr1rr<rNrCrOrPs@r4rrsH,GM HC&V&& !C!r5rc\eZdZUdZdgZeed<d deddffd ZdedefdZ defdZ xZ S) ra/Applies the Gaussian Error Linear Units function. .. math:: \text{GELU}(x) = x * \Phi(x) where :math:`\Phi(x)` is the Cumulative Distribution Function for Gaussian Distribution. When the approximate argument is 'tanh', Gelu is estimated with: .. math:: \text{GELU}(x) = 0.5 * x * (1 + \text{Tanh}(\sqrt{2 / \pi} * (x + 0.044715 * x^3))) Args: approximate (str, optional): the gelu approximation algorithm to use: ``'none'`` | ``'tanh'``. Default: ``'none'`` Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/GELU.png Examples:: >>> m = nn.GELU() >>> input = torch.randn(2) >>> output = m(input) approximater-Nc0t|||_yr/)r0r1r)r2rr3s r4r1z GELU.__init__(s &r5r6cDtj||jS)r9)r)r:gelurr;s r4r<z GELU.forward,svve)9)9::r5c2dt|jS)r>z approximate=)reprrrs r4rCzGELU.extra_repr2sd4#3#34566r5)none) rFrGrHrIrJrNrLr1rr<rCrOrPs@r4rr sI6#OM'C'T';V;; 7C7r5rc\eZdZUdZdgZeed<d deddffd ZdedefdZ de fdZ xZ S) raApplies the Hard Shrinkage (Hardshrink) function element-wise. Hardshrink is defined as: .. math:: \text{HardShrink}(x) = \begin{cases} x, & \text{ if } x > \lambda \\ x, & \text{ if } x < -\lambda \\ 0, & \text{ otherwise } \end{cases} Args: lambd: the :math:`\lambda` value for the Hardshrink formulation. Default: 0.5 Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/Hardshrink.png Examples:: >>> m = nn.Hardshrink() >>> input = torch.randn(2) >>> output = m(input) lambdr-Nc0t|||_yr/r0r1rr2rr3s r4r1zHardshrink.__init__Y  r5r6cBtj||jSz# Run forward pass. )r: hardshrinkrr;s r4r<zHardshrink.forward]||E4::..r5c|jSr>)rrs r4rCzHardshrink.extra_reprcs**r5g? rFrGrHrIrJrKrLr1rr<rNrCrOrPs@r4rr9sH8IM Led/V// Cr5rcleZdZUdZddgZeed<eed<d dededdffd Zde de fdZ de fd Z xZ S) ra}Applies the LeakyReLU function element-wise. .. math:: \text{LeakyReLU}(x) = \max(0, x) + \text{negative\_slope} * \min(0, x) or .. math:: \text{LeakyReLU}(x) = \begin{cases} x, & \text{ if } x \geq 0 \\ \text{negative\_slope} \times x, & \text{ otherwise } \end{cases} Args: negative_slope: Controls the angle of the negative slope (which is used for negative input values). Default: 1e-2 inplace: can optionally do the operation in-place. Default: ``False`` Shape: - Input: :math:`(*)` where `*` means, any number of additional dimensions - Output: :math:`(*)`, same shape as the input .. image:: ../scripts/activation_images/LeakyReLU.png Examples:: >>> m = nn.LeakyReLU(0.1) >>> input = torch.randn(2) >>> output = m(input) r,negative_sloper-Nc>t|||_||_yr/)r0r1rr,)r2rr,r3s r4r1zLeakyReLU.__init__s , r5r6cXtj||j|jSr)r: leaky_relurr,r;s r4r<zLeakyReLU.forwards!||E4#6#6 EEr5cD|jrdnd}d|j|S)r>r?r@znegative_slope=)r,rrAs r4rCzLeakyReLU.extra_reprs*+/,,&B  !4!4 5k]CCr5)g{Gz?F)rFrGrHrIrJrMrLrKr1rr<rNrCrOrPs@r4rrjs_ D 01M Mudt FVFF DCDr5rc eZdZdZdedefdZy)raApplies the Logsigmoid function element-wise. .. math:: \text{LogSigmoid}(x) = \log\left(\frac{ 1 }{ 1 + \exp(-x)}\right) Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/LogSigmoid.png Examples:: >>> m = nn.LogSigmoid() >>> input = torch.randn(2) >>> output = m(input) r6r-c,tj|Sr)r: logsigmoidr;s r4r<zLogSigmoid.forward||E""r5Nr~rr5r4rr$#V##r5rcleZdZUdZddgZeed<eed<d dededdffd ZdedefdZ de fd Z xZ S) rayApplies the Softplus function element-wise. .. math:: \text{Softplus}(x) = \frac{1}{\beta} * \log(1 + \exp(\beta * x)) SoftPlus is a smooth approximation to the ReLU function and can be used to constrain the output of a machine to always be positive. For numerical stability the implementation reverts to the linear function when :math:`input \times \beta > threshold`. Args: beta: the :math:`\beta` value for the Softplus formulation. Default: 1 threshold: values above this revert to a linear function. Default: 20 Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/Softplus.png Examples:: >>> m = nn.Softplus() >>> input = torch.randn(2) >>> output = m(input) betar*r-Nc>t|||_||_yr/)r0r1rr*)r2rr*r3s r4r1zSoftplus.__init__s  "r5r6cXtj||j|jSr)r:softplusrr*r;s r4r<zSoftplus.forwardszz%DNN;;r5c:d|jd|jS)r>zbeta=z , threshold=)rr*rs r4rCzSoftplus.extra_reprs tyykdnn-=>>r5)rug4@rrPs@r4rrsX8[)M K#U#U#d#  \lambda \\ x + \lambda, & \text{ if } x < -\lambda \\ 0, & \text{ otherwise } \end{cases} Args: lambd: the :math:`\lambda` (must be no less than zero) value for the Softshrink formulation. Default: 0.5 Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/Softshrink.png Examples:: >>> m = nn.Softshrink() >>> input = torch.randn(2) >>> output = m(input) rr-Nc0t|||_yr/rrs r4r1zSoftshrink.__init__rr5r6cBtj||jSr)r: softshrinkrr;s r4r<zSoftshrink.forwardrr5c,t|jSr)rNrrs r4rCzSoftshrink.extra_reprs4::r5rrrPs@r4r r sH4IM Led/V// Cr5r xr-c|=|jjddtjjj fvSy)NcpucudaT)devicetyper|utilsbackend_registration_privateuse1_backend_namers r4_check_arg_devicer s?}xx}}   KK , , F F!   r5c | |jSy)NF) requires_gradrs r4_arg_requires_gradr*s} r5ctjjs:tjjj }t d|DSy)Nc3K|]>}t|tjjjj k(@ywr/)rr|fx experimental proxy_tensorProxyTorchDispatchMode.0rs r4 z&_is_make_fx_tracing..5s7  Guxx,,99PP P sAAF)r|jit is_scriptingr_python_dispatch _get_current_dispatch_mode_stackany)torch_dispatch_mode_stacks r4_is_make_fx_tracingr0sN 99 ! ! # KK ( ( I I K " .   r5c*eZdZUdZdgZeejed<eejed< d dfd Z ddZ fdZ dd ed ed ed eed e deede de de eeeffdZdeed eed ede eeeeffdZxZS)r!aAllows the model to jointly attend to information from different representation subspaces. This MultiheadAttention layer implements the original architecture described in the `Attention Is All You Need `_ paper. The intent of this layer is as a reference implementation for foundational understanding and thus it contains only limited features relative to newer architectures. Given the fast pace of innovation in transformer-like architectures, we recommend exploring this `tutorial `_ to build efficient layers from building blocks in core or using higher level libraries from the `PyTorch Ecosystem `_. Multi-Head Attention is defined as: .. math:: \text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1,\dots,\text{head}_h)W^O where :math:`\text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V)`. ``nn.MultiheadAttention`` will use the optimized implementations of ``scaled_dot_product_attention()`` when possible. In addition to support for the new ``scaled_dot_product_attention()`` function, for speeding up Inference, MHA will use fastpath inference with support for Nested Tensors, iff: - self attention is being computed (i.e., ``query``, ``key``, and ``value`` are the same tensor). - inputs are batched (3D) with ``batch_first==True`` - Either autograd is disabled (using ``torch.inference_mode`` or ``torch.no_grad``) or no tensor argument ``requires_grad`` - training is disabled (using ``.eval()``) - ``add_bias_kv`` is ``False`` - ``add_zero_attn`` is ``False`` - ``kdim`` and ``vdim`` are equal to ``embed_dim`` - if a `NestedTensor `_ is passed, neither ``key_padding_mask`` nor ``attn_mask`` is passed - autocast is disabled If the optimized inference fastpath implementation is in use, a `NestedTensor `_ can be passed for ``query``/``key``/``value`` to represent padding more efficiently than using a padding mask. In this case, a `NestedTensor `_ will be returned, and an additional speedup proportional to the fraction of the input that is padding can be expected. Args: embed_dim: Total dimension of the model. num_heads: Number of parallel attention heads. Note that ``embed_dim`` will be split across ``num_heads`` (i.e. each head will have dimension ``embed_dim // num_heads``). dropout: Dropout probability on ``attn_output_weights``. Default: ``0.0`` (no dropout). bias: If specified, adds bias to input / output projection layers. Default: ``True``. add_bias_kv: If specified, adds bias to the key and value sequences at dim=0. Default: ``False``. add_zero_attn: If specified, adds a new batch of zeros to the key and value sequences at dim=1. Default: ``False``. kdim: Total number of features for keys. Default: ``None`` (uses ``kdim=embed_dim``). vdim: Total number of features for values. Default: ``None`` (uses ``vdim=embed_dim``). batch_first: If ``True``, then the input and output tensors are provided as (batch, seq, feature). Default: ``False`` (seq, batch, feature). Examples:: >>> # xdoctest: +SKIP >>> multihead_attn = nn.MultiheadAttention(embed_dim, num_heads) >>> attn_output, attn_output_weights = multihead_attn(query, key, value) .. _`FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness`: https://arxiv.org/abs/2205.14135 batch_firstbias_kbias_vr-c |dks|dkrtd|d|d| | d} t | ||_||n||_||n||_|j|k(xr|j |k(|_||_||_| |_ ||z|_ |j|z|jk(sJd|j sttj||ffi| |_ttj||jffi| |_ttj||j ffi| |_|j#ddn_ttjd|z|ffi| |_|j#d d|j#d d|j#d d|r(ttjd|zfi| |_n|j#d dt)||fd |i| |_|rOttjdd|ffi| |_ttjdd|ffi| |_ndx|_|_||_|j3y)Nrz>embed_dim and num_heads must be greater than 0, got embed_dim=z and num_heads=z insteadrdtypez(embed_dim must be divisible by num_headsin_proj_weight q_proj_weight k_proj_weight v_proj_weight in_proj_biasbiasr ) ValueErrorr0r1 embed_dimkdimvdim_qkv_same_embed_dim num_headsdropoutrhead_dimrr|emptyrrrregister_parameterr rr out_projrr  add_zero_attn_reset_parameters)r2rrrr add_bias_kvrrrrrr factory_kwargsr3s r4r1zMultiheadAttention.__init__s >Y!^""+OI;hP %+U; " ,D)  ,D) #'99 #9#Tdii9>T " &!Y. }}y(DNN: 6 :''!* Y 2EnE"D "+ Y 2EnE"D "+ Y 2EnE"D   # #$4d ;"+ Q]I6I.I#D   # #OT :  # #OT :  # #OT :  )%++a)m*V~*V WD   # #ND 97 y '+ /=   #EKKAy0A$T^$TUDK#EKKAy0A$T^$TUDK(, ,DK$+*  r5c|jrt|jn?t|jt|jt|j |j 6t|j dt|jjd|jt|j|jt|jyy)Nrx) rrr rrrrrrrrrr rs r4r z$MultiheadAttention._reset_parameterss  # # D// 0 D.. / D.. / D.. /    ( d'' - dmm((# . ;; " 4;; ' ;; " 4;; ' #r5c6d|vrd|d<t||y)NrT)r0 __setstate__r2stater3s r4r%zMultiheadAttention.__setstate__s#  -+/E' ( U#r5querykeyr+key_padding_mask need_weights attn_maskaverage_attn_weights is_causalc  d} |tj|s|tj|rd} |jdk(} tj|dtj |d|j }tj|ddd|j d }tjjj} | sd } nl| sd |j} nU||us||urd } nI|jK|j |jj k7r(d |j d|jj d} n|jd} n|j |jj k7r(d |j d|jj d} n|jrd} n|jdzdk7rd} nt|jsd} ne|jd} nV|j d} nG|j"rd} n8|j$sd} n)|j&r||d} ntj(rd} | sw||||j|j|j*j,|j*j.f} tj0j3| rd} not5rd} nbt7d| Ds(d tj8j:j<} n(tj>rtAd!| Drd"} | s|jC|||\} }|j}|jqtjD||||jF|j|j|j|j*j,|j*j.| ||| S|j&xs|j&xs |j&}|r Jd#d$| z|jrB| r@||ur,||ur|jId%dx}x}}n!d&||fD\}}|}nd'|||fD\}}}|j$stjJ||||jF|j|j|j|j|j |j"|jL|j*j,|j*j.f |j|||d(|jN|jP|jR||d) \}}ntjJ||||jF|j|j|j|j|j |j"|jL|j*j,|j*j.|j|||||*\}}|jr| r|jId%d|fS||fS)+aICompute attention outputs using query, key, and value embeddings. Supports optional parameters for padding, masks and attention weights. Args: query: Query embeddings of shape :math:`(L, E_q)` for unbatched input, :math:`(L, N, E_q)` when ``batch_first=False`` or :math:`(N, L, E_q)` when ``batch_first=True``, where :math:`L` is the target sequence length, :math:`N` is the batch size, and :math:`E_q` is the query embedding dimension ``embed_dim``. Queries are compared against key-value pairs to produce the output. See "Attention Is All You Need" for more details. key: Key embeddings of shape :math:`(S, E_k)` for unbatched input, :math:`(S, N, E_k)` when ``batch_first=False`` or :math:`(N, S, E_k)` when ``batch_first=True``, where :math:`S` is the source sequence length, :math:`N` is the batch size, and :math:`E_k` is the key embedding dimension ``kdim``. See "Attention Is All You Need" for more details. value: Value embeddings of shape :math:`(S, E_v)` for unbatched input, :math:`(S, N, E_v)` when ``batch_first=False`` or :math:`(N, S, E_v)` when ``batch_first=True``, where :math:`S` is the source sequence length, :math:`N` is the batch size, and :math:`E_v` is the value embedding dimension ``vdim``. See "Attention Is All You Need" for more details. key_padding_mask: If specified, a mask of shape :math:`(N, S)` indicating which elements within ``key`` to ignore for the purpose of attention (i.e. treat as "padding"). For unbatched `query`, shape should be :math:`(S)`. Binary and float masks are supported. For a binary mask, a ``True`` value indicates that the corresponding ``key`` value will be ignored for the purpose of attention. For a float mask, it will be directly added to the corresponding ``key`` value. need_weights: If specified, returns ``attn_output_weights`` in addition to ``attn_outputs``. Set ``need_weights=False`` to use the optimized ``scaled_dot_product_attention`` and achieve the best performance for MHA. Default: ``True``. attn_mask: If specified, a 2D or 3D mask preventing attention to certain positions. Must be of shape :math:`(L, S)` or :math:`(N\cdot\text{num\_heads}, L, S)`, where :math:`N` is the batch size, :math:`L` is the target sequence length, and :math:`S` is the source sequence length. A 2D mask will be broadcasted across the batch while a 3D mask allows for a different mask for each entry in the batch. Binary and float masks are supported. For a binary mask, a ``True`` value indicates that the corresponding position is not allowed to attend. For a float mask, the mask values will be added to the attention weight. If both attn_mask and key_padding_mask are supplied, their types should match. average_attn_weights: If true, indicates that the returned ``attn_weights`` should be averaged across heads. Otherwise, ``attn_weights`` are provided separately per head. Note that this flag only has an effect when ``need_weights=True``. Default: ``True`` (i.e. average weights across heads) is_causal: If specified, applies a causal mask as attention mask. Default: ``False``. Warning: ``is_causal`` provides a hint that ``attn_mask`` is the causal mask. Providing incorrect hints can result in incorrect execution, including forward and backward compatibility. Outputs: - **attn_output** - Attention outputs of shape :math:`(L, E)` when input is unbatched, :math:`(L, N, E)` when ``batch_first=False`` or :math:`(N, L, E)` when ``batch_first=True``, where :math:`L` is the target sequence length, :math:`N` is the batch size, and :math:`E` is the embedding dimension ``embed_dim``. - **attn_output_weights** - Only returned when ``need_weights=True``. If ``average_attn_weights=True``, returns attention weights averaged across heads of shape :math:`(L, S)` when input is unbatched or :math:`(N, L, S)`, where :math:`N` is the batch size, :math:`L` is the target sequence length, and :math:`S` is the source sequence length. If ``average_attn_weights=False``, returns attention weights per head of shape :math:`(\text{num\_heads}, L, S)` when input is unbatched or :math:`(N, \text{num\_heads}, L, S)`. .. note:: `batch_first` argument is ignored for unbatched inputs. r@Nz5floating-point masks are not supported for fast path.rr*r,)mask mask_name other_type other_name target_typeF)r0r1r2r3r4 check_otherz6torch.backends.mha.get_fastpath_enabled() was not Truez5input not batched; expected query.dim() of 3 but got zKnon-self attention was used (query, key, and value are not the same Tensor)zdtypes of query (z) and self.in_proj_bias (z ) don't matchzin_proj_weight was Nonez) and self.in_proj_weight (ztraining is enabledrmrzself.num_heads is not evenzbatch_first was not Truezself.bias_k was not Nonezself.bias_v was not Nonezadd_zero_attn was enabledz _qkv_same_embed_dim was not Truezsupplying both src_key_padding_mask and src_mask at the same time is not supported with NestedTensor inputzautocast is enabledz'some Tensor argument has_torch_functionzwe are running make_fx tracingc32K|]}t|ywr/)rrs r4rz-MultiheadAttention.forward..sC!*1-Cz=some Tensor argument's device is neither one of cpu, cuda or c32K|]}t|ywr/)rrs r4rz-MultiheadAttention.forward..s1*+"1%1r7zhgrad is enabled and at least one of query or the input/output projection weights or biases requires_gradzKMultiheadAttention does not support NestedTensor outside of its fast path. z"The fast path was not hit because r c3@K|]}|jddywr rN transposers r4rz-MultiheadAttention.forward..s!J!++a"3!Jc3@K|]}|jddywr:r;rs r4rz-MultiheadAttention.forward..s$T1Q[[A%6$Tr=T) rer*r+r,use_separate_proj_weightrrrr-r.)rer*r+r,r-r.)*r|is_floating_pointrr:_canonical_mask_none_or_dtyper backendsmhaget_fastpath_enabledrr rerrrr rr is_nestedis_autocast_enabledrweightr overrideshas_torch_functionrallrrris_grad_enabledr merge_masks_native_multi_head_attentionrr<multi_head_attention_forwardrrrr)r2r(r)r+r*r+r,r-r.why_not_fast_path is_batchedis_fastpath_enabled tensor_args merged_mask mask_type any_nested attn_outputattn_output_weightss r4r<zMultiheadAttention.forwardsN  "u'>'>y'I ,''(89 W YY[A% ,,!('' 2"  %%!   $nn00EEG" X G }U # E!1!n     *u{{d>O>O>U>U/U"3EKK=@YZ^ZkZkZqZqYrr!A   ( 9  [[D//55 5"3EKK=@[\`\o\o\u\u[vwD!E  ]] 5 nnq Q & < !! :  [[ $ :  [[ $ :     ; )) B  __  (I,AK  & & ( 5  ##!! $$ ""K11+>$M!$&$D!C{CC$$)KK$D$D$^$^#_a"&&(S1/:1.N"%)-)9)9/*& Y$$0T5H5H5T ==++)) ,, **#$,! __H H  Y23D2EF G ~    e|C<*///!Q*??E?C%!JeS\!JJE3E$TU@S$T!sE''/0/M/M##!!  ""  $$ ""0!1)#)-"00"00"00%9#/0 ,K,401/M/M##!!  ""  $$ ""!1)#%9#'0 ,K,*    ((A.0CC C 33 3r5chd}d}|d}|}||j\}}}d}|jdk(r|j|d||} n0|jdd||j||jdd} | }|5|j|dd|jd|jdd} | | z}||fS)aDetermine mask type and combine masks if necessary. If only one mask is provided, that mask and the corresponding mask type will be returned. If both masks are provided, they will be both expanded to shape ``(batch_size, num_heads, seq_len, seq_len)``, combined with logical ``or`` and mask type 2 will be returned Args: attn_mask: attention mask of shape ``(seq_len, seq_len)``, mask type 0 key_padding_mask: padding mask of shape ``(batch_size, seq_len)``, mask type 1 query: query embeddings of shape ``(batch_size, seq_len, embed_dim)`` Returns: merged_mask: merged mask mask_type: merged mask type (0, 1, or 2) Nr rmrr)shaperviewexpandr) r2r,r*r(rUrT batch_sizeseq_len_attn_mask_expandedkey_padding_mask_expandeds r4rMzMultiheadAttention.merge_maskss($( (,  'I*K  %*[[ "JI}}!#%.^^JGW%U"%.^^Aq'7%K%R%RB&"-K+,<,A,A1g-&T^^R4*13LL I%%r5) rxTFFNNFNNr-N)NTNTF)rFrGrHrIrJrr|rrLr1r r%rMtupler<rrMrOrPs@r4r!r!=sDBH#OM U\\ "" U\\ ""    E! E!N( $.2!&*%)D4D4D4 D4 #6* D4  D4F#D4#D4D4 vx'' (D4L0&F#0&#6*0& 0& x#. / 0&r5r!cjeZdZUdZdgZeed< d dededdffd Zd dZ de de fd Z de fd Z xZS) r"aApplies the element-wise PReLU function. .. math:: \text{PReLU}(x) = \max(0,x) + a * \min(0,x) or .. math:: \text{PReLU}(x) = \begin{cases} x, & \text{ if } x \ge 0 \\ ax, & \text{ otherwise } \end{cases} Here :math:`a` is a learnable parameter. When called without arguments, `nn.PReLU()` uses a single parameter :math:`a` across all input channels. If called with `nn.PReLU(nChannels)`, a separate :math:`a` is used for each input channel. .. note:: weight decay should not be used when learning :math:`a` for good performance. .. note:: Channel dim is the 2nd dim of input. When input has dims < 2, then there is no channel dim and the number of channels = 1. Args: num_parameters (int): number of :math:`a` to learn. Although it takes an int as input, there is only two values are legitimate: 1, or the number of channels at input. Default: 1 init (float): the initial value of :math:`a`. Default: 0.25 Shape: - Input: :math:`( *)` where `*` means, any number of additional dimensions. - Output: :math:`(*)`, same shape as the input. Attributes: weight (Tensor): the learnable weights of shape (:attr:`num_parameters`). .. image:: ../scripts/activation_images/PReLU.png Examples:: >>> m = nn.PReLU() >>> input = torch.randn(2) >>> output = m(input) num_parametersNinitr-c||d}||_t| ||_t t j |fi||_|jy)Nr ) rer0r1rfrr|rrHreset_parameters)r2rerfrr r"r3s r4r1zPReLU.__init__RsO%+U;,   N Mn MN  r5ctjjj|j|jy)zW Resets parameters based on their initialization used in ``__init__``. N)r|nnrfrrHrs r4rhzPReLU.reset_parameters\s$   TYY7r5r6cBtj||jSr8)r:prelurHr;s r4r<z PReLU.forwardbswwudkk**r5c d|jS)r>znum_parameters=)rers r4rCzPReLU.extra_reprhs!!4!4 566r5)r g?NNrb)rFrGrHrIrJrrLrKr1rhrr<rNrCrOrPs@r4r"r"sa/b&&MOS ! -2  8 +V++ 7C7r5r"c eZdZdZdedefdZy)r#aApplies the element-wise Softsign function. .. math:: \text{SoftSign}(x) = \frac{x}{ 1 + |x|} Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/Softsign.png Examples:: >>> m = nn.Softsign() >>> input = torch.randn(2) >>> output = m(input) r6r-c,tj|Sr8)r:softsignr;s r4r<zSoftsign.forwardszz%  r5Nr~rr5r4r#r#os$!V!!r5r#c eZdZdZdedefdZy)r$aApplies the element-wise Tanhshrink function. .. math:: \text{Tanhshrink}(x) = x - \tanh(x) Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/Tanhshrink.png Examples:: >>> m = nn.Tanhshrink() >>> input = torch.randn(2) >>> output = m(input) r6r-c,tj|Sr8)r: tanhshrinkr;s r4r<zTanhshrink.forwardrr5Nr~rr5r4r$r$rr5r$creZdZUdZdgZeeed<d deeddffd ZfdZ de de fdZ de fd Z xZS) r%a:Applies the Softmin function to an n-dimensional input Tensor. Rescales them so that the elements of the n-dimensional output Tensor lie in the range `[0, 1]` and sum to 1. Softmin is defined as: .. math:: \text{Softmin}(x_{i}) = \frac{\exp(-x_i)}{\sum_j \exp(-x_j)} Shape: - Input: :math:`(*)` where `*` means, any number of additional dimensions - Output: :math:`(*)`, same shape as the input Args: dim (int): A dimension along which Softmin will be computed (so every slice along dim will sum to 1). Returns: a Tensor of the same dimension and shape as the input, with values in the range [0, 1] Examples:: >>> m = nn.Softmin(dim=1) >>> input = torch.randn(2, 3) >>> output = m(input) rNr-c0t|||_yr/rrs r4r1zSoftmin.__init__rr5cLt||t|dsd|_yyNrr0r%hasattrrr&s r4r%zSoftmin.__setstate__& U#tU#DH$r5r6cFtj||jdSr9 _stacklevel)r:softminrr;s r4r<zSoftmin.forwardyya88r5c d|jSrrrs r4rCzSoftmin.extra_reprrr5r/rFrGrHrIrJrrrLr1r%rr<rNrCrOrPs@r4r%r%sV<GM #HSMT 9V99 !C!r5r%creZdZUdZdgZeeed<d deeddffd ZfdZ de de fdZ de fd Z xZS) r&aApplies the Softmax function to an n-dimensional input Tensor. Rescales them so that the elements of the n-dimensional output Tensor lie in the range [0,1] and sum to 1. Softmax is defined as: .. math:: \text{Softmax}(x_{i}) = \frac{\exp(x_i)}{\sum_j \exp(x_j)} When the input Tensor is a sparse tensor then the unspecified values are treated as ``-inf``. Shape: - Input: :math:`(*)` where `*` means, any number of additional dimensions - Output: :math:`(*)`, same shape as the input Returns: a Tensor of the same dimension and shape as the input with values in the range [0, 1] Args: dim (int): A dimension along which Softmax will be computed (so every slice along dim will sum to 1). .. note:: This module doesn't work directly with NLLLoss, which expects the Log to be computed between the Softmax and itself. Use `LogSoftmax` instead (it's faster and has better numerical properties). Examples:: >>> m = nn.Softmax(dim=1) >>> input = torch.randn(2, 3) >>> output = m(input) rNr-c0t|||_yr/rrs r4r1zSoftmax.__init__rr5cLt||t|dsd|_yyrwrxr&s r4r%zSoftmax.__setstate__ rzr5r6cFtj||jdSr|)r:softmaxrr;s r4r<zSoftmax.forwardrr5c d|jSrrrs r4rCzSoftmax.extra_reprrr5r/rrPs@r4r&r&sW%NGM #HSMT 9V99 !C!r5r&c eZdZdZdedefdZy)r'a|Applies SoftMax over features to each spatial location. When given an image of ``Channels x Height x Width``, it will apply `Softmax` to each location :math:`(Channels, h_i, w_j)` Shape: - Input: :math:`(N, C, H, W)` or :math:`(C, H, W)`. - Output: :math:`(N, C, H, W)` or :math:`(C, H, W)` (same shape as input) Returns: a Tensor of the same dimension and shape as the input with values in the range [0, 1] Examples:: >>> m = nn.Softmax2d() >>> # you softmax over the 2nd dimension >>> input = torch.randn(2, 3, 12, 13) >>> output = m(input) r6r-c|jdvrtd|jdtj|ddS)r9)rz.Softmax2d: expected input to be 3D or 4D, got z D insteadr}r~)rrr:rr;s r4r<zSoftmax2d.forward2sF 99;f $@ YW yy22r5Nr~rr5r4r'r's*3V33r5r'creZdZUdZdgZeeed<d deeddffd ZfdZ de de fdZ de fd Z xZS) r(aApplies the :math:`\log(\text{Softmax}(x))` function to an n-dimensional input Tensor. The LogSoftmax formulation can be simplified as: .. math:: \text{LogSoftmax}(x_{i}) = \log\left(\frac{\exp(x_i) }{ \sum_j \exp(x_j)} \right) Shape: - Input: :math:`(*)` where `*` means, any number of additional dimensions - Output: :math:`(*)`, same shape as the input Args: dim (int): A dimension along which LogSoftmax will be computed. Returns: a Tensor of the same dimension and shape as the input with values in the range [-inf, 0) Examples:: >>> m = nn.LogSoftmax(dim=1) >>> input = torch.randn(2, 3) >>> output = m(input) rNr-c0t|||_yr/rrs r4r1zLogSoftmax.__init__[rr5cLt||t|dsd|_yyrwrxr&s r4r%zLogSoftmax.__setstate___rzr5r6cFtj||jdSr|)r: log_softmaxrr;s r4r<zLogSoftmax.forwardds}}UDHH!<rs DD(3 B6N6Nr.6.b?FF?FDONvONdHB$f$6(2&(2V!6!8/6/d*6*Z*0*0Z32&32l02602f565p(!&(!V-76-7`..b7D7Dt##40?v0?f,,^%,,/D(5<<0T ]&]&@O7FO7d!v!4##45!f5!p>!f>!B33B1!1!r5