L iDdZddlZddlmZmZddlmZddlZGddeZGddeZ Gd d eZ Gd d eZ Gd deZ GddeZ GddeZGddeZdZdZd$dZdZd$dZd$dZdZdZdZdZdZd Zd!Zd"Zd#Zy)%zPruning methods.N)ABCabstractmethod)Iterablec\eZdZUdZeed<dZedZdZ e dddZ d d Z d Z y) BasePruningMethodzAbstract base class for creation of new pruning techniques. Provides a skeleton for customization requiring the overriding of methods such as :meth:`compute_mask` and :meth:`apply`. _tensor_namecPt||j|j|y)aMultiply the mask into original tensor and store the result. Multiplies the mask (stored in ``module[name + '_mask']``) into the original tensor (stored in ``module[name + '_orig']``) and stores the result into ``module[name]`` by using :meth:`apply_mask`. Args: module (nn.Module): module containing the tensor to prune inputs: not used. N)setattrr apply_mask)selfmoduleinputss Z/mnt/ssd/data/python-lab/Trading/venv/lib/python3.12/site-packages/torch/nn/utils/prune.py__call__zBasePruningMethod.__call__s ))4??6+BCcy)aCompute and returns a mask for the input tensor ``t``. Starting from a base ``default_mask`` (which should be a mask of ones if the tensor has not been pruned yet), generate a random mask to apply on top of the ``default_mask`` according to the specific pruning method recipe. Args: t (torch.Tensor): tensor representing the importance scores of the parameter to prune. default_mask (torch.Tensor): Base mask from previous pruning iterations, that need to be respected after the new mask is applied. Same dims as ``t``. Returns: mask (torch.Tensor): mask to apply to ``t``, of same dims as ``t`` N)r t default_masks r compute_maskzBasePruningMethod.compute_mask!src|j Jd|dt||jdz}t||jdz}|j|j|z}|S)aSimply handles the multiplication between the parameter being pruned and the generated mask. Fetches the mask and the original tensor from the module and returns the pruned version of the tensor. Args: module (nn.Module): module containing the tensor to prune Returns: pruned_tensor (torch.Tensor): pruned version of the input tensor Module z has to be pruned_mask_origdtype)rgetattrtor)r r maskorig pruned_tensors rr zBasePruningMethod.apply_mask5sy  , fX. / ,vt007:;vt007:;djj1D8 rNimportance_scorescd}||||g|i|}t||}|&|j|jk(sJd|d||}t|ts8|j |dz||j |=t j|} n;t||dzjjt j} |j|| } |j|dz| t|||j||j||S#t $rH} t|ts1t||dz}|j |||j |dz=| d} ~ wwxYw) aAdd pruning on the fly and reparametrization of a tensor. Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask. Args: module (nn.Module): module containing the tensor to prune name (str): parameter name within ``module`` on which pruning will act. args: arguments passed on to a subclass of :class:`BasePruningMethod` importance_scores (torch.Tensor): tensor of importance scores (of same shape as module parameter) used to compute mask for pruning. The values in this tensor indicate the importance of the corresponding elements in the parameter being pruned. If unspecified or None, the parameter will be used in its place. kwargs: keyword arguments passed on to a subclass of a :class:`BasePruningMethod` cd}d}g}|jjD]>\}} t| ts| j|k(s'| }|j ||dz }@|dksJd|d|d|D]}|j|=||i|} || _|St|t r|j| |} | St|trt |} | j| | } | S)NrzFAvoid adding multiple pruning hooks to the same tensor z of module z. Use a PruningContainer.)_forward_pre_hooksitems isinstancerrappendPruningContaineradd_pruning_method) clsr nameargskwargs old_methodfoundhooks_to_removekhookmethod containers r_get_composite_methodz6BasePruningMethod.apply.._get_composite_methodbs4JE!O!44::< 4d$564;L;LPT;T!%J#**1-QJE  A: !F+fX5NP : % 1--a0 1 $)&)F"&F %j*:;11&9'FM  ,=> 0 4 8 FD&"3"3F"; <  , ,V 4  f&67vtg~6))$5&&tg~6G  s AD## E4,AE//E4c| |j|jk(s Jd|}||ntj|}||j||zS)aCompute and returns a pruned version of input tensor ``t``. According to the pruning rule specified in :meth:`compute_mask`. Args: t (torch.Tensor): tensor to prune (of same dimensions as ``default_mask``). importance_scores (torch.Tensor): tensor of importance scores (of same shape as ``t``) used to compute mask for pruning ``t``. The values in this tensor indicate the importance of the corresponding elements in the ``t`` that is being pruned. If unspecified or None, the tensor ``t`` will be used in its place. default_mask (torch.Tensor, optional): mask from previous pruning iteration, if any. To be considered when determining what portion of the tensor that pruning should act on. If None, default to a mask of ones. Returns: pruned version of tensor ``t``. z8importance_scores should have the same shape as tensor tr;)r<r?r@r)r rrr#s rprunezBasePruningMethod.prunesf*  ($**agg5 J 5!" '3'?|U__UVEW 4$$%6\$RRRrc|j Jd|d|j|}t||jrt||j|j|jdz}|j |_|j|jdz=|j |jdz=t||j|y)arRemove the pruning reparameterization from a module. The pruned parameter named ``name`` remains permanently pruned, and the parameter named ``name+'_orig'`` is removed from the parameter list. Similarly, the buffer named ``name+'_mask'`` is removed from the buffers. Note: Pruning itself is NOT undone or reversed! Nrz: has to be pruned before pruning can be removedrr)rr hasattrdelattrr>data_buffersr )r r weightr s rremovezBasePruningMethod.removes  , fXW X , ( 64,, - FD-- .!!$"3"3g"=>KK   t007: ; OOD--7 8))40r)NN)__name__ __module__ __qualname____doc__str__annotations__rrrr classmethodrHrJrQrrrrr sS  D  &,:>~~@S<1rrc4eZdZdZdZdZdZdZdZdZ y) r+aContainer holding a sequence of pruning methods for iterative pruning. Keeps track of the order in which pruning methods are applied and handles combining successive pruning calls. Accepts as argument an instance of a BasePruningMethod or an iterable of them. cd|_t|ts#|j|_|j |yt |dk(r)|dj|_|j |dy|D]}|j |y)Nrr&r)_pruning_methodsr)rrr,len)r r/r6s r__init__zPruningContainer.__init__s|?A$) $ 1 1D   # #D ) Y!^ $Q 4 4D   # #DG , 0''/ 0rct|ts|tt|d|F|j|jk7r-t d|jd|dd|jdz|xj |fz c_y)zAdd a child pruning ``method`` to the container. Args: method (subclass of BasePruningMethod): child pruning method to be added to the container. Nz$ is not a BasePruningMethod subclasszd}|jd}||||}|S)aApply the latest ``method`` by computing the new partial masks and returning its combination with the ``default_mask``. The new partial mask should be computed on the entries or channels that were not zeroed out by the ``default_mask``. Which portions of the tensor ``t`` the new mask will be calculated from depends on the ``PRUNING_TYPE`` (handled by the type handler): * for 'unstructured', the mask will be computed from the raveled list of nonmasked entries; * for 'structured', the mask will be computed from the nonmasked channels in the tensor; * for 'global', the mask will be computed across all entries. Args: t (torch.Tensor): tensor representing the parameter to prune (of same dimensions as ``default_mask``). default_mask (torch.Tensor): mask from previous pruning iteration. Returns: mask (torch.Tensor): new mask that combines the effects of the ``default_mask`` and the new mask from the current pruning ``method`` (of same dimensions as ``default_mask`` and ``t``). c|}|j|j}|jdk(r|dk(}n|jdk(rt|ds t d|j }|j }|dkr||z}|dkrt d||jt|Dcgc] }||k7s | c} dk7}td g|z}|||<nL|jd k(r%t|j}td g|z}ntd |jt|tr t|}|j!|||| } | j|j||<|Scc}w)aCombine the masks from all pruning methods and returns a new mask. Args: method (a BasePruningMethod subclass): pruning method currently being applied. t (torch.Tensor): tensor representing the parameter to prune (of same dimensions as mask). mask (torch.Tensor): mask from previous pruning iteration Returns: new_mask (torch.Tensor): new mask that combines the effects of the old mask and the new mask from the current pruning method (of same dimensions as mask and t). r unstructuredr& structureddimzVPruning methods of PRUNING_TYPE "structured" need to have the attribute `dim` defined.rz2Index is out of bounds for tensor with dimensions )rsNglobalzUnrecognized PRUNING_TYPE r;)rr PRUNING_TYPErLAttributeErrorrs IndexErrorsumrangeslicer\r<rcr)listtupler) r6rrnew_maskslcn_dimsrsd keep_channel partial_masks r_combine_masksz5PruningContainer.compute_mask.._combine_masksTsH{{{1H""n4ai$$ 4vu-(Qjj7 3,C7$LVHU $xxf ,R1cQ,RxSWXX T{mf,'C$$0QWWT{mf,!#=f>Q>Q=R!STT#t$Cj!..qvDI.NL(OO(..OAHSMO%-Ss + E;6E;rl)r rrrr6rs rrzPruningContainer.compute_mask8s-8? B&&r*fa6 rN) rRrSrTrUr]r,rgrjrnrrrrr+r+s& 0+&*+*_rr+c6eZdZdZdZdZefdZxZS)IdentityzsUtility pruning method that does not prune any units but generates the pruning parametrization with a mask of ones.rqc |}|Srerr rrrs rrzIdentity.compute_masks  rc$t|||S)Add pruning on the fly and reparametrization of a tensor. Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask. Args: module (nn.Module): module containing the tensor to prune name (str): parameter name within ``module`` on which pruning will act. superrH)r-r r. __class__s rrHzIdentity.applysw}VT**r) rRrSrTrUrurrXrH __classcell__rs@rrrs$~!L + +rrc<eZdZdZdZdZdZefdZxZ S)RandomUnstructuredaPrune (currently unpruned) units in a tensor at random. Args: name (str): parameter name within ``module`` on which pruning will act. amount (int or float): quantity of parameters to prune. If ``float``, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If ``int``, it represents the absolute number of parameters to prune. rqc(t|||_yre_validate_pruning_amount_initamountr rs rr]zRandomUnstructured.__init__%f- rcf|j}t|j|}t|||j t j }|dk7rYt j|}t j|jd|}d|jd|j<|S)Nr9rrr4) nelement_compute_nparams_topruner_validate_pruning_amountrBr?rC rand_liketopkviewindices)r rr tensor_sizenparams_toprunerprobrs rrzRandomUnstructured.compute_masksjjl 34;; L !+>!!0G0G!H a ??1%D::diim?D*+DIIbM$,, ' rc(t||||S)aAdd pruning on the fly and reparametrization of a tensor. Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask. Args: module (nn.Module): module containing the tensor to prune name (str): parameter name within ``module`` on which pruning will act. amount (int or float): quantity of parameters to prune. If ``float``, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If ``int``, it represents the absolute number of parameters to prune. rr)r-r r.rrs rrHzRandomUnstructured.applys"w}VT&}99r rRrSrTrUrur]rrXrHrrs@rrrs- "L (::rrc>eZdZdZdZdZdZedfd ZxZ S)L1UnstructuredavPrune (currently unpruned) units in a tensor by zeroing out the ones with the lowest L1-norm. Args: amount (int or float): quantity of parameters to prune. If ``float``, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If ``int``, it represents the absolute number of parameters to prune. rqc(t|||_yrerrs rr]zL1Unstructured.__init__rrcd|j}t|j|}t|||j t j }|dk7rXt jt j|jd|d}d|jd|j<|S)Nr9rrFr4largest) rrrrrBr?rCrabsrr)r rrrrrrs rrzL1Unstructured.compute_masksjjl 34;; L !+>!!0G0G!H a ::eiil//3PUVD*+DIIbM$,, ' rc*t|||||S)aJAdd pruning on the fly and reparametrization of a tensor. Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask. Args: module (nn.Module): module containing the tensor to prune name (str): parameter name within ``module`` on which pruning will act. amount (int or float): quantity of parameters to prune. If ``float``, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If ``int``, it represents the absolute number of parameters to prune. importance_scores (torch.Tensor): tensor of importance scores (of same shape as module parameter) used to compute mask for pruning. The values in this tensor indicate the importance of the corresponding elements in the parameter being pruned. If unspecified or None, the module parameter will be used in its place. rr#r)r-r r.rr#rs rrHzL1Unstructured.applys%,w} D;L  rrerrs@rrrs-"L ,  rrc@eZdZdZdZddZdZedfd ZxZ S)RandomStructuredaPrune entire (currently unpruned) channels in a tensor at random. Args: amount (int or float): quantity of parameters to prune. If ``float``, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If ``int``, it represents the absolute number of parameters to prune. dim (int, optional): index of the dim along which we define channels to prune. Default: -1. rrc6t|||_||_yre)rrrs)r rrss rr]zRandomStructured.__init__>s%f- rc>t|t||j|j|j}t |j |}t ||d}|dk(r|}|S|||j||}||j|jz}|S)aCompute and returns a mask for the input tensor ``t``. Starting from a base ``default_mask`` (which should be a mask of ones if the tensor has not been pruned yet), generate a random mask to apply on top of the ``default_mask`` by randomly zeroing out channels along the specified dim of the tensor. Args: t (torch.Tensor): tensor representing the parameter to prune default_mask (torch.Tensor): Base mask from previous pruning iterations, that need to be respected after the new mask is applied. Same dims as ``t``. Returns: mask (torch.Tensor): mask to apply to ``t``, of same dims as ``t`` Raises: IndexError: if ``self.dim >= len(t.shape)`` ctj|}tj||j}||kD}tj|}t dgt |jz}|||<t|}d||<|S)Nrr&) r?randkthvaluevalues zeros_likerzr\r<r|) rrs nchannelsnchannels_topruner threshold channel_maskrr~s r make_maskz0RandomStructured.compute_mask..make_maskls{::i(Dt/@AHHI)+L##A&D;-#agg,.C#CH*CDIKrrr) _validate_structured_pruning_validate_pruning_dimrsr<rrrrr)r rrrrrrs rrzRandomStructured.compute_maskDs, %Q' a*ggdhh' 34;; L !+>  a D  Q+GD LOO$**O5 5D rc*t|||||S)a;Add pruning on the fly and reparametrization of a tensor. Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask. Args: module (nn.Module): module containing the tensor to prune name (str): parameter name within ``module`` on which pruning will act. amount (int or float): quantity of parameters to prune. If ``float``, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If ``int``, it represents the absolute number of parameters to prune. dim (int, optional): index of the dim along which we define channels to prune. Default: -1. )rrsr)r-r r.rrsrs rrHzRandomStructured.applys&w}VT&c}BBrrrrs@rrr0s0  L >@CCrrc@eZdZdZdZddZdZedfd ZxZ S) LnStructuredaePrune entire (currently unpruned) channels in a tensor based on their L\ ``n``-norm. Args: amount (int or float): quantity of channels to prune. If ``float``, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If ``int``, it represents the absolute number of parameters to prune. n (int, float, inf, -inf, 'fro', 'nuc'): See documentation of valid entries for argument ``p`` in :func:`torch.norm`. dim (int, optional): index of the dim along which we define channels to prune. Default: -1. rrcDt|||_||_||_yre)rrnrs)r rrrss rr]zLnStructured.__init__s%f- rct|t||j|j|j}t |j |}||z }t ||t||j|j}tj||d}d}|dk(r|} | S|||j|j} | |j| jz} | S)aCompute and returns a mask for the input tensor ``t``. Starting from a base ``default_mask`` (which should be a mask of ones if the tensor has not been pruned yet), generate a mask to apply on top of the ``default_mask`` by zeroing out the channels along the specified dim with the lowest L\ ``n``-norm. Args: t (torch.Tensor): tensor representing the parameter to prune default_mask (torch.Tensor): Base mask from previous pruning iterations, that need to be respected after the new mask is applied. Same dims as ``t``. Returns: mask (torch.Tensor): mask to apply to ``t``, of same dims as ``t`` Raises: IndexError: if ``self.dim >= len(t.shape)`` Trctj|}tdgt|jz}|||<t |}d||<|S)Nr&)r?rrzr\r<r|)rrsrrr~s rrz,LnStructured.compute_mask..make_masksN##A&D;-#agg,.CCH*CDIKrrr)rrrsr<rrr _compute_normrr?rrrr) r rrrrnparams_tokeepnormrrrs rrzLnStructured.compute_masks, %Q'a*ggdhh' 34;; L$6 !+> Q1zz$.$?  a D  Q$,,7D LOO$**O5 5D rc.t|||||||S)a=Add pruning on the fly and reparametrization of a tensor. Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask. Args: module (nn.Module): module containing the tensor to prune name (str): parameter name within ``module`` on which pruning will act. amount (int or float): quantity of parameters to prune. If ``float``, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If ``int``, it represents the absolute number of parameters to prune. n (int, float, inf, -inf, 'fro', 'nuc'): See documentation of valid entries for argument ``p`` in :func:`torch.norm`. dim (int): index of the dim along which we define channels to prune. importance_scores (torch.Tensor): tensor of importance scores (of same shape as module parameter) used to compute mask for pruning. The values in this tensor indicate the importance of the corresponding elements in the parameter being pruned. If unspecified or None, the module parameter will be used in its place. )rrrsr#r)r-r r.rrrsr#rs rrHzLnStructured.applys-4w}  /   rrrerrs@rrrs/  LEN    rrc8eZdZdZdZdZefdZxZS)CustomFromMaskrtc||_yrer)r rs rr]zCustomFromMask.__init__s  rc|j|jjk(sJ||jj|jz}|S)Nr)r<rrrrs rrzCustomFromMask.compute_mask"sB!!TYY__444diill1C1ClDD rc(t||||S)rrr)r-r r.rrs rrHzCustomFromMask.apply'sw}VT}55r) rRrSrTrur]rrXrHrrs@rrrs&L  6 6rrc2tj|||S)aBApply pruning reparametrization without pruning any units. Applies pruning reparametrization to the tensor corresponding to the parameter called ``name`` in ``module`` without actually pruning any units. Modifies module in place (and also return the modified module) by: 1) adding a named buffer called ``name+'_mask'`` corresponding to the binary mask applied to the parameter ``name`` by the pruning method. 2) replacing the parameter ``name`` by its pruned version, while the original (unpruned) parameter is stored in a new parameter named ``name+'_orig'``. Note: The mask is a tensor of ones. Args: module (nn.Module): module containing the tensor to prune. name (str): parameter name within ``module`` on which pruning will act. Returns: module (nn.Module): modified (i.e. pruned) version of the input module Examples: >>> # xdoctest: +SKIP >>> m = prune.identity(nn.Linear(2, 3), "bias") >>> print(m.bias_mask) tensor([1., 1., 1.]) )rrH)r r.s ridentityr7s> NN64 Mrc4tj||||S)a?Prune tensor by removing random (currently unpruned) units. Prunes tensor corresponding to parameter called ``name`` in ``module`` by removing the specified ``amount`` of (currently unpruned) units selected at random. Modifies module in place (and also return the modified module) by: 1) adding a named buffer called ``name+'_mask'`` corresponding to the binary mask applied to the parameter ``name`` by the pruning method. 2) replacing the parameter ``name`` by its pruned version, while the original (unpruned) parameter is stored in a new parameter named ``name+'_orig'``. Args: module (nn.Module): module containing the tensor to prune name (str): parameter name within ``module`` on which pruning will act. amount (int or float): quantity of parameters to prune. If ``float``, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If ``int``, it represents the absolute number of parameters to prune. Returns: module (nn.Module): modified (i.e. pruned) version of the input module Examples: >>> # xdoctest: +SKIP >>> m = prune.random_unstructured(nn.Linear(2, 3), "weight", amount=1) >>> torch.sum(m.weight_mask == 0) tensor(1) )rrH)r r.rs rrandom_unstructuredrZsBVT62 Mrc8tj|||||S)aPrune tensor by removing units with the lowest L1-norm. Prunes tensor corresponding to parameter called ``name`` in ``module`` by removing the specified `amount` of (currently unpruned) units with the lowest L1-norm. Modifies module in place (and also return the modified module) by: 1) adding a named buffer called ``name+'_mask'`` corresponding to the binary mask applied to the parameter ``name`` by the pruning method. 2) replacing the parameter ``name`` by its pruned version, while the original (unpruned) parameter is stored in a new parameter named ``name+'_orig'``. Args: module (nn.Module): module containing the tensor to prune name (str): parameter name within ``module`` on which pruning will act. amount (int or float): quantity of parameters to prune. If ``float``, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If ``int``, it represents the absolute number of parameters to prune. importance_scores (torch.Tensor): tensor of importance scores (of same shape as module parameter) used to compute mask for pruning. The values in this tensor indicate the importance of the corresponding elements in the parameter being pruned. If unspecified or None, the module parameter will be used in its place. Returns: module (nn.Module): modified (i.e. pruned) version of the input module Examples: >>> # xdoctest: +SKIP >>> m = prune.l1_unstructured(nn.Linear(2, 3), "weight", amount=0.2) >>> m.state_dict().keys() odict_keys(['bias', 'weight_orig', 'weight_mask']) r)rrH)r r.rr#s rl1_unstructuredrs)LV7H Mrc6tj|||||S)aPrune tensor by removing random channels along the specified dimension. Prunes tensor corresponding to parameter called ``name`` in ``module`` by removing the specified ``amount`` of (currently unpruned) channels along the specified ``dim`` selected at random. Modifies module in place (and also return the modified module) by: 1) adding a named buffer called ``name+'_mask'`` corresponding to the binary mask applied to the parameter ``name`` by the pruning method. 2) replacing the parameter ``name`` by its pruned version, while the original (unpruned) parameter is stored in a new parameter named ``name+'_orig'``. Args: module (nn.Module): module containing the tensor to prune name (str): parameter name within ``module`` on which pruning will act. amount (int or float): quantity of parameters to prune. If ``float``, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If ``int``, it represents the absolute number of parameters to prune. dim (int): index of the dim along which we define channels to prune. Returns: module (nn.Module): modified (i.e. pruned) version of the input module Examples: >>> # xdoctest: +SKIP >>> m = prune.random_structured(nn.Linear(5, 3), "weight", amount=3, dim=1) >>> columns_pruned = int(sum(torch.sum(m.weight, dim=0) == 0)) >>> print(columns_pruned) 3 )rrH)r r.rrss rrandom_structuredrsF645 Mrc<tj|||||||S)aPrune tensor by removing channels with the lowest L\ ``n``-norm along the specified dimension. Prunes tensor corresponding to parameter called ``name`` in ``module`` by removing the specified ``amount`` of (currently unpruned) channels along the specified ``dim`` with the lowest L\ ``n``-norm. Modifies module in place (and also return the modified module) by: 1) adding a named buffer called ``name+'_mask'`` corresponding to the binary mask applied to the parameter ``name`` by the pruning method. 2) replacing the parameter ``name`` by its pruned version, while the original (unpruned) parameter is stored in a new parameter named ``name+'_orig'``. Args: module (nn.Module): module containing the tensor to prune name (str): parameter name within ``module`` on which pruning will act. amount (int or float): quantity of parameters to prune. If ``float``, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If ``int``, it represents the absolute number of parameters to prune. n (int, float, inf, -inf, 'fro', 'nuc'): See documentation of valid entries for argument ``p`` in :func:`torch.norm`. dim (int): index of the dim along which we define channels to prune. importance_scores (torch.Tensor): tensor of importance scores (of same shape as module parameter) used to compute mask for pruning. The values in this tensor indicate the importance of the corresponding elements in the parameter being pruned. If unspecified or None, the module parameter will be used in its place. Returns: module (nn.Module): modified (i.e. pruned) version of the input module Examples: >>> from torch.nn.utils import prune >>> m = prune.ln_structured( ... nn.Conv2d(5, 3, 2), "weight", amount=0.3, dim=1, n=float("-inf") ... ) r")rrH)r r.rrrsr#s r ln_structuredrs-Rfa8I Mrc t|ts td||ni}t|ts tdtj j j|Dcgc]#\}}|j||ft||%c}}}tj j j|Dcgc]2\}}t||dzt jt||4c}}}t}d|_ |d i|} d| _ | jdk7rtd|d| j|j| |j||} d } |D]K\}}t||} | j!} | | | | zj#| }t%||| | | z } Mycc}}wcc}}w) a Globally prunes tensors corresponding to all parameters in ``parameters`` by applying the specified ``pruning_method``. Modifies modules in place by: 1) adding a named buffer called ``name+'_mask'`` corresponding to the binary mask applied to the parameter ``name`` by the pruning method. 2) replacing the parameter ``name`` by its pruned version, while the original (unpruned) parameter is stored in a new parameter named ``name+'_orig'``. Args: parameters (Iterable of (module, name) tuples): parameters of the model to prune in a global fashion, i.e. by aggregating all weights prior to deciding which ones to prune. module must be of type :class:`nn.Module`, and name must be a string. pruning_method (function): a valid pruning function from this module, or a custom one implemented by the user that satisfies the implementation guidelines and has ``PRUNING_TYPE='unstructured'``. importance_scores (dict): a dictionary mapping (module, name) tuples to the corresponding parameter's importance scores tensor. The tensor should be the same shape as the parameter, and is used for computing mask for pruning. If unspecified or None, the parameter will be used in place of its importance scores. kwargs: other keyword arguments such as: amount (int or float): quantity of parameters to prune across the specified parameters. If ``float``, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If ``int``, it represents the absolute number of parameters to prune. Raises: TypeError: if ``PRUNING_TYPE != 'unstructured'`` Note: Since global structured pruning doesn't make much sense unless the norm is normalized by the size of the parameter, we now limit the scope of global pruning to unstructured methods. Examples: >>> from torch.nn.utils import prune >>> from collections import OrderedDict >>> net = nn.Sequential( ... OrderedDict( ... [ ... ("first", nn.Linear(10, 4)), ... ("second", nn.Linear(4, 1)), ... ] ... ) ... ) >>> parameters_to_prune = ( ... (net.first, "weight"), ... (net.second, "weight"), ... ) >>> prune.global_unstructured( ... parameters_to_prune, ... pruning_method=prune.L1Unstructured, ... amount=10, ... ) >>> print(sum(torch.nn.utils.parameters_to_vector(net.buffers()) == 0)) tensor(10) z4global_unstructured(): parameters is not an IterableNz=global_unstructured(): importance_scores must be of type dictrtemprqzROnly "unstructured" PRUNING_TYPE supported for the `pruning_method`. Found method z of type rrr)r)rradictr?nnutilsparameters_to_vectorgetrr@r+rrur,rnumelview_ascustom_from_mask) parameterspruning_methodr#r0r r.relevant_importance_scoresrr7r6 final_maskpointerparam num_param param_masks rglobal_unstructuredrsD j( +NOO->-J)PR ' .WXX"'!D!D#-   ! !64.'&$2G H "88>>66#-  FD7NEOOGFDFFuM  J7 9 E  s %(F< =7G c4tj||||S)aEPrune tensor corresponding to parameter called ``name`` in ``module`` by applying the pre-computed mask in ``mask``. Modifies module in place (and also return the modified module) by: 1) adding a named buffer called ``name+'_mask'`` corresponding to the binary mask applied to the parameter ``name`` by the pruning method. 2) replacing the parameter ``name`` by its pruned version, while the original (unpruned) parameter is stored in a new parameter named ``name+'_orig'``. Args: module (nn.Module): module containing the tensor to prune name (str): parameter name within ``module`` on which pruning will act. mask (Tensor): binary mask to be applied to the parameter. Returns: module (nn.Module): modified (i.e. pruned) version of the input module Examples: >>> from torch.nn.utils import prune >>> m = prune.custom_from_mask( ... nn.Linear(5, 3), name="bias", mask=torch.tensor([0, 1, 0]) ... ) >>> print(m.bias_mask) tensor([0., 1., 0.]) )rrH)r r.rs rrr|s:t, Mrc|jjD]G\}}t|ts|j|k(s'|j ||j|=|cSt d|d|d)aRemove the pruning reparameterization from a module and the pruning method from the forward hook. The pruned parameter named ``name`` remains permanently pruned, and the parameter named ``name+'_orig'`` is removed from the parameter list. Similarly, the buffer named ``name+'_mask'`` is removed from the buffers. Note: Pruning itself is NOT undone or reversed! Args: module (nn.Module): module containing the tensor to prune name (str): parameter name within ``module`` on which pruning will act. Examples: >>> m = random_unstructured(nn.Linear(5, 7), name="weight", amount=0.2) >>> m = remove(m, name="weight") z Parameter 'z ' of module z/ has to be pruned before pruning can be removed)r'r(r)rrrQrc)r r.r4r5s rrQrQs&,,2244 d- .43D3D3L KK ))!,M   dV<x/^_ rc|jD]7\}}|jjD]}t|tsy9y)atCheck if a module is pruned by looking for pruning pre-hooks. Check whether ``module`` is pruned by looking for ``forward_pre_hooks`` in its modules that inherit from the :class:`BasePruningMethod`. Args: module (nn.Module): object that is either pruned or unpruned Returns: binary answer to whether ``module`` is pruned. Examples: >>> from torch.nn.utils import prune >>> m = nn.Linear(5, 7) >>> print(prune.is_pruned(m)) False >>> prune.random_unstructured(m, name="weight", amount=0.2) >>> print(prune.is_pruned(m)) True TF) named_modulesr'rr)r)r _ submoduler5s r is_prunedrsQ,,,. 900779 D$ 12  rc t|tjstd|dt|tjr|dks6t|tjs,t |dkDst |dkrt d|dyy) aValidate helper to check the range of amount at init. Args: amount (int or float): quantity of parameters to prune. If float, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If int, it represents the absolute number of parameters to prune. Raises: ValueError: if amount is a float not in [0, 1], or if it's a negative integer. TypeError: if amount is neither a float nor an integer. Note: This does not take into account the number of parameters in the tensor to be pruned, which is known only at prune. zInvalid type for amount: z. Must be int or float.rg?gamount=zG should either be a float in the range [0, 1] or a non-negative integerN)r)numbersRealraIntegralfloatrcrs rrrs$ fgll +3F8;RSTT67++,! vw// 0 6]S E&MC$7fXd e  %8 1rcft|tjr||kDrtd|d|yy)a?Validate that the pruning amount is meaningful wrt to the size of the data. Validation helper to check that the amount of parameters to prune is meaningful wrt to the size of the data (`tensor_size`). Args: amount (int or float): quantity of parameters to prune. If float, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If int, it represents the absolute number of parameters to prune. tensor_size (int): absolute number of parameters in the tensor to prune. rz: should be smaller than the number of parameters to prune=N)r)rrrcrrs rrrsD$&'**+0DfXWXcWd e  1E+rcn|j}t|dkrtd|dt|dy)aiValidate that the tensor to be pruned is at least 2-Dimensional. Validation helper to check that the tensor to be pruned is multi- dimensional, such that the concept of "channels" is well-defined. Args: t (torch.Tensor): tensor representing the parameter to prune Raises: ValueError: if the tensor `t` is not at least 2D. r&zZStructured pruning can only be applied to multidimensional tensors. Found tensor of shape z with z dimsN)r<r\rc)rr<s rrrsE GGE 5zQ ?gVCJ#.B177)LMMrctt|j}|dkr||}|j|t j |||}|S)aZCompute the L_n-norm of a tensor along all dimensions except for the specified dimension. The L_n-norm will be computed across all entries in tensor `t` along all dimension except for the one identified by dim. Example: if `t` is of shape, say, 3x2x4 and dim=2 (the last dim), then norm will have Size [4], and each entry will represent the `L_n`-norm computed using the 3x2=6 entries for each of the 4 channels. Args: t (torch.Tensor): tensor representing the parameter to prune n (int, float, inf, -inf, 'fro', 'nuc'): See documentation of valid entries for argument p in torch.norm dim (int): dim identifying the channels to prune Returns: norm (torch.Tensor): L_n norm computed across all dimensions except for `dim`. By construction, `norm.shape = t.shape[-1]`. r)prs)r{ryrsrQr?r)rrrsdimsrs rrrHsK( aeeg D Qw3iKK ::a1$ 'D Krre)rUrabcrrcollections.abcrr?rr+rrrrrrrrrrrrrrQrrrrrrrrrrrs#$ x1x1vQ(Qh+ +28:*8:v? &? DgC(gCT $ D6&66 F"J)X$N,^xvB<: < 0 *+4Nr