Dj/YddlmZddlmZmZmZddlmZddlm Z ddl Z ddZ ddZ ddZd Zd Z dd Zdd Zifd ZGddZGddeZy))RegularizedResults)_calc_nodewise_row_calc_nodewise_weight_calc_approx_inv_cov)LikelihoodModelResults)OLSNcT| td|jdi|jS)aestimates the regularized fitted parameters. Parameters ---------- mod : statsmodels model class instance The model for the current partition. pnum : scalar Index of current partition partitions : scalar Total number of partitions fit_kwds : dict-like or None Keyword arguments to be given to fit_regularized Returns ------- An array of the parameters for the regularized fit zD_est_regularized_naive currently requires that fit_kwds not be None.) ValueErrorfit_regularizedparamsmodpnum partitionsfit_kwdss kC:\Crop_Prediction\Backend\crop-ai-system\venv\Lib\site-packages\statsmodels/base/distributed_estimation.py_est_regularized_naiverKs<&?@ @ 3   * * 1 11cT| td|jdi|jS)aestimates the unregularized fitted parameters. Parameters ---------- mod : statsmodels model class instance The model for the current partition. pnum : scalar Index of current partition partitions : scalar Total number of partitions fit_kwds : dict-like or None Keyword arguments to be given to fit Returns ------- An array of the parameters for the fit zF_est_unregularized_naive currently requires that fit_kwds not be None.r )r fitr rs r_est_unregularized_naiveres:&?@ @ 377 X  % %%rct|d}t|}tj|}|D]}||z } ||z}d|tj||k<|S)a joins the results from each run of _est__naive and returns the mean estimate of the coefficients Parameters ---------- params_l : list A list of arrays of coefficients. threshold : scalar The threshold at which the coefficients will be cut. r)lennpzerosabs)params_l thresholdpr params_mnr s r _join_naiver"si HQKAXJ IV  I/0IbffY)+, rcj|jtj|fi| }||d|z zz }|S)acalculates the log-likelihood gradient for the debiasing Parameters ---------- mod : statsmodels model class instance The model for the current partition. params : array_like The estimated coefficients for the current partition. alpha : scalar or array_like The penalty weight. If a scalar, the same penalty weight applies to all variables in the model. If a vector, it must have the same length as `params`, and contains a penalty weight for each coefficient. L1_wt : scalar The fraction of the penalty given to the L1 penalty term. Must be between 0 and 1 (inclusive). If 0, the fit is a ridge fit, if 1 it is a lasso fit. score_kwds : dict-like or None Keyword arguments for the score function. Returns ------- An array-like object of the same dimension as params Notes ----- In general: gradient l_k(params) where k corresponds to the index of the partition For OLS: X^T(y - X^T params) )scorerasarray)rr alphaL1_wt score_kwdsgrads r _calc_gradr+s>L CIIbjj( 7J 7 7DEQY D Krctj|jtj|fi|}|dddf|jzS)acalculates the weighted design matrix necessary to generate the approximate inverse covariance matrix Parameters ---------- mod : statsmodels model class instance The model for the current partition. params : array_like The estimated coefficients for the current partition. hess_kwds : dict-like or None Keyword arguments for the hessian function. Returns ------- An array-like object, updated design matrix, same dimension as mod.exog N)rsqrthessian_factorr&exog)rr hess_kwdsrhesss r_calc_wdesign_matr2sE& GG&C&&rzz&'9GYG HE D>CHH $$rc|in|}|in|}| td|d}d|vr|d}nd}|jj\}} tt j d| z|z } |j di|j} t|| ||||z } t|| |} g}g}t|| zt|dz| z| D]?}t| ||}|j|t| |||}|j|A| | ||fS)aestimates the regularized fitted parameters, is the default estimation_method for class DistributedModel. Parameters ---------- mod : statsmodels model class instance The model for the current partition. mnum : scalar Index of current partition. partitions : scalar Total number of partitions. fit_kwds : dict-like or None Keyword arguments to be given to fit_regularized score_kwds : dict-like or None Keyword arguments for the score function. hess_kwds : dict-like or None Keyword arguments for the Hessian function. Returns ------- A tuple of parameters for regularized fit An array-like object of the fitted parameters, params An array-like object for the gradient A list of array like objects for nodewise_row A list of array like objects for nodewise_weight zG_est_regularized_debiased currently requires that fit_kwds not be None.r'r(r$g?r )r r/shapeintrceilr r r+r2rangeminrappendr)rmnumrrr)r0r'r(nobsr p_partr r*wexognodewise_row_lnodewise_weight_lidx nodewise_rownodewise_weights r_est_regularized_debiasedrCsJ:")zJ'YI?@ @!(!hhnnGD! "q&J./ 0F S ,8 , 3 3F c65% r?rapprox_inv_covdebiased_paramss r_join_debiasedrOs  IaLOAYJ IhhqkGN 'QqT 1Q4ad#  1& 'XXn-N!23 I sZG).:KLN."4"4W"==O;D "%6D "  "#%D #2D  -D *D   DN&DN  !3D !.D    "D  ,D rcB|i}|dk(r|j|||}n(|dk(r|j||||}ntd|z|j|fi|j}|j dgdgfi|j }|j||fi|jS)aePerforms the distributed estimation using the corresponding DistributedModel Parameters ---------- data_generator : generator A generator that produces a sequence of tuples where the first element in the tuple corresponds to an endog array and the element corresponds to an exog array. fit_kwds : dict-like or None Keywords needed for the model fitting. parallel_method : str type of distributed estimation to be used, currently "sequential", "joblib" and "dask" are supported. parallel_backend : None or joblib parallel_backend object used to allow support for more complicated backends, ex: dask.distributed init_kwds_generator : generator or None Additional keyword generator that produces model init_kwds that may vary based on data partition. The current usecase is for WLS and GLS Returns ------- join_method result. For the default, _join_debiased, it returns a p length array. sequentialjoblibz.parallel_method: %s is currently not supportedr) fit_sequential fit_joblibr rbrcrTrQrdre) rWdata_generatorrparallel_methodparallel_backendinit_kwds_generatorrJr res_mods rrzDistributedModel.fits<  H l *++NH,?AI ((8(;=I M./0 0"!!)>t~~>#$""A3>t~~>!t!!'6GT5F5FGGrc  g}|8t|D](\}\}}t|||||}|j|*|Stt||} | D],\}\\}}} t|||||| }|j|.|S)a*Sequentially performs the distributed estimation using the corresponding DistributedModel Parameters ---------- data_generator : generator A generator that produces a sequence of tuples where the first element in the tuple corresponds to an endog array and the element corresponds to an exog array. fit_kwds : dict-like Keywords needed for the model fitting. init_kwds_generator : generator or None Additional keyword generator that produces model init_kwds that may vary based on data partition. The current usecase is for WLS and GLS Returns ------- join_method result. For the default, _join_debiased, it returns a p length array. ) enumerater]r9zip) rWrlrrorJrrXr/r\tup_genrYs rrjzDistributedModel.fit_sequentials0  &'0'@ *#mud/dE408:  )  *" N$7!9:G7> *22}tk/dE408+G  )  * rc ddlm}|tj\} }| || fdt |D}|S|+|)|5| fdt |D}ddd|S|,|*t t ||} | fd| D}|S|7|5t t ||} |5| fd| D}ddd|SS#1swYSxYw#1swYSxYw)aPerforms the distributed estimation in parallel using joblib Parameters ---------- data_generator : generator A generator that produces a sequence of tuples where the first element in the tuple corresponds to an endog array and the element corresponds to an exog array. fit_kwds : dict-like Keywords needed for the model fitting. parallel_backend : None or joblib parallel_backend object used to allow support for more complicated backends, ex: dask.distributed init_kwds_generator : generator or None Additional keyword generator that produces model init_kwds that may vary based on data partition. The current usecase is for WLS and GLS Returns ------- join_method result. For the default, _join_debiased, it returns a p length array. r) parallel_funcNc3BK|]\}\}}|||ywrar .0rrXr/frrWs r z.DistributedModel.fit_joblib..cs.: 3muddD%x@:c3BK|]\}\}}|||ywrar rxs rr{z.DistributedModel.fit_joblib..is. >$7D-5$!"$eT8 D >r|c 3JK|]\}\\}}}||||ywrar ryrrXr/rQrzrrWs rr{z.DistributedModel.fit_joblib..os5( @&@}tidD%xK( #c 3JK|]\}\\}}}||||ywrar rs rr{z.DistributedModel.fit_joblib..vs5 ,$DD*D=E4)!"$eT8Y O ,r)statsmodels.tools.parallelrvr]rrrrs) rWrlrrnrorvparn_jobsrJrtrzs ` ` @rrkzDistributedModel.fit_joblibDs:4 =&' >#,^#< >>  >$ %*=*IN4G HIG(&((I ).A.MN4G HIG! , ,#* ,,  , y% >$  , s C=C*C'*C4)NNNNNNNN)NrhNNra)__name__ __module__ __qualname____doc__rfrrjrkr rrr_r_msD=G~0437CG26--^BN7;7Ht,0-`(,6rr_c(eZdZdZfdZdZxZS)DistributedResultsaT Class to contain model results Parameters ---------- model : class instance Class instance for model used for distributed data, this particular instance uses fake data and is really only to allow use of methods like predict. params : ndarray Parameter estimates from the fit model. c&t|||yra)superrf)rWr[r __class__s rrfzDistributedResults.__init__s 'rcX|jj|j|g|i|S)aCalls self.model.predict for the provided exog. See Results.predict. Parameters ---------- exog : array_like NOT optional The values for which we want to predict, unlike standard predict this is NOT optional since the data in self.model is fake. *args : Some models can take additional arguments. See the predict method of the model for the details. **kwargs : Some models can take additional keywords arguments. See the predict method of the model for the details. Returns ------- prediction : ndarray, pandas.Series or pandas.DataFrame See self.model.predict )r[predictr )rWr/argskwargss rrzDistributedResults.predicts+."tzz!!$++tEdEfEEr)rrrrrfr __classcell__)rs@rrr}s (Frrra)r)NNN)statsmodels.base.elastic_netr(statsmodels.stats.regularized_covariancerrrstatsmodels.base.modelr#statsmodels.regression.linear_modelrnumpyrrrr"r+r2rCrOr]r_rr rrrsx;0093@F24&42(V%.?C9=>;B*\')"JMM`(F/(Fr