MCj2jdZddlZddlZddlZddlmZddlZddlZ ejejdeje Zddddd dd Zd ed e j fd Zdde j ded e j fdZde j d e j fdZddZe dk(reyy)aJ Crop yield data preprocessing. Cleans raw crop production data (ICRISAT VDSA or APY government format) into a standardized long-format table: one row per district-year-crop. Usage: python -m ml_pipeline.preprocessing.clean_crop_data --input data/raw/icrisat/yield.csv --output data/processed/crop_clean.csv N)Pathz%%(asctime)s %(levelname)s %(message)s)levelformat  )kharifrabisummerz whole yearautumnwinter input_pathreturncJ|jstd|tj|jDcgc]0}|j j jdd2c}_djvrLdjvr!tdtjtjddd<djvr"d jvrjdd i hd }|tjz }|r$td |d tjgd}tfd|Dd}|%td|dtjd|dk7r9j|di |dk(r tjdddz d<djvrFddg}tfd|Dd}|rj|di StjdScc}w)NzInput file not found:  _districtstateuWInput has neither a 'district' nor a 'state' column — cannot proceed. Found columns: uNo district column found in input — using 'state' as a stopgap district value. Model granularity will be state-level, not district-level, until real district data is sourced. crop_yearyearcolumns>croprrz"Input is missing required columns z . Found: ) yield_ton_ha yield_kg_hayield productivityc3@K|]}|jvs|ywNr.0cdfs VC:\Crop_Prediction\Backend\crop-ai-system\ml_pipeline\preprocessing\clean_crop_data.py z$load_and_validate..DsH!RZZAHz/No recognizable yield column found. Looked for z, found columns: z?. Rename the source column or add the new alias to this script.rrz9Converting yield_kg_ha to yield_ton_ha (dividing by 1000)iarea_haarea area_hectaresc3@K|]}|jvs|ywr rr!s r%r&z$load_and_validate..SsJQ!rzz/qJr'u;No area column found — area_ha will be absent from output)existsFileNotFoundErrorpdread_csvrstriplowerreplace ValueErrorlistloggerwarningrenamesetnextinfo) rr#requiredmissing yield_aliasesfound_yield_col area_aliasesfound_area_colr$s @r%load_and_validaterAs-    "8 EFF Z B?AzzJ!!'')//#++C5JBJ# "** $++/ +;*<>   C G:bjj V2::%= YY V4Y 5+HRZZ(G=gYiPTUWU_U_P`OabccMMH}H$OO=m_M"2::./00 1  .( YY@Y A m + KKS T!#N!3d!:B~  "0 J,JDQ NI#>?B I NNX Y IsKs5H r$ std_thresholdc|jddjddgj}gd|_|j |dd}|dj |dd k(zt j|d|d z ||dzkz}t|}||jd dg jd }t|}tjd||z d||z |z dd|S)u Drops rows more than std_threshold standard deviations from their crop's mean yield. Implemented via boolean masking rather than groupby().apply() returning filtered sub-frames — recent pandas versions (2.2+) can silently drop the grouping column itself when apply() returns a row-filtered subset of the original frame, which would corrupt 'crop' out of the output entirely. rrmeanstd)r _group_mean _group_stdleft)onhowrGrrFrTdropzOutlier removal: dropped z rows (z.1%)) groupbyagg reset_indexrmergeisnanpabslenrLr5r:)r$rB group_statsmerged keep_maskbeforeresultafters r%remove_outliers_per_cropr\\s**V$^488&%IUUWK?K XXkf&X 9F |!!# , 1 $ & 66&(6-+@@ AMTZ[gThDh h j WF I  # #]L,I # J V V\` V aF KE KK+FUN+;7FUNV\C\]`Baabcd Mc|djtjjjj|d<|djtjjjj|d<d|jvr|djtjjjj|d<|dj t |d<|djj}|dkDrPtj|dtt jntjdd|d<tj|d d |d <t|}||d dkD}t|}tj!d ||z d t#|}|j%d dgj'd}|S)NrrseasonmonthrzO rows have an unrecognized season value and got month=NaN. Recognized seasons: uMNo 'season' column present — month will be absent, defaulting to 6 (kharif)rrcoerce)errorszDropped z/ rows with zero, negative, or non-numeric yield)subsetTrK)astypestrr0r1rmapSEASON_TO_MONTHisnullsumr5r6r4keysr. to_numericrUr:r\dropnarP)r$unmappedrYr[s r%cleanrnxs ^**3/3399;??EEGBzNF""3'++11377==?BvJ2::(|**3/3399;??EEG8 l&&77 g;%%'++- a< NN*''+O,@,@,B'C&DF  fg7 r.'9(KB~ WF B~  " #B GE KK(6E>**YZ[ !" %B >73 4 @ @d @ KB Ir]ctjd}|jdd|jdd|j}t |j }t |j }tjd|t|}tjdt|}t|d k(r*tjd tjd |jj!dd |j#|d tjdt|d|tjdt%|dj'y)NzClean raw crop yield data) descriptionz--inputT)r;z--outputzLoading raw crop data from z8Cleaning: type coercion, season mapping, outlier removalrzKZero rows remain after cleaning. Check the source file and column mappings.)parentsexist_okF)indexzSaved z cleaned rows to zCrops present: r)argparseArgumentParser add_argument parse_argsrinputoutputr5r:rArnrUerrorsysexitparentmkdirto_csvsortedunique)parserargsr output_pathraw_dfclean_dfs r%mainrs)  $ $1L MF  D1  T2    Ddjj!Jt{{#K KK-j\:; z *F KKJKV}H 8} bc  TD9 OOKuO- KK&X'8 FG KK/&&)9)@)@)B"C!DEFr]__main__)g@)rN)__doc__ruloggingr|pathlibrnumpyrSpandasr. basicConfigINFO getLogger__name__r5rg DataFramerAfloatr\rnrr]r%rs  ',,/VW   8 $aqTUacd>$>2<<>Bebll8bllr||:G2 zFr]