{KgJdZddlZddlZddlmZddlmZddlmZddl Z ddl Z ddl mZddlmZdd lmZmZdd lmZdd lmZd ed edefdZd ed ede j2fdZdZ ddZ ddZ ddZy)z9Implementation of ARFF parsers: via LIAC-ARFF and pandas.N) OrderedDict) Generator)List)_arff)ArffSparseDataType)chunk_generatorget_chunk_n_rows)check_pandas_support) pd_fillna arff_datainclude_columnsreturncNtttf}t|Dcic]\}}|| }}}t|d|d|dD]J\}}}||vs |dj||dj||dj||L|Scc}}w)aObtains several columns from sparse ARFF representation. Additionally, the column indices are re-labelled, given the columns that are not included. (e.g., when including [1, 2, 3], the columns will be relabelled to [0, 1, 2]). Parameters ---------- arff_data : tuple A tuple of three lists of equal size; first list indicating the value, second the x coordinate and the third the y coordinate. include_columns : list A list of columns to include. Returns ------- arff_data_new : tuple Subset of arff data with only the include columns indicated by the include_columns argument. rr)list enumeratezipappend) r r arff_data_new array_idx column_idxreindexed_columnsvalrow_idxcol_idxs a/home/alanp/www/video.onchill/myenv/lib/python3.12/site-packages/sklearn/datasets/_arff_parser.py_split_sparse_columnsrs.*.(@M;D_;U;U"7)Z I;U"%Yq\9Q<1!NWg o % !  # #C ( !  # #G , !  # #$5g$> ? "O s B!c0t|ddz}|t|f}t|Dcic]\}}|| }}}tj|tj }t |d|d|dD]\}} } | |vs ||| || f<|Scc}}w)Nrdtyperr)maxlenrnpemptyfloat64r) r rnum_obsy_shaperrryrrrs r_sparse_data_to_arrayr*6s )A,!#GO,-G;D_;U;U"7)Z I;U  +A!$Yq\9Q<1!NWg o %58Ag(11 2"O Hs Bcz||}t|dk\r ||}||fSt|dk(r ||d}||fSd}||fS)a Post process a dataframe to select the desired columns in `X` and `y`. Parameters ---------- frame : dataframe The dataframe to split into `X` and `y`. feature_names : list of str The list of feature names to populate `X`. target_names : list of str The list of target names to populate `y`. Returns ------- X : dataframe The dataframe containing the features. y : {series, dataframe} or None The series or dataframe containing the target. rrrN)r#)frame feature_names target_namesXr)s r_post_process_framer0Hsh, mA <A ,  a4K \ a  ,q/ " a4K  a4Kc d}||}|dk(rtjntj}|dk( } tj||| } ||z} | dD cic]\} } t | t r| | vr| | }} } |dk(rt d}t| d}t |j}t| d}|j|g|d }|jd j}t|}|Dcgc] }|| vs| }}||g}t| d|D](}|j|j||d |*t!|d k\r$|d j#|dj$|d <|j'|d }t)||}~~i}|j*D]N} || d}|j-dk(rd|| <$|j-dk(rd|| <=|j$| || <P|j#|}t/|||\}}nn| d}|D cgc]} t1|| d}!} |D cgc]} t1|| d}"} t |t2rz| t5d|d dk(rd}#n |d |dz}#t7j8t:j<j?|d|#}|j@|}|dd|!f}|dd|"f}nt |tBrtE||!}$tG|ddz}%|%t!|!f}&tHjJjM|$d |$d|$d ff|&t6jN}|jQ}tS||"}nt5dtU||D chc]} | |v}'} |'sntW|'rt7jXt[|D( cgc]`\}(} t7j\t7j^|ja| d|dd|(|(dzfj#t0dbc} }(}ntc|'r t5d |jdddk(r|jAd!}n|jddd k(rd}|dk(r||dfS||d|fScc} } wcc}wcc} wcc} wcc} wcc} }(w)"aARFF parser using the LIAC-ARFF library coded purely in Python. This parser is quite slow but consumes a generator. Currently it is needed to parse sparse datasets. For dense datasets, it is recommended to instead use the pandas-based parser, although it does not always handles the dtypes exactly the same. Parameters ---------- gzip_file : GzipFile instance The file compressed to be read. output_arrays_type : {"numpy", "sparse", "pandas"} The type of the arrays that will be returned. The possibilities ara: - `"numpy"`: both `X` and `y` will be NumPy arrays; - `"sparse"`: `X` will be sparse matrix and `y` will be a NumPy array; - `"pandas"`: `X` will be a pandas DataFrame and `y` will be either a pandas Series or DataFrame. columns_info : dict The information provided by OpenML regarding the columns of the ARFF file. feature_names_to_select : list of str A list of the feature names to be selected. target_names_to_select : list of str A list of the target names to be selected. Returns ------- X : {ndarray, sparse matrix, dataframe} The data matrix. y : {ndarray, dataframe, series} The target. frame : dataframe or None A dataframe containing both `X` and `y`. `None` if `output_array_type != "pandas"`. categories : list of str or None The names of the features that are categorical. `None` if `output_array_type == "pandas"`. c3@K|D]}|jdyw)Nutf-8)decode) gzip_filelines r_io_to_generatorz+_liac_arff_parser.._io_to_generatorsD++g& &ssparsepandas) return_typeencode_nominal attributeszfetch_openml with as_frame=TruedataF)columnscopyT)deeprrr) ignore_index data_typeintegerInt64nominalcategoryindexNz6shape must be provided when arr['data'] is a Generatorr&)r!count)shaper!z-Unexpected type for data obtained from arff: Or )r@zAMix of nominal and non-nominal targets is not currently supported)rI)3rCOO DENSE_GENload isinstancerr rkeysnext DataFrame memory_usagesumr r rr#astypedtypesconcatr r?lowerr0intr ValueErrorr$fromiter itertoolschain from_iterablereshapetuplerr"spr9 coo_matrixr&tocsrr*typeallhstackrtakeasarraypopanyrK))r6output_arrays_typeopenml_columns_infofeature_names_to_selecttarget_names_to_selectrKr8streamr;r<arff_containercolumns_to_selectnamecat categoriespd columns_info columns_names first_rowfirst_df row_bytes chunksizecolcolumns_to_keepdfsr>r,rW column_dtyper/r)r col_namefeature_indices_to_selecttarget_indices_to_selectrJ arff_data_Xr'X_shapeis_classificationis) r_liac_arff_parserrhspn'i (F 2X=%))5??K-89NZZKN02HH( 55ID# c4 T->%> c 5 X% !"C D">,#?@ \..01 /0 << ]<O))t)488: $Y/ +8T-33BS;S3-T()#N6$:IFD JJ T=u EoV G s8q=V]]3q6==1CF  #D 1"e$ MMD.t4[AL!!#y0 't ##%2)t $||D1t " V$" *,B 1#6* 4% 3 #H-g6 73 "% 3$ 2 #H-g6 72 !$ i +} LQx2~a58+;;--i8D  4<<'DQ112AQ001A  5 )/ ;TUK)A,'!+G$= >?G $$Q+a.+a.!ABjj%A  A%i1IJA?Y?PQ  4J 3IxH "3I  !  " # (11G'H  (I 8 GG :>>(#;3G!QQY,..s.?(I A" #S  771:? % A WWQZ1_AX%!UD  az !!E&UL% $ N s+!R. R4R48R9R>) S!A%S c ddl}|D]2}|jdjjds2ni}|D]<} || d} | jdk(rd|| <$| jdk(s8d || <>t |D cic]\} } | |vr| || } } } dd d gd d d dd| d } i| |xsi}|j |fi|} |D cgc]} | c} |_||z}|j Dcgc] }||vs| }}||}tjdfd}|jjD cgc]\} }t||jr| }} }|D]#}||j j#|||<%t%|||\}}|dk(r|||dfS|j'|j'}}|jjD cic]6\} }t||jr| |j(j+8}} }||d|fScc} } wcc} w#t$r!}|jjd|d}~wwxYwcc}wcc}} wcc}} w)a^ARFF parser using `pandas.read_csv`. This parser uses the metadata fetched directly from OpenML and skips the metadata headers of ARFF file itself. The data is loaded as a CSV file. Parameters ---------- gzip_file : GzipFile instance The GZip compressed file with the ARFF formatted payload. output_arrays_type : {"numpy", "sparse", "pandas"} The type of the arrays that will be returned. The possibilities are: - `"numpy"`: both `X` and `y` will be NumPy arrays; - `"sparse"`: `X` will be sparse matrix and `y` will be a NumPy array; - `"pandas"`: `X` will be a pandas DataFrame and `y` will be either a pandas Series or DataFrame. openml_columns_info : dict The information provided by OpenML regarding the columns of the ARFF file. feature_names_to_select : list of str A list of the feature names to be selected to build `X`. target_names_to_select : list of str A list of the target names to be selected to build `y`. read_csv_kwargs : dict, default=None Keyword arguments to pass to `pandas.read_csv`. It allows to overwrite the default options. Returns ------- X : {ndarray, sparse matrix, dataframe} The data matrix. y : {ndarray, dataframe, series} The target. frame : dataframe or None A dataframe containing both `X` and `y`. `None` if `output_array_type != "pandas"`. categories : list of str or None The names of the features that are categorical. `None` if `output_array_type == "pandas"`. rNr4z@datarCrDrErFrGF?%"T\) header index_col na_valueskeep_default_nacomment quotecharskipinitialspace escapecharr!zwThe number of columns provided by OpenML does not match the number of columns inferred by pandas when reading the file.z^'(?P.*)'$cZtj|}||S|jdS)Ncontents)researchgroup) input_stringmatchsingle_quote_patterns rstrip_single_quotesz0_pandas_arff_parser..strip_single_quotess. . = = {{:&&r1r:)r:r5rY startswithrread_csvr?r[errors ParserErrorrcompilerWitemsrPCategoricalDtypertrename_categoriesr0to_numpyrutolist)r6rlrmrnroread_csv_kwargsrvr7rWrsrrdtypes_positionaldefault_read_csv_kwargsr,excrrr}r~rr!categorical_columnsr/r)rurs @r_pandas_arff_parserr4sp ;;w  % % ' 2 27 ; F#*40=    9 ,#F4L    !Y .%F4L$'':;;MGT 6> ;U  " M0L_5JLO BKK 5_ 5E  +>>*=$*=> 02HH&+mmPmss>O7OsmOP / "E::&=>'!<<--//KD% eR00 1 / #3Z^^556IJc # u&=?U VDAqX%!UD  zz|QZZ\1!<<--//KD% eR00 1 e%%''/ az !!W0? ii## @  Q.sHH-H8 H3H88 I%I%"I*);I03H88 I"II"ct|dk(rt||||||S|dk(rt||||||Std|d)a6Load a compressed ARFF file using a given parser. Parameters ---------- gzip_file : GzipFile instance The file compressed to be read. parser : {"pandas", "liac-arff"} The parser used to parse the ARFF file. "pandas" is recommended but only supports loading dense datasets. output_type : {"numpy", "sparse", "pandas"} The type of the arrays that will be returned. The possibilities ara: - `"numpy"`: both `X` and `y` will be NumPy arrays; - `"sparse"`: `X` will be sparse matrix and `y` will be a NumPy array; - `"pandas"`: `X` will be a pandas DataFrame and `y` will be either a pandas Series or DataFrame. openml_columns_info : dict The information provided by OpenML regarding the columns of the ARFF file. feature_names_to_select : list of str A list of the feature names to be selected. target_names_to_select : list of str A list of the target names to be selected. read_csv_kwargs : dict, default=None Keyword arguments to pass to `pandas.read_csv`. It allows to overwrite the default options. Returns ------- X : {ndarray, sparse matrix, dataframe} The data matrix. y : {ndarray, dataframe, series} The target. frame : dataframe or None A dataframe containing both `X` and `y`. `None` if `output_array_type != "pandas"`. categories : list of str or None The names of the features that are categorical. `None` if `output_array_type == "pandas"`. z liac-arffr:zUnknown parser: 'z%'. Should be 'liac-arff' or 'pandas'.)rrr[)r6parser output_typermrnrorKrs rload_arff_from_gzip_filersqv    # "     8 "    # "    x'L M  r1)N)NN)__doc__r]r collectionsrcollections.abcrtypingrnumpyr$scipyrb externalsrexternals._arffrutils._chunkingr r utils._optional_dependenciesr utils.fixesr rndarrayr*r0rrrr1rrs? #%0??# ! 48  F ! 48 ZZ $L  I"d U"~ P r1