In general, we will df.drop('column_name', axis=1) to remove a column in a DataFrame.
I want to add this transformer into a Pipeline
Example:
numerical_transformer = Pipeline(steps=[('imputer', SimpleImputer(strategy='mean')),
('scaler', StandardScaler(with_mean=False))
])
How can I do it?
4 Answers
You can write a custom Transformer like this :
class columnDropperTransformer():
def __init__(self,columns):
self.columns=columns
def transform(self,X,y=None):
return X.drop(self.columns,axis=1)
def fit(self, X, y=None):
return self
And use it in a pipeline :
import pandas as pd
# sample dataframe
df = pd.DataFrame({
"col_1":["a","b","c","d"],
"col_2":["e","f","g","h"],
"col_3":[1,2,3,4],
"col_4":[5,6,7,8]
})
# your pipline
pipeline = Pipeline([
("columnDropper", columnDropperTransformer(['col_2','col_3']))
])
# apply the pipeline to dataframe
pipeline.fit_transform(df)
Output :
col_1 col_4
0 a 5
1 b 6
2 c 7
3 d 8
You can encapsulate your Pipeline into a ColumnTransformer which allows you to select the data that is processed through the pipeline as follows:
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.compose import make_column_selector, make_column_transformer
col_to_exclude = 'A'
df = pd.DataFrame({'A' : [ 0]*10, 'B' : [ 1]*10, 'C' : [ 2]*10})
numerical_transformer = make_pipeline
SimpleImputer(strategy='mean'),
StandardScaler(with_mean=False)
)
transform = ColumnTransformer(
(numerical_transformer, make_column_selector(pattern=f'^(?!{col_to_exclude})'))
)
transform.fit_transform(df)
NOTE: I am using here a regex pattern to exclude the column A.
The simplest way is to use the transformer special value of 'drop' in sklearn.compose.ColumnTransformer:
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
# Specify columns to drop
columns_to_drop = ['feature1', 'feature3']
# Create a pipeline with ColumnTransformer to drop columns
preprocessor = ColumnTransformer(
transformers=[
('column_dropper', 'drop', columns_to_drop),
]
)
pipeline = Pipeline(
steps=[
('preprocessing', preprocessor),
]
)
# Transform the DataFrame using the pipeline
transformed_data = pipeline.fit_transform(df)
I think all of the answers here are actually just overcomplicated. This type of procedure is what FunctionTransformer is for:
from sklearn.pipeline import FunctionTransformer, make_pipeline
pipeline = make_pipeline(
FunctionTransformer(lambda df: df.drop(columns_to_drop, axis=1)),
)
As the name implies, you can define an arbitrary function. Since the operation here is as basic as it gets, I don't think this needs a standalone class.
