lazyprep 0.0.19
An efficient data preprocessing tool for data science
Requires Python: >=3.6

# `lazyprep`

**`lazyprep`** is a modular and extensible Python package for preprocessing structured datasets, purpose-built for efficiency and ease of use. It streamlines critical steps like file loading, missing value imputation, outlier handling, and data cleaning—making it ideal for rapid prototyping, exploratory data analysis, and machine learning pipelines.

---

## 🔧 Key Features

### ✅ Modular Architecture
Each component is decoupled and independently reusable, allowing you to plug and play as needed.

---

## 📦 Installation

Install lazyprep from PyPI:

```bash
pip install lazyprep
```

---

## 🚀 Quick Start: End-to-End Preprocessing with `PreprocessingPipeline`

The `PreprocessingPipeline` is the primary way to use `lazyprep` for end-to-end data preprocessing. It orchestrates all the individual components (Loader, Analyzer, Cleaner, Imputer, Outlier, DateProcessor, TextProcessor) into a seamless workflow.

Here's how to quickly get started:

```python
from lazyprep.pipeline import PreprocessingPipeline

config = {
    'cleaner': {'column_threshold': 0.8, 'row_threshold': 0.8, 'drop_null': True},
    'imputer': {'numeric_strategy': 'mean', 'categorical_strategy': 'mode', 'custom_strategies': {'age': 'median'}},
    'outlier': {'default_strategy': 'iqr', 'action': 'drop'},
    'date_processor': {'extract_features': ['year', 'month', 'dayofweek']},
    'text_processor': {'text_columns': ['description', 'comments'], 'lemmatize': True, 'remove_stopwords': True}
}
pipeline = PreprocessingPipeline(
    file_path="path/to/your/dataset.csv", # Replace with your actual data file path
    target_column="target_variable",      # Replace with your target column name, if applicable
    config=config                         # Pass your defined configuration
)

processed_df = pipeline.run()

print("Processed DataFrame Head:")
print(processed_df.head())

metadata = pipeline.get_metadata()
print("\nCollected Metadata:")
print(metadata)

print("\nPipeline Steps:")
print(pipeline.get_pipeline_steps())
```

---

## 🧪 Preprocessing Pipeline (Detailed)

The `PreprocessingPipeline` class provides an end-to-end solution for data preprocessing, allowing you to chain multiple components together.

**Key Features:**

*   **Component Composition**: Combines `Loader`, `Analyzer`, `Imputer`, `Outlier`, `Cleaner`, `DateProcessor`, and `TextProcessor` components.
*   **Configuration**: Uses a configuration dictionary to specify which components to include and their parameters.
*   **Metadata Management**: Collects and stores metadata throughout the pipeline, which can be used for further analysis or model training.
*   **Custom Steps**: Allows adding custom preprocessing steps.

**Configuration:**

The `config` parameter in the `PreprocessingPipeline` constructor allows you to specify the components to include and their parameters. The configuration is a dictionary where keys are component names (e.g., 'cleaner', 'imputer', 'outlier') and values are dictionaries of parameters for those components.

---

## 🧩 Core Components

This library is built around several core components, each designed to handle a specific aspect of data preprocessing. When using the `PreprocessingPipeline`, you configure these components via the `config` dictionary passed during pipeline initialization.

### 📂 Loader

The `Loader` component is automatically included as the first step in the `PreprocessingPipeline`. It handles file loading and initial column normalization.

**Configurable Parameters (within `pipeline.config['loader']`):**
*   `normalize_columns` (bool, default: `True`): Whether to normalize column names (lowercase, replace spaces/special chars with underscores).
*   `encoding` (str, optional): Specify file encoding for CSV files. If not provided, `chardet` will attempt to detect it.

**Example Configuration:**
```python
config = {
    'loader': {'normalize_columns': True, 'encoding': 'utf-8'}
}
```


###  Imputer

The `Imputer` component handles missing values using various strategies.

**Configurable Parameters (within `pipeline.config['imputer']`):**
*   `numeric_strategy` (str, default: `'mean'`): Default strategy for numeric columns. Options include: `'mean'`, `'median'`, `'winsorized_mean'`, `'iterative_rf'`, `'knn'`, `'bayesian'`, `'regression'`, `'kmeans'`.
*   `categorical_strategy` (str, default: `'mode'`): Default strategy for categorical columns. Options include: `'mode'`, `'constant'`.
*   `custom_strategies` (Dict[str, str], optional): A dictionary to override default strategies for specific columns. E.g., `{'age': 'median', 'city': 'constant'}`.

**Example Configuration:**
```python
config = {
    'imputer': {
        'numeric_strategy': 'median',
        'categorical_strategy': 'constant',
        'custom_strategies': {'temperature': 'mean', 'city': 'Missing'}
    }
}
```

---

### ⚠️ Outlier Handler

The `Outlier` component detects and handles outliers in numeric columns.

**Configurable Parameters (within `pipeline.config['outlier']`):**
*   `default_strategy` (str, default: `'iqr'`): Default outlier detection method. Options include: `'iqr'`, `'z_score'`, `'modified_z_score'`, `'percentile'`, `'mad'`, `'log_iqr'`.
*   `custom_strategies` (Dict[str, str], optional): A dictionary to override default strategies for specific columns. E.g., `{'income': 'z_score'}`.
*   `action` (str, default: `'drop'`): Action to take on outliers. Options: `'drop'` (removes rows with outliers), `'cap'` (caps outliers at bounds - *Note: capping logic needs to be implemented*).

**Example Configuration:**
```python
config = {
    'outlier': {
        'default_strategy': 'z_score',
        'custom_strategies': {'price': 'mad'},
        'action': 'drop'
    }
}
```

---

### 🧹 Cleaner

The `Cleaner` component performs general data cleaning tasks.

**Configurable Parameters (within `pipeline.config['cleaner']`):**
*   `column_threshold` (float, default: `0.7`): Threshold (0-1) for dropping columns with a high percentage of null values.
*   `row_threshold` (float, default: `0.7`): Threshold (0-1) for dropping rows with a high percentage of null values.
*   `drop_null` (bool, default: `False`): If `True`, drops any remaining rows with *any* null values after other cleaning steps.
*   `target_column` (str, optional): The name of the target column. If specified, rows with null values in this column will be dropped.

**Example Configuration:**
```python
config = {
    'cleaner': {
        'column_threshold': 0.8,
        'row_threshold': 0.8,
        'drop_null': True,
        'target_column': 'my_target_variable'
    }
}
```

---

### 📅 DateProcessor

The `DateProcessor` component extracts features from datetime columns.

**Configurable Parameters (within `pipeline.config['date_processor']`):**
*   `extract_features` (List[str], optional): A list of specific datetime features to extract (e.g., `['year', 'month', 'day', 'hour', 'minute', 'second', 'dayofweek', 'quarter']`). If `None`, a default set of features will be extracted based on the column type (date, time, or datetime).
*   `drop_original` (bool, default: `True`): If `True`, the original datetime columns will be dropped after feature extraction.

**Example Configuration:**
```python
config = {
    'date_processor': {
        'extract_features': ['year', 'month', 'dayofweek'],
        'drop_original': True
    }
}
```

---

### 📝 TextProcessor

The `TextProcessor` component handles text cleaning, tokenization, and lemmatization.

**Configurable Parameters (within `pipeline.config['text_processor']`):**
*   `text_columns` (List[str], optional): A list of column names to be processed as text. If `None`, text columns will be inferred from metadata (e.g., high cardinality categorical/object columns).
*   `lemmatize` (bool, default: `True`): Whether to apply lemmatization to text tokens.
*   `remove_stopwords` (bool, default: `True`): Whether to remove common English stopwords.
*   `output_format` (str, default: `'tokens'`): The format for the processed text. Options: `'tokens'` (list of words), `'string'` (space-separated string).

**Example Configuration:**
```python
config = {
    'text_processor': {
        'text_columns': ['product_description', 'review_text'],
        'lemmatize': True,
        'remove_stopwords': False,
        'output_format': 'string'
    }
}
```

---

---

## 🧩 Components

- `lazyprep.loader` – File loading utilities
- `lazyprep.analyzer` – Data type and distribution analysis
- `lazyprep.imputer` – Missing value imputation
- `lazyprep.outlier` – Outlier detection and removal
- `lazyprep.cleaner` – Data cleaning utilities
- `lazyprep.pipeline` – End-to-end preprocessing pipeline
- `lazyprep.date_processor` – Date and Time feature extraction
- `lazyprep.text_processor` – Text processing

---

## 🤝 Contributing

Contributions to `lazyprep` are welcome! To contribute:

1.  Fork the repository.
2.  Create a new branch for your feature or bug fix.
3.  Make your changes and write tests.
4.  Submit a pull request.

---

## 📄 License

This project is licensed under the MIT License.

---

##  Author

- NaveenkumarD (<naveenkumar19082004@gmail.com>)

---

For more details, see the documentation or explore the source code in the `src/LazyPrep/` directory.