Odyssey to Python Mastery: 4 Jedi Techniques
Tags: python, programming, tutorial
Published: 2023-10-02 13:16:54


Enfold the odyssey to Python brilliance, a journey illuminated with the allure of elegance and efficiency. This article, inspired by [Saa](https://github.com/Proteusiq/saa), my Python package that translates time into human-friendly spoken expressions, takes on a voyage to explore four advanced Python concepts, unearthing the true prowess of Python language. With every stride, witness your code transform, mirroring the elegance of poetry and the finesse of an art masterpiece.

⚠️ _code heavy: set out in not-too-distant galax, this article explores advance concepts_ 
___

## 1. **Single Dispatcher: The Chameleon Functions**

![chameleon](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1xawg1l6szvjfj5dap6s.jpeg)

Observe the allure of `singledispatch` from `functools`, a decorator that bestows your functions the gift to morph based on the type of the first argument, a beacon for writing seamless generic code.

```python
from __future__ import annotations
from functools import singledispatch
from datetime import time, datetime


TimeType = str | time | datetime


@singledispatch
def clock(_: TimeType) -> time:
    """Clock Parser

    Accepts string, time or datetime and return time object

    Args:
        _ (TimeType): string, time or datetime object

    Raises:
        NotImplementedError: shell for dispatching

    Returns:
        time: python time object
    """
    raise NotImplementedError


@clock.register(str)
def _(t: str) -> time:
    return datetime.strptime(t, "%H:%M").time()


@clock.register(datetime)
def _(t: datetime) -> time:
    return t.time()


@clock.register(time)
def _(t: time) -> time:
    return t
```

Test we must, young Padawan. Reflect upon the use cases, we shall. In the vast galaxy where stars twinkle, mirror the universe’s myriad possibilities, our trials will.

```python
from datetime import datetime, time
import pytest
import clock  


@pytest.fixture(params=["12:34", datetime.now().replace(hour=12, minute=34), time(12, 34)])
def valid_time_input(request):
    yield request.param


@pytest.fixture(params=[1234, 12.34, [12, 34], {"hour": 12, "minute": 34}])
def invalid_time_input(request):
    yield request.param

def test_clock_with_valid_input(valid_time_input):
    "Test with valid inputs"

    result = clock(valid_time_input)
    assert isinstance(result, time)
    assert result.hour == 12
    assert result.minute == 34


def test_clock_with_invalid_input(invalid_time_input):
    "Test with invalid input, expecting NotImplementedError"
    with pytest.raises(NotImplementedError):
        clock(invalid_time_input)

```

### Reflect and Contemplate:
Pave this path cautiously. Ensure the base function is etched to raise a `NotImplementedError` and adorn each additional implementation with `@function_name.register(type)`.
___
## 2. **Setters and Getters: The Shield and Sword**

![setter](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/oo7wrsvwj2f08lt2c7ak.jpeg)

Conjure the `@property` decorator, your shield and sword, ensuring your objects stand resilient, cloaked in encapsulation.

Beginning with elemental steps, let's unveil the potential to not only encapsulate but also validate inputs at the setting.


```python
from __future__ import annotations

π = 22/7 #3.14...

def radius_validator(value: int|float) -> int | float:
    
    if not isinstance(value, (int, float)):
        raise TypeError(f"Radius has to be a positive int or float 😞")
    elif value < 0:
        raise ValueError(f"Radius cannot be negative 😞")       
    return value
            

class Circle:
    def __init__(self, radius: int|float):
        self._radius = radius_validator(radius)

    @property
    def radius(self) -> int|float:
        return self._radius

    @radius.setter
    def radius(self, value: int|float):
        
        self._radius = radius_validator(value)

    def area(self) -> float:
        return π * self._radius**2 

    def circumference(self) -> float:
        return 2 * π * self._radius

    def __repr__(self):

        return f"area={self.area(): .2f};circumference={self.circumference():.2f}"

# c = Circle(radius=42) 
# c.radius = -1   # => throws ValueError(f"Radius cannot be negative 😞")
# c.radius = "42" # => throws TypeError(f"Radius has to be a positive int or float 😞") 

```

Up to this point, things are looking promising. The power of setters and getters extends far beyond mere value validation at assignments; they unlock Jedi-like capabilities. Consider this: in the realm of classical Machine Learning, what if we employed a setter to save a fitted transformer and later, during prediction, utilized a getter to retrieve it? Intriguing, isn't it?


```python
import pickle
from pathlib import Path
from typing import Literal
import pandas as pd
from sklearn.compose import ColumnTransformer


class TransformerTask:
    def __init__(self, transformer:ColumnTransformer, 
                 stage: Literal["train", "predict"] = "train", 
                 file_path:str="models/transformer.pkl",):
        self.stage = stage
        self.file_path = Path(file_path)
        self._transformer = transformer

    @property
    def transformer(self) -> ColumnTransformer:
        if self.stage == "predict" and not self.file_path.exists():
            raise FileNotFoundError(f"{self.file_path} was not found.")

        if self.stage == "predict" and self.file_path.exists():
            self._transformer = pickle.loads(self.file_path.read_bytes())
            
        return self._transformer

    @transformer.setter
    def transformer(self, transformer:ColumnTransformer) -> None: 
        self.file_path.write_bytes(pickle.dumps(transformer))


    def run(self, data: pd.DataFrame) -> pd.DataFrame:
        operations = {
            "train": self._train,
            "predict": self._predict
        }
        return operations.get(self.stage, lambda d: d)(data)

    def _train(self, data: pd.DataFrame):
        cleaned_data = self.transformer.fit_transform(data)
        self.transformer = self._transformer
        return cleaned_data

    def _predict(self, data: pd.DataFrame):
        return self.transformer.transform(data)
```
Test we must, again young Padawan ...

```python
from sklearn.compose import make_column_transformer
from sklearn.preprocessing import OneHotEncoder
from sklearn import set_config

from srp import config

set_config(transform_output="pandas")


transformer = make_column_transformer(
    (
        OneHotEncoder(sparse_output=False),[
            config.COLUMNS_TO_ONEHOTENCODE,
        ],
    ),
    verbose_feature_names_out=False,
    remainder="passthrough",
)
```
Oops, now test, we can 🥹

```python
from pathlib import Path
import pytest
import pandas as pd
from tasks import TransformerTask 
from preprocessors import transformer 



# URI for the dataset
URI = "https://raw.githubusercontent.com/mwaskom/seaborn-data/master/penguins.csv"

# Model File path
MODEL_FILE = Path("models/transformer.pkl")

def cleanup(file=MODEL_FILE):
    if MODEL_FILE.exists():
        MODEL_FILE.unlink() 

@pytest.fixture(autouse=True)
def startup_and_teardown():
    "Execute before and after a test run"
    # Startup: remove model file
    cleanup()
    yield
    # Teardown : remove model file
    cleanup()

@pytest.fixture
def load_data():
    yield pd.read_csv(URI)


@pytest.fixture
def transformer_task():
    yield TransformerTask(transformer, stage="train")


def test_transformer_train(load_data, transformer_task):
    "Test the training stage"
    
    result = transformer_task.run(load_data) 
    assert not result.empty
    assert MODEL_FILE.exists()


def test_transformer_predict(transformer_task, load_data):
    "Test the prediction stage"
    # First, run the training stage to ensure the transformer is fitted and saved
    transformer_task.run(load_data)
    transformer_task.stage = "predict"

    # Testing prediction stage
    assert MODEL_FILE.exists()
    assert transformer_task.stage == "predict" 
    
    result = transformer_task.run(load_data)
    assert not result.empty

```

### Reflect and Contemplate:
Wield this tool with a balance to avert the shadows of accidental recursion or the echoes of unwanted side effects.
___
## 3. **Decorators: The Enchanters**
![russian doll](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/is5votm48pq4ag8qltxc.jpeg)

Summon the might of Decorators, enchanters that weave their spells to transform the behaviour of your functions or methods, bestowing upon them new realms of possibilities.

Create a decorator, we must. Allow it will, to profile bottlenecks within our functions/methods, hmmm.
 
Start simple, we shall. Expand later, we will, young Padawan.

```python
from typing import Callable
import numpy as np

def regression(func: Callable) -> Callable:
    func.kind = "regression"
    return func

@regression
def mse(y_true, y_pred):
    return ((y_true - y_pred)**2).mean()

print((mse.__name__, mse.kind))
# ('mse', 'regression')
```

Let's level up, doing a hyperspace jump 

```python
from cProfile import Profile
from functools import wraps
import pstats

def profiler(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        with Profile() as p:
            # run the function and collect profile data
            results = p.runcall(func, *args, **kwargs)
            
            # sort => name, time, file https://docs.python.org/3/library/profile.html
            ps = pstats.Stats(p, stream=None).sort_stats('cumulative') 
            ps.print_stats(5) # restriction of print
        return results
    return wrapper


@profiler
@regression
def mse(y_true, y_pred):
    return ((y_true - y_pred)**2).mean()

if __name__ == "__main__":
    y_true = np.array([1, 2, 3, 4, 5])
    y_pred = np.array([1, 3, 2, 3, 5])
    print((mse.__name__, mse.kind, results))
    # some profile logs
    # ('mse', 'regression', 0.6)


```
Another way of writing the same decorator, using `class`, would look like this:

```python
class profiler:
    def __init__(self, func):
        self.func = func
        self.__name__ = func.__name__
        
    def __call__(self, *args, **kwargs):
        with Profile() as p:
            results = p.runcall(self.func, *args, **kwargs)

            ps = pstats.Stats(p, stream=None).sort_stats('cumulative')
            ps.print_stats(5) 
        
        return results
```


Hark! A disturbance in the Force. Hardcoded sorting and stats printing lines. A desire to inject them, I sense. Embark on this quest, we must. Bring balance to the code, we shall. Doing that, let us.

```python
# functional way
def profiler(sort_by='cumulative', restriction=5, streams=None):
    def inner_function(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            with Profile() as p:
                result = p.runcall(func, *args, **kwargs)  
                ps = pstats.Stats(p, stream=streams).sort_stats(sort_by)  
                ps.print_stats(restriction)  
            return result
        return wrapper
    return inner_function
    
# class way
class profiler:
    def __init__(self, sort_by='cumulative', restriction=5, streams=None):
        self.sort_by = sort_by
        self.restriction = restriction
        self.streams = streams
    
    def __call__(self, func):
        def wrapper(*args, **kwargs):
            with Profile() as p:
                result = p.runcall(func, *args, **kwargs)  
                ps = pstats.Stats(p, stream=self.streams).sort_stats(self.sort_by)  
                ps.print_stats(self.restriction)  
            return result
        wrapper.__name__ = func.__name__
        return wrapper

@regression
@profiler(sort_by="time", restriction=3)
def mse(y_true, y_pred):
    return ((y_true - y_pred)**2).mean()

if __name__ == "__main__":
    y_true = np.array([1, 2, 3, 4, 5])
    y_pred = np.array([1, 3, 2, 3, 5])
    results = mse(y_true, y_pred)

    print((mse.__name__, mse.kind, results))
    # some profile logs are sorted by time and show 3 lines
    # ('mse', 'regression', 0.6)
```

Ah, a wise query emerges amidst the stars! Speak of adorning methods within a cosmic class, do you? Fear not, for a class decorator we shall craft. Bestow our decorator upon the class methods, it shall. In unity, traverse the celestial pathways of Python, we will!

```python
def profilerx(decorate=None):
    if decorate is None:
        decorate = lambda d: d

    def wrapper(cls):
        name_functions = {name:func for name, func in vars(cls).items() if not name.startswith("__")}
 
        for name, func in vars(cls).items():
            if callable(func):
                setattr(cls, name, decorate(func))
        
        return cls
        wrapper.__name__ = cls.__name__
        
    return wrapper

# Decorate our class, we can ☺️

@profilerx(decorate=profiler(sort_by="cumulative", restriction=5))
class Circle:
    def __init__(self, radius: int|float):
        self._radius = radius_validator(radius)
   ...
```  

### Reflect and Contemplate:
Embrace this magic with mindfulness, for within its allure, lies the labyrinth of complexity, whispering tales of code entangled in its own enchantment.

___
## 4. **Metaprogramming: The Arcane Arts**
![meta](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qofx2qxiamjmor1ntsfs.jpeg)

As we traverse the realms of decorators, we find ourselves at the gateway to Metaprogramming, a universe where code breathes life into more code. Imagine a scenario where the reins of our crafted code slip out of our grasp, handed over to realms and teams beyond our dominion. In this abyss, where control eludes our touch, how do we ensure our shields are not tampered with? Behold the luminescence of metaprogramming, a beacon in the shadows of constraint and order.

Envision a creation of our own, a code entity named `RejectPrint`, destined to embark upon a voyage to distant teams and uses. As it journeys beyond our realm, we yearn to engrave a solemn vow upon its essence – the exile of the `print` command, guarding the sanctity of silence amidst its ventures in the unknown.


```python
# our modulex.py
import logging
import inspect
from typing import Callable


def check_bad_usage(name: str, obj: Callable)-> bool:
	"""
	checks if an object has a function with certain name
	"""
	
	func_names = obj.__code__.co_names
	is_bad_usage = False
	
	if name not in func_names:
		return is_bad_usage
		
	is_bad_usage = True
	for func_name in func_names:
		
		 
		if name == func_name:
			# echo the code with bad usage
			logging.error(inspect.getsource(obj.__code__))
	return is_bad_usage
			
	

class RejectPrintMeta(type):
	
	def __new__(cls, name, bases, body):
		
		callable_obj = (v for v in body.values() if callable(v))
		
		for obj in callable_obj:
			is_bad_usage = check_bad_usage("print", obj)
			if is_bad_usage:
				raise UserWarning(f"No way, `{obj.__name__}` contains `print` function!")
			
		return super().__new__(cls, name, bases, body)
	

	
class RejectPrint(metaclass=RejectPrintMeta):
	pass
	
```

As users employ our package, a silent guardian emerges at the birth of their classes. The use of `print` is gently yet firmly barred, ensuring unbroken harmony in the world of our tool’s operation.


```python	
# example.py
# from modulex import RejectPrint

class UserPrint(RejectPrint):
	
	def no_print_used(self):
		pass
		
	def print_used(self):
		print('using print :-o')
```

![beautiful error message](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/95kobdm1aui37cyo2s78.png)


### Reflect and Contemplate:
Tread with care in the realm of metaprogramming, for the powers it unleashes, while potent, whisper the tales of complexity and enigma.

___
## Conclusion: Beginning of the Start

I incorporated these techniques in my repertoire, as a young Padawan. I watched the doors to Python mastery unfold before me. It is my hope that you continue to explore, learn, and adapt, and let the Force guide your own enlightening journey.

And so, the odyssey through the realms of advanced Python concludes yet begins. May your code flourish in elegance and efficiency, nurtured by the seeds of wisdom sown today. 


`Until` then, may the force keep you coding.

_Note:_ Generators, context managers, and plugins are additional Jedi tools that I've had to set aside for this post to avoid making it overly lengthy. If your curiosity is piqued and you desire to explore these realms further, do let me know. A sequel awaits the beckoning of your interest, ready to guide you further on this path to Python mastery.