# Magic Methods in Python

## Summary


Magic methods are special methods that are called implicitly by the Python interpreter to execute corresponding operations. They help customize common operators or constructs. They make code more intuitive and Pythonic.

For example, to add two Python objects we might write `a + b`. The actual execution would call the magic method `a.&lowbar;&lowbar;add&lowbar;&lowbar;()` implicitly to perform this addition. If this magic method doesn't exist, the Python interpreter would call `b.&lowbar;&lowbar;radd&lowbar;&lowbar;()`. Application code where addition is invoked is itself simple and readable. 

There are several conceptual groups of magic methods such as object lifecycle, object representation, object comparison, object attributes, class creation, callability, containers, numerics, context management, and more. This article describes many of these and shares how to use them.

## Discussion

### What are the characteristics of magic methods?

Magic methods are simply special methods. They have system-defined names **starting and ending with double underscores**, informally known as *dunder* names. Not all of these are magic methods. 

Several built-in classes define various magic methods. Output of `dir()` can be filtered to see only dunder names but not all are magic methods. Further, `callable()` can be used to introspect these names, but not all callables are magic methods. Ultimately, Python documentation clarifies which attributes are magic methods. For example, as illustrated in the figure, `&lowbar;&lowbar;doc&lowbar;&lowbar;` isn't one and `&lowbar;&lowbar;class&lowbar;&lowbar;` despite being callable isn't one. 

The code above shows implicit calls to `&lowbar;&lowbar;str&lowbar;&lowbar;()` and `&lowbar;&lowbar;repr&lowbar;&lowbar;()` magic methods by `str()` and `repr()` respectively. These return suitable string representations of the `Point2d` instances. Magic methods thus serve these purposes:

    + Enable classes to define their own behaviour corresponding to Python operators or constructs.
    + Enable developers to write Pythonic code. A developer could have written `to_string()` method but magic methods provide a uniform convention. This reduces technical debt.
    + Enable code reuse and design patterns. For example, custom objects can be compared and sorted based on magic methods without extra effort.

### Could you give an overview of the main magic methods?

Python 3.10.1 has dozens of magic methods. We discuss them within a few conceptual groups:  

    + **Object Lifecycle**: These relate to creation, initialization and destruction of objects. Eg. `&lowbar;&lowbar;new&lowbar;&lowbar;()`, `&lowbar;&lowbar;init&lowbar;&lowbar;()`, `&lowbar;&lowbar;del&lowbar;&lowbar;()`.
    + **String Representation**: These represent objects as strings. Eg. `&lowbar;&lowbar;repr&lowbar;&lowbar;()`, `&lowbar;&lowbar;str&lowbar;&lowbar;()`.
    + **Comparisons**: To compare objects via operators `<`, `==`, `>=`, etc. Eg. `&lowbar;&lowbar;lt&lowbar;&lowbar;()`, `&lowbar;&lowbar;eq&lowbar;&lowbar;()`, `&lowbar;&lowbar;ge&lowbar;&lowbar;()`.
    + **Math Operations**: Arithmetic operators `+`, `*`; bit operators `&`, `<<`; and math functions. Eg. `&lowbar;&lowbar;add&lowbar;&lowbar;()`, `&lowbar;&lowbar;rshift&lowbar;&lowbar;()`, `&lowbar;&lowbar;floor&lowbar;&lowbar;()`.
    + **Type Conversions**: Convert from one type to another. Eg. `&lowbar;&lowbar;bool&lowbar;&lowbar;()`, `&lowbar;&lowbar;int&lowbar;&lowbar;()`.
    + **Type Check**: Check class hierarchy. Eg. `&lowbar;&lowbar;instancecheck&lowbar;&lowbar;()`, `&lowbar;&lowbar;subclasscheck&lowbar;&lowbar;()`.
    + **Context Manager**: To set up and clean up objects within a context, such as open/close of files. Eg. `&lowbar;&lowbar;enter&lowbar;&lowbar;()`, `&lowbar;&lowbar;exit&lowbar;&lowbar;()`.
    + **Attribute Access**: Control access or inspect instance attributes. Eg. `&lowbar;&lowbar;getattr&lowbar;&lowbar;()` , `&lowbar;&lowbar;dir&lowbar;&lowbar;()` , `&lowbar;&lowbar;set&lowbar;&lowbar;()` .
    + **Operations on Sequences**: Find length, get/set/delete items or iterate through items of a sequence. Eg. `&lowbar;&lowbar;len&lowbar;&lowbar;()`, `&lowbar;&lowbar;getitem&lowbar;&lowbar;()`, `&lowbar;&lowbar;contains&lowbar;&lowbar;()`.
    + **Asynchronous**: Pertaining to asynchronous operations. Eg. `&lowbar;&lowbar;await&lowbar;&lowbar;()`, `&lowbar;&lowbar;aiter&lowbar;&lowbar;()`, `&lowbar;&lowbar;aenter&lowbar;&lowbar;()`.
    + **Miscellaneous**: Not covered above. Eg. `&lowbar;&lowbar;call&lowbar;&lowbar;()`, `&lowbar;&lowbar;mro_entries&lowbar;&lowbar;()`.Some Python modules add their own magic methods. Examples include `copy.&lowbar;&lowbar;deepcopy&lowbar;&lowbar;()` and `pickle.&lowbar;&lowbar;reduce&lowbar;&lowbar;()`. 


### Are magic methods Python's approach to operator overloading?

Indeed, magic methods are Python's approach to operator overloading. 

However, there are many magic methods that don't correspond to an operator. For example, `&lowbar;&lowbar;str&lowbar;&lowbar;()`, `&lowbar;&lowbar;bool&lowbar;&lowbar;()` and `&lowbar;&lowbar;repr&lowbar;&lowbar;()` are magic methods corresponding to functions rather than operators.

Further, for example, the `object` class implements `&lowbar;&lowbar;eq&lowbar;&lowbar;()`, which corresponds to the `==` operator. When we implement this magic method we're technically overriding a base class method in OOP parlance.  However, we're actually overloading the `==` operator. 

This is why it's better to think of magic methods as *intuitive interfaces*.


### What magic methods differ between Python 2 and Python 3?

For object comparisons, Python 2's magic method `&lowbar;&lowbar;cmp&lowbar;&lowbar;()` is replaced with *rich comparison methods* in Python 3: `&lowbar;&lowbar;lt&lowbar;&lowbar;()`, `&lowbar;&lowbar;le&lowbar;&lowbar;()`, `&lowbar;&lowbar;eq&lowbar;&lowbar;()`, `&lowbar;&lowbar;ne&lowbar;&lowbar;()`, `&lowbar;&lowbar;gt&lowbar;&lowbar;()`, `&lowbar;&lowbar;ge&lowbar;&lowbar;()`. Because it can be extra work to implement all these magic methods for a class, Python 3 provides the decorator `@functools.total_ordering`. We can provide implementations for `&lowbar;&lowbar;eq&lowbar;&lowbar;()` and one of the ordering operators. The decorator will supply the rest. 

Python 2's magic method `&lowbar;&lowbar;nonzero&lowbar;&lowbar;()` is replaced with `&lowbar;&lowbar;bool&lowbar;&lowbar;()`. Method `&lowbar;&lowbar;div&lowbar;&lowbar;()` is replaced with `&lowbar;&lowbar;truediv&lowbar;&lowbar;()`. Method `&lowbar;&lowbar;unicode&lowbar;&lowbar;()` is removed since strings in Python 3 are Unicode by default. 

Python 2's `&lowbar;&lowbar;coerce&lowbar;&lowbar;` is no longer needed since other magic methods do this in Python 3. For example, `2 + 3.5` would do type coercion in Python 2 but in Python 3, `&lowbar;&lowbar;radd&lowbar;&lowbar;()` on the float object is called to add the integer. 


### Which magic methods are available for object lifecycle constructs?

`&lowbar;&lowbar;new&lowbar;&lowbar;()`, `&lowbar;&lowbar;init&lowbar;&lowbar;()` and `&lowbar;&lowbar;del&lowbar;&lowbar;()` are magic methods that allow customization of object creation, initialization and destruction respectively.

`&lowbar;&lowbar;new&lowbar;&lowbar;()` is mainly intended to allow subclasses of immutable types to customize instance creation and to customize class creation in custom metaclasses. `&lowbar;&lowbar;init&lowbar;&lowbar;()` is mainly intended for object customization before returning to the caller; that is, `&lowbar;&lowbar;new&lowbar;&lowbar;()` deals with creation and is a static class method while `&lowbar;&lowbar;init&lowbar;&lowbar;()` deals with initialization and is an instance method. `&lowbar;&lowbar;del&lowbar;&lowbar;()` is for runtime clean up just before an instance is destroyed, which only happens if the corresponding reference count becomes zero.  

There are magic methods to customize class creation itself. Classes themselves are objects: a class is created from its metaclass. Classes themselves use memory, can be assigned, copied and passed as parameters, and even dynamically created. Magic method `&lowbar;&lowbar;init_subclass&lowbar;&lowbar;()` influences the behaviour of future subclasses as this method is called whenever the class is subclassed. Related magic methods include `&lowbar;&lowbar;set_name&lowbar;&lowbar;()`, `&lowbar;&lowbar;prepare&lowbar;&lowbar;()`, `&lowbar;&lowbar;instancecheck&lowbar;&lowbar;()`, `&lowbar;&lowbar;subclasscheck&lowbar;&lowbar;()`, and `&lowbar;&lowbar;class_getitem&lowbar;&lowbar;()`. 


### Which magic methods are available for object representation?

Objects of type `str` are inherently strings but any Python object can be given a string representation. Built-in functions `str()`, `repr()`, `bytes()`, `format()`, and `print()` serve this purpose. However, their default behaviours can be customized by defining suitable magic methods.

Built-in function `repr()` calls magic method `&lowbar;&lowbar;repr&lowbar;&lowbar;()` to obtain the *official* string representation. Built-in functions `str()` and `print()` call magic method `&lowbar;&lowbar;str&lowbar;&lowbar;()` to obtain an *informal* and printable string. Since `&lowbar;&lowbar;repr&lowbar;&lowbar;()` is typically used for debugging, it should return an information-rich and unambiguous string. If possible, it should be a valid Python expression string for data serialization so that the instance can be recreated from the string. 

Built-in function `bytes()` calls magic method `&lowbar;&lowbar;bytes&lowbar;&lowbar;()` to obtain a byte-string representation, which is primarily useful for storage and network transfers. 

Built-in function `format()` and string method `str.format()` call magic method `&lowbar;&lowbar;format&lowbar;&lowbar;()` to evaluate formatted string literals, also known as *format-strings*. If this magic method is not defined, `&lowbar;&lowbar;str&lowbar;&lowbar;()` is called instead. 


### Which magic methods are available for object comparison?

Comparison magic methods allow custom classes to implement specific comparison behaviours. Python defines six object comparison magic methods: `&lowbar;&lowbar;lt&lowbar;&lowbar;()`, `&lowbar;&lowbar;le&lowbar;&lowbar;()`, `&lowbar;&lowbar;eq&lowbar;&lowbar;()`, `&lowbar;&lowbar;ne&lowbar;&lowbar;()`, `&lowbar;&lowbar;gt&lowbar;&lowbar;()` and `&lowbar;&lowbar;ge&lowbar;&lowbar;()`, corresponding to `<`, `<=`, `==`, `!=`, `>`, and `>=` respectively. 

The magic methods should return the singleton `NotImplemented` if the comparison is not implemented for the requested operands. They should return a boolean value. Otherwise, Python will call `bool()` on the returned object. 

If `&lowbar;&lowbar;ne&lowbar;&lowbar;()` is undefined, its default behaviour is to call `&lowbar;&lowbar;eq&lowbar;&lowbar;()` and invert the result. No such relation exists among other comparison operators. For example, Python doesn't enforce that `x < y` is same as `y > x`. However, implementations should maintain semantic consistency.


### How can I customize object conversions?

Any Python object can be used in boolean context, such as `if obj: pass`. This operation is trivial for a `bool` type object. For other types, we need a way to evaluate the object in a boolean context. Magic method `&lowbar;&lowbar;bool&lowbar;&lowbar;()` does this.

Evaluating an object in a boolean context or directly calling the built-in function `bool()` calls `&lowbar;&lowbar;bool&lowbar;&lowbar;()`, which must return either `True` or `False`. Any other return value will cause Python to raise an exception. If `&lowbar;&lowbar;bool&lowbar;&lowbar;()` is undefined, `bool()` calls `&lowbar;&lowbar;len&lowbar;&lowbar;()`. The result is considered `True` if nonzero and `False` otherwise. If a class doesn't define either of these, all its instances are considered `True`. 

Likewise, built-in functions `complex()`, `int()` and `float()` trigger other magic methods. A call of `int()` checks for `&lowbar;&lowbar;int&lowbar;&lowbar;()`, `&lowbar;&lowbar;index&lowbar;&lowbar;()` and `&lowbar;&lowbar;trunc&lowbar;&lowbar;()` in that order; `float()` checks for `&lowbar;&lowbar;float&lowbar;&lowbar;()` and `&lowbar;&lowbar;index&lowbar;&lowbar;()`; `complex()` checks for `&lowbar;&lowbar;complex&lowbar;&lowbar;()`, `&lowbar;&lowbar;float&lowbar;&lowbar;()` and `&lowbar;&lowbar;index&lowbar;&lowbar;()`. 

When present, `&lowbar;&lowbar;index&lowbar;&lowbar;()` indicates that the object is of integer type. It must return an integer. Python slicing and functions `operator.index()`, `bin()`, `hex()` and `oct()` rely on this. 


### Which magic methods emulate numeric objects?

Numbers can be added, multiplied or bit-shifted. What if we wish to emulate these behaviours on non-numeric objects? For example, we could add two RGB colour objects or left shift beads on an abacus object. Magic methods associated with numeric operators enable this.

There are three categories of magic methods (we describe only the addition operator): 

    + **Left Operand**: Given `a + b`, the call `a.&lowbar;&lowbar;add&lowbar;&lowbar;(b)` is made.
    + **Right Operand**: Given `a + b`, the call `a.&lowbar;&lowbar;add&lowbar;&lowbar;(b)` is attempted first. If this doesn't exist or returns `NotImplemented`, `b.&lowbar;&lowbar;radd&lowbar;&lowbar;(a)` is called. For example, `3 + 5.5` will call `3.&lowbar;&lowbar;add&lowbar;&lowbar;(5.5)`. Since an integer doesn't know how to add a float, this returns `NotImplemented`. Then `5.5.&lowbar;&lowbar;radd&lowbar;&lowbar;(3)` is called and this succeeds with the addition.
    + **In-Place Operation**: Given `a += b`, the call `a.&lowbar;&lowbar;iadd&lowbar;&lowbar;(b)` is made. This method should modify the object in-place. If this method doesn't exist, `a.&lowbar;&lowbar;add&lowbar;&lowbar;(b)` and `b.&lowbar;&lowbar;radd&lowbar;&lowbar;(a)` are tried in that order.Related to these are unary operators `+` and `-` that call `&lowbar;&lowbar;pos&lowbar;&lowbar;()` and `&lowbar;&lowbar;neg&lowbar;&lowbar;()`; built-in function `abs()` calls `&lowbar;&lowbar;abs&lowbar;&lowbar;()`; similar built-in functions (some in `math` module) call `&lowbar;&lowbar;round&lowbar;&lowbar;()`, `&lowbar;&lowbar;trunc&lowbar;&lowbar;()`, `&lowbar;&lowbar;floor&lowbar;&lowbar;()`, and `&lowbar;&lowbar;ceil&lowbar;&lowbar;()`. 


### Which magic method makes class instances callable?

Given a class `A` and its initializer `&lowbar;&lowbar;init&lowbar;&lowbar;(self, i)`, we can instantiate an object by doing `a = A(2)`. Now `a` is an instance. But it can't be called like a method or function. However, if the class `A` includes the magic method `&lowbar;&lowbar;call&lowbar;&lowbar;()`, we can write `a()`. Thus, the instance becomes callable (with or without arguments) when the magic method `&lowbar;&lowbar;call&lowbar;&lowbar;()` is defined. 

Class instantiation itself is just calling the corresponding *class object*. With `&lowbar;&lowbar;call&lowbar;&lowbar;()`, we're enabling a similar syntax on instances.

Callable instances have practical use cases. With `&lowbar;&lowbar;init&lowbar;&lowbar;()` an object can be initialized in a fundamental way. Suppose it has only one specific behaviour. Hence it defines no methods other than `&lowbar;&lowbar;call&lowbar;&lowbar;()`. Different types of instances are then created and called. An example is initializing a `Hasher` object with an algorithm: `md5 = Hasher(hashlib.md5)` and `sha1 = Hasher(hashlib.sha1)`. Actual hashing happens later by calling these instances: `md5('somefile.txt')` or `sha1('somefile.txt')`. 

Implementing decorators or presenting a consistent API are other use cases of `&lowbar;&lowbar;call&lowbar;&lowbar;()`. 


### Which magic methods are available for customization of attribute access?

`&lowbar;&lowbar;getattr&lowbar;&lowbar;()`, `&lowbar;&lowbar;getattribute&lowbar;&lowbar;()`, `&lowbar;&lowbar;setattr&lowbar;&lowbar;()` and `&lowbar;&lowbar;delattr&lowbar;&lowbar;()` allow us to customize attribute access for class instances. This includes use of, assignment to or deletion of `a.x`, where `x` is an attribute of instance `a`. 

Built-in function `dir()` calls magic method `&lowbar;&lowbar;dir&lowbar;&lowbar;()` to customize the list of valid instance attributes. Moreover, `&lowbar;&lowbar;getattr&lowbar;&lowbar;()` and `&lowbar;&lowbar;dir&lowbar;&lowbar;()` can be used to customize access to module attributes. 

A feature of the language called **descriptor** introduces further magic methods: `&lowbar;&lowbar;get&lowbar;&lowbar;()`, `&lowbar;&lowbar;set&lowbar;&lowbar;()` and `&lowbar;&lowbar;delete&lowbar;&lowbar;()`. These are more powerful and present a more modular way to control attribute access. 


### How can I make a custom object hashable?

Certain data structures in Python such as sets, frozensets and dictionaries are implemented using hash tables for efficient lookups and insertions. This requires computing a hash value on the object by calling the built-in function `hash()`. Internally, this calls the magic method `&lowbar;&lowbar;hash&lowbar;&lowbar;()`.

By defining this magic method on a class, we can compute the hash value for objects of that class. It must return an integer. Objects that compare equal via the method `&lowbar;&lowbar;eq&lowbar;&lowbar;()` must have the same hash value. One suggested way to compute a proper hash is to pack comparison-sensitive components of an object into a tuple and hash that tuple. For example, `return hash((self.x, self.y))` as the body of `&lowbar;&lowbar;hash&lowbar;&lowbar;()` method on a class `Point2d`. Note that `&lowbar;&lowbar;hash&lowbar;&lowbar;()` in general is tightly coupled with `&lowbar;&lowbar;eq&lowbar;&lowbar;()` and has atypical behaviour related to `&lowbar;&lowbar;hash&lowbar;&lowbar;()` implementations of `str` and `bytes` types. 

## Milestones

Sep  
2000

Python 1.6 is released. Magic method `&lowbar;&lowbar;contains&lowbar;&lowbar;()` is introduced to allow overriding the `in` operator. 

Apr  
2001

Python 2.1 is released. Magic methods `&lowbar;&lowbar;lt&lowbar;&lowbar;()`, `&lowbar;&lowbar;le&lowbar;&lowbar;()`, `&lowbar;&lowbar;eq&lowbar;&lowbar;()`, `&lowbar;&lowbar;ne&lowbar;&lowbar;()`, `&lowbar;&lowbar;gt&lowbar;&lowbar;()` and `&lowbar;&lowbar;ge&lowbar;&lowbar;()` are introduced to allow overriding the corresponding comparison operators. 

Dec  
2001

Python 2.2 is released. This introduces **descriptors** into the language via PEP 252. Descriptors themselves have the following attributes: `&lowbar;&lowbar;name&lowbar;&lowbar;`, `&lowbar;&lowbar;doc&lowbar;&lowbar;`, `&lowbar;&lowbar;get&lowbar;&lowbar;()`, `&lowbar;&lowbar;set&lowbar;&lowbar;()` and `&lowbar;&lowbar;delete&lowbar;&lowbar;()`. Descriptors are now used to support static methods and class methods. New features slots and properties are also new kinds of descriptors. 

Sep  
2006

Python 2.5 is released. This introduces the `with` statement and its associated magic methods `&lowbar;&lowbar;enter&lowbar;&lowbar;()` and `&lowbar;&lowbar;exit&lowbar;&lowbar;()`. A common use case of this is to open a file in `&lowbar;&lowbar;enter&lowbar;&lowbar;()` when the `with` context is entered and close the file in `&lowbar;&lowbar;exit&lowbar;&lowbar;()` as soon as execution comes out of the `with` context. Another new method is `&lowbar;&lowbar;index&lowbar;&lowbar;()` that solves a problem faced by NumPy users. It's now possible to use NumPy types as slice indexes. 

Oct  
2008

Python 2.6 is released. Some features of Python 3.0 are backported to Python 2.6. One of these is the `&lowbar;&lowbar;complex&lowbar;&lowbar;()` magic method. Magic method `&lowbar;&lowbar;dir&lowbar;&lowbar;()` is introduced. Built-in function `dir()` now looks for it on the object it receives. 

Dec  
2008

Python 3.0 is released. Magic method `&lowbar;&lowbar;cmp&lowbar;&lowbar;()` is removed. Instead use `&lowbar;&lowbar;lt&lowbar;&lowbar;()` for sorting, `&lowbar;&lowbar;eq&lowbar;&lowbar;()` with `&lowbar;&lowbar;hash&lowbar;&lowbar;()` and other rich comparisons. Operator `!=` returns the opposite of `==`. Methods `&lowbar;&lowbar;getslice&lowbar;&lowbar;()`, `&lowbar;&lowbar;setslice&lowbar;&lowbar;()` and `&lowbar;&lowbar;delslice&lowbar;&lowbar;()` are removed. The slice `a[i:j]` is implemented as `a.&lowbar;&lowbar;getitem&lowbar;&lowbar;(slice(i, j))`. Methods `&lowbar;&lowbar;setitem&lowbar;&lowbar;()` and `&lowbar;&lowbar;delitem&lowbar;&lowbar;()` are introduced. Converters `oct()` and `hex()` now use `&lowbar;&lowbar;index&lowbar;&lowbar;()`. Method `&lowbar;&lowbar;nonzero&lowbar;&lowbar;()` is now `&lowbar;&lowbar;bool&lowbar;&lowbar;()`. 

Sep  
2015

Python 3.5 is released. Magic methods for asynchronous programming are introduced: `&lowbar;&lowbar;await&lowbar;&lowbar;()`, `&lowbar;&lowbar;aiter&lowbar;&lowbar;()`, `&lowbar;&lowbar;anext&lowbar;&lowbar;()`, `&lowbar;&lowbar;aenter&lowbar;&lowbar;()`, and `&lowbar;&lowbar;aexit&lowbar;&lowbar;()`. 

Jun  
2018

Python 3.7 is released. Module-level methods introduced are `&lowbar;&lowbar;getattr&lowbar;&lowbar;()` and `&lowbar;&lowbar;dir&lowbar;&lowbar;()`. These can be used to implement module attribute deprecation and lazy loading. New methods `&lowbar;&lowbar;class_getitem&lowbar;&lowbar;()` and `&lowbar;&lowbar;mro_entries&lowbar;&lowbar;()` bring 7x improvement to performance in relation to types and the `typing` module. New module `dataclasses` has decorators and functions to automatically add some magic methods to user-defined classes. 

Oct  
2019

Python 3.8 is released. Constructors of `int`, `float` and `complex` will now use `&lowbar;&lowbar;index&lowbar;&lowbar;()` if their corresponding magic methods are not available. 

Oct  
2021

Python 3.10 is released. If `&lowbar;&lowbar;ipow&lowbar;&lowbar;()` returns `NotImplemented`, `&lowbar;&lowbar;pow&lowbar;&lowbar;()` and `&lowbar;&lowbar;rpow&lowbar;&lowbar;()` are the fallbacks.