# Python Descriptor

## Summary


Python objects have attributes that can be accessed, modified or deleted. Python provides a default manner in which these operations are performed but what if we wish to customize these operations? This is where descriptors are helpful. 

Descriptors can be used to manage access to an attribute, validate attribute values, customize error messages, track data corruption bugs, or perform dynamic computations on attribute lookup. Descriptors are commonly defined at class level but when defined at module level they can be used to display deprecation warnings or implement lazy loading. 

It's been said that a good understanding of descriptors can improve a developer's coding skills. Descriptors are commonly used by library developers.

## Discussion

### What are some use cases of Python descriptors?

Suppose there's a `Directory` class that has a `size` attribute to reflect the number of files in that directory. Naturally, its value can change dynamically as files are added or removed. Without descriptors, we might implement this as a method `get_size()`. The method will query the file system and obtain the current size. 

Descriptors allow us to get the size dynamically while also accessing the size as a data attribute. Thus, we can say `d.size` rather than `d.get_size()`. Moreover, the behaviour of querying the file system was previously within `Directory.get_size()`. With descriptors, the behaviour is nicely encapsulated within the descriptor class `DirectorySize`.

Another use case of descriptors is to manage data access. For example, public attribute `age` is exposed in a `Person` class but managed privately as `_age` within a descriptor class. The descriptor class could also perform validations. For example, when assigning a value to `age` it can check if the value is a non-negative integer.  

A real-world example of a descriptor is `GenericForeignKey` in Django. 


### How can I implement Python's descriptor protocol?

Assume one or more instances of TextWrapper class are instantiated as members of Email class. TextWrapper is the **descriptor class** and Email is the **owner class**. To support the descriptor protocol, TextWrapper implements the following methods (but not necessarily all): 

    + `&lowbar;&lowbar;get&lowbar;&lowbar;(self, instance, owner=None)`: Called when the attribute of the owner class or its instance is accessed. Eg. `print(e.sender)`.
    + `&lowbar;&lowbar;set&lowbar;&lowbar;(self, instance, value)`: Called to set the attribute on an instance of the owner class. Eg. `e.sender = 'foo'`.
    + `&lowbar;&lowbar;delete&lowbar;&lowbar;(self, instance)`: Called to delete the attribute on an instance of the owner class. Eg. `del(e.sender)`.
    + `&lowbar;&lowbar;set_name&lowbar;&lowbar;(self, owner, name)`: Called when the owner class is created. This allows descriptor instances to access their own name as defined in the owner class.An object that has defined one or more of `&lowbar;&lowbar;get&lowbar;&lowbar;`, `&lowbar;&lowbar;set&lowbar;&lowbar;` or `&lowbar;&lowbar;delete&lowbar;&lowbar;` is a descriptor. Descriptors work only as class variables of the owner class. 

The statement `e.sender += 'bar'` will automatically call both `&lowbar;&lowbar;get&lowbar;&lowbar;` and `&lowbar;&lowbar;set&lowbar;&lowbar;`. 


### What's the difference between data and non-data descriptors?

A descriptor that defines `&lowbar;&lowbar;set&lowbar;&lowbar;` and/or `&lowbar;&lowbar;delete` is a **data descriptor**. A descriptor that defines only `&lowbar;&lowbar;get&lowbar;&lowbar;` is a **non-data descriptor**. 

Data descriptors override redefinitions in instance dictionary whereas instances can override non-data descriptors. 

For a descriptor that doesn't define `&lowbar;&lowbar;get&lowbar;&lowbar;`, attribute access will return the value from the instance dictionary. If this doesn't exist, the descriptor object is returned. 

Non-data descriptors are typically used for methods. In fact, Python methods are implemented as non-data descriptors, including those decorated with `@staticmethod` and `@classmethod`. Instances can override these methods. On the other hand, `property()` function is implemented as a data descriptor, meaning that instances can't override its behaviour. 


### Why do we need the descriptor protocol when we have &lowbar;&lowbar;getattr&lowbar;&lowbar;, &lowbar;&lowbar;setattr&lowbar;&lowbar; and &lowbar;&lowbar;delattr&lowbar;&lowbar;?

Methods `&lowbar;&lowbar;getattr&lowbar;&lowbar;`, `&lowbar;&lowbar;setattr&lowbar;&lowbar;` and `&lowbar;&lowbar;delattr&lowbar;&lowbar;` allow us to customize the behaviour of attribute access. These methods are defined on the owner class. If the owner class has many attributes to manage, these methods could become complex to maintain. Method implementations may have ugly `if-else` constructs to manage the different attributes.

Descriptors offer a more modular, maintainable and extensible approach. Instead of the owner class controlling how attributes should be managed, the control is now with the class of the attribute being managed. In other words, descriptors have a say on how their instances should be managed. 

In the figure, we see how any value assigned to a name attribute is capitalized. Method `&lowbar;&lowbar;setattr&lowbar;&lowbar;` of Person class can be used for this but requires the attribute be named "name". The descriptor protocol offers a better way. Class CapitalizedValue is a data descriptor that does the capitalization. Person class defines an attribute of name "name" but any other name could have been given. 


### How do Python descriptors work under the hood?

How Python descriptors work under the hood is basically about how the language does attribute access. Python applies a bunch of rules in a well-defined order of precedence. This is shown in the figure for both get and set/delete operations. These rules check whether class overrides the access methods `&lowbar;&lowbar;getattribute&lowbar;&lowbar;` and `&lowbar;&lowbar;getattr&lowbar;&lowbar;`, whether class or instance access is performed, and whether attribute is a descriptor object and what parts of the descriptor protocol are implemented. 

We can see that for a get operation, data descriptor has higher precedence than instance attribute, which in turn has higher precedence than non-data descriptor. 

We can summarize the lookups for instance and class attribute accesses: 

    + **Get**: `obj.x` ⇒ `Cls.&lowbar;&lowbar;dict&lowbar;&lowbar;['x'].&lowbar;&lowbar;get&lowbar;&lowbar;(obj, Cls)`and `Cls.x` ⇒ `Cls.&lowbar;&lowbar;dict&lowbar;&lowbar;['x'].&lowbar;&lowbar;get&lowbar;&lowbar;(None, Cls)`
    + **Set**: `obj.x = v` ⇒ `Cls.&lowbar;&lowbar;dict&lowbar;&lowbar;['x'].&lowbar;&lowbar;set&lowbar;&lowbar;(obj, val)` and `Cls.x = v` ⇒ Regular override
    + **Delete**: `del obj.x` ⇒ `Cls.&lowbar;&lowbar;dict&lowbar;&lowbar;['x'].&lowbar;&lowbar;delete&lowbar;&lowbar;(obj)` and `del Cls.x` ⇒ Regular deletion

### What other features of Python are implemented by descriptors?

Built-in functions `classmethod()`, `staticmethod()`, `property()`, and `functools.cached_property()` are all implemented as descriptors. 

A **property attribute** is one that triggers method calls when accessed. Property is implemented as a data descriptor.  It has an easier interface than a descriptor and a different abstraction. The methods are typically defined in the same class in which the attribute resides. There are two ways to link methods to a property: (i) using the `property(fget=None, fset=None, fdel=None, doc=None)` built-in function; (ii) using decorators `@property`, `@x.setter` and `@x.deleter` where `x` is the name of the property and also the name of the methods so decorated.  

Another Python feature is **slots**, available as a class variable `&lowbar;&lowbar;slots&lowbar;&lowbar;`. This variable can be assigned all allowed attributes of the class and its instances. Thus, instances can't add new attributes. Instances also don't contain `&lowbar;&lowbar;dict&lowbar;&lowbar;` and `&lowbar;&lowbar;weakref&lowbar;&lowbar;` unless explicitly included in `&lowbar;&lowbar;slots&lowbar;&lowbar;`. Every item in `&lowbar;&lowbar;slots&lowbar;&lowbar;` is implemented as a descriptor. 


### Could you share some tips for working with Python descriptors?

For most use cases, or at least for beginners, property is a simpler interface than defining a descriptor from scratch. In fact, property has been called a "high-level descriptor builder" or a "descriptor factory". 

A read-only data descriptor can be constructed by defining both `&lowbar;&lowbar;get&lowbar;&lowbar;` and `&lowbar;&lowbar;set&lowbar;&lowbar;` but the latter is coded to raise an `AttributeError` exception. 

We can use a descriptor to implement a one-time initialization. This is done by setting an internal flag in `&lowbar;&lowbar;set&lowbar;&lowbar;`. 

Descriptors enable many other use cases elegantly. Logging all accesses of an attribute can be implemented by a descriptor. When an attribute changes, we can update all dependent attributes, such as, updating area when circle radius changes. Lazy evaluation is another use case in which behaviour is invoked only when necessary. 

## Milestones

Jan  
1994

Python 1.0 is released. Descriptors don't exist in this initial release. 

Dec  
2001

Python 2.2 is released. This introduces **descriptors** into the language via PEP 252. Descriptors themselves have the following attributes: `&lowbar;&lowbar;name&lowbar;&lowbar;`, `&lowbar;&lowbar;doc&lowbar;&lowbar;`, `&lowbar;&lowbar;get&lowbar;&lowbar;()`, `&lowbar;&lowbar;set&lowbar;&lowbar;()` and `&lowbar;&lowbar;delete&lowbar;&lowbar;()`. Descriptors are now used to support static methods and class methods. New features slots and properties are also new kinds of descriptors. 

Jun  
2011

Python 2.7.2 is released. In the accompanying documentation, Hettinger states in the *Descriptor HowTo Guide* that,

> Learning about descriptors not only provides access to a larger toolset, it creates a deeper understanding of how Python works and an appreciation for the elegance of its design.

Feb  
2015

In PEP 487, Teichmann proposes the use of `&lowbar;&lowbar;set_name&lowbar;&lowbar;()` initializer for descriptors. While a descriptor knows its owner when `&lowbar;&lowbar;get&lowbar;&lowbar;()` is called, it doesn't know its own name, that is, the instance name that the owner has defined. Method `&lowbar;&lowbar;set_name&lowbar;&lowbar;()` solves this problem. This feature is accepted for Python 3.6 (December 2016). 

Sep  
2017

In PEP 549, Hastings proposes **instance descriptors**, since currently Python supports descriptors only as members of the type of an object. This proposal is rejected in preference to a simpler proposal by Levkivskyi in PEP 562. The idea is to support `&lowbar;&lowbar;getattr&lowbar;&lowbar;()` and `&lowbar;&lowbar;dir&lowbar;&lowbar;()` at module level. These allow customization of module attribute access. Two use cases for this are deprecation warnings and lazy loading. This feature is accepted for Python 3.7 (June 2018). 

Aug  
2018

At PyCon Australia, Egan reveals his analysis of the use of the descriptor protocol in public GitHub repositories. Method `&lowbar;&lowbar;get&lowbar;&lowbar;()` is implemented 1.2 million times. Method `_set_name&lowbar;&lowbar;()` is implemented the least since it's been around only since Python 3.6.