How to migrate Python 2 projects to 3 & some useful tips
Tags: python, python3, migration, 2to3
Published: 2021-01-12 12:08:22

Python 2.7 came to end of life in January 2020 and even before that lot of software migrations to Python 3 have been happening. However, I was a little late to the party and got to work on one such project in late 2020. So in this blog, I'll try and summarize what steps we followed and some of the tips which can be helpful for a fellow developer who may need to migrate a Python project in future. 

Before we get started, let me also add a small disclaimer, that I'll not be touching upon the differences of Python 2 & 3, there is a lot of content available on the internet for the same so please feel free to check it out.

We'll be talking about the migration where we were not worried about Python2 compatibility because that was the case with us. However, I'll share some reference links for py2&3 compatible migrations as well. So let's get started and see what are the steps one can follow when migration to Python 3 is to be done. We migrated from Python 2.7 to 3.7 but the latest version is 3.9 so if you are confident enough to choose the latest Python version, you can do that as well.

## Steps for migration

- The first step would be to understand some fundamental difference between Python 2 & 3, that will help fix issues
- Check the code coverage of the code and make sure that we have decent test coverage. The more the code coverage, the easier would be to migrate the code
- Set up a virtual environment for both Python2.7 and Python 3.7+ versions


Virtual Env for Python2
```bash
python2.7 -m  pip install virtualenv
python2.7 -m virtualenv py27
source py27/bin/activate
```

Virtual Env for Python3
```bash
python3.7 -m virtualenv py3
source py3/bin/activate
```
- Also, do create a separate branch for Python 3 codebase and maintain these two versions parallelly unless the full migration is completed
- The next steps shall be to figure out the dependencies of the codebase compatible with Python 3, for that I'd recommend using the following tool
    - [canisepython3](https://pypi.org/project/caniusepython3/): This helps to find out which existing dependencies are compatible with python3
        ```bash 
        pip install caniusepython3
        ```
    - [pip-tools](https://pypi.org/project/pip-tools/): When we are migrating and we may also worried about pinning the dependencies & sub-deps then pip-tools is very handy, do explore it
        ```bash
        pip install pip-tools
        ```
- Once dependencies are sorted then run the test cases and find out the number of failures or any major issues, keep a note of test case failures, that will help you estimate the work up to some level
- After this, we need to move to the conversion of the code, so we already have some auto conversions tools available like [2to3](https://docs.python.org/3/library/2to3.html)
- In our case we used a library called [futurize](http://python-future.org/automatic_conversion.html) which is also based on [2to3](https://docs.python.org/3/library/2to3.html)
    ```bash
    pip install future
    ```
- We can use below command to auto-convert the code to Python 3
    ```bash
    futurize --stage1 mypackage/**/*.py
    ```
- Once the code is auto-converted, run the test cases and check the number of failures and start fixing them
- Once the test cases are fixed, we can proceed with our functional testing, regression testing and CI pipeline upgrades

Note - While fixing the functional test cases, it will be handy to have both Python 2 & Python 3 version of apps running in two diff virtual envs, because then we can go back n forth between two versions and see the behaviour change if we are stuck with fixing a bug


## Few Gotchas

As it's mentioned in futurize library's documentation that its auto conversion output may not be fully accurate and it has some issues and bugs as well, so it is important for us to review the automated fixes before merging the code, so from our experience few things can be taken care:

- Auto-conversion will end up adding list inbuilt function to all such cases where we might be using ```dict.iteritems()```, after the conversion it becomes ``` list(dict.items())``` because ```iteritems``` returns a `list` in Python 2.7 and in Python 3 it returns ```<class 'dict_items'>```. However, It is an unnecessary call because conversion to list takes time and you can still iterate through ```dict.items()``` the same way you do over the converted list
- Futurize can also add a lot of unused imports to your codebase, we can get rid of them if we are not worried about backward compatibility, but be mindful of not removing all unused import and do check and analyze the module before doing it
- If you might have used zip function in Python 2.7 and then tried converting it to dict then futurize might wrap `zip` with a `list` function call, you won't need that `list()` function call in most cases after migrating to Python 3

```bash 
all_vals = dict(list(zip(array1, array2))) # Unnecessary list function call
all_vals = dict(zip(array1, array2)) # Correct
```
- Relative imports are no longer supported in Python 3. Any code that used relative imports in Python 2 now has to be modified to support absolute imports
- In Python2, `CSV` files should be opened in binary mode. but, in Python 3, the files are to be opened in the text mode

## Common Error Fixes

So during the migration, we may end up fixing some very basic level errors which may get introduced because of futurize or because of the Python 2 & 3 differences. So I'm listing few of them which we faced and that might help you to quickly get rid of those -

- The old division would not work in Python3, and we can fix such cases like this 
```bash 
from past.utils import old_div
x = y / 1000 # Would give unexpected result in Python 3 and existing test case may fail because of this
x = old_div(y, 1000) # This would result as Python 2
```
- In all exception handling cases, we might end up getting this common error, wherein Python 2 we had an attribute called message for the exception class and python 3 doesn't have it

```bash
# Old case
except Exception as e:
    raise CustomError(message="message: {}".format(e.message))

# How to fix
except Exception as e:
    raise CustomError(message="message: {}".format(e))
```
- For the same exception handling errors when we remove message attribute from the ```e``` variable than in some case we might face below error where we will do concat operation on ```e``` and some string, that will cause a failure 
```bash
# Failure case
try:
    1/0
except Exception as e:
    print("custom_message"+e) # This would give an error
# How to fix
try:
    1/0
except Exception as e:
    print("custom_message"+str(e))
```
- `hasattr(dict, 'iteritems')` would return `False` in Python 3, to fix please use this
```bash
hasattr(dict,'items’) 
```
- In Python 2 we used to use ```string.replace()``` as well in somecases to replace string within a string, that will not work in Python 3, hence fix it like this -
```bash
a = "asdfghj"
string.replace(a, "asd", "xcv") # Will not work
a.replace("asd", "xcv") # correct
```
- `cmp()` function which used to be available in Python 2 is not available in Python 3, so you might need to implement something like this to fix the errors - 
```bash
def cmp(a, b):
    return (a > b) - (a < b) 
```

## Conclusion
I hope this information was helpful for you and I also hope that this may expedite your migration time. Overall, I must also confess that even though migrations steps seems kind of pretty straight forward but the whole process can be tiring because of fixing some common errors again and again. If you are able to fix these small errors quickly then you can focus on the functional part of the migration which is most important. So which is why I tried to list some of the common issues I faced during the migration, hope it can be useful for someone. Thanks for reading and feel free to share any feedback in the comment section.

#### References:
https://wiki.openstack.org/wiki/Python3#Port_Python_2_code_to_Python_3
http://python3porting.com/
https://docs.python.org/dev/howto/pyporting.html
https://python-future.org/quickstart.html
https://python-future.org/quickstart.html#to-convert-existing-python-2-code (On python 2 compatibility)

Disclaimer: This is a personal [blog, post, statement, opinion]. The views and opinions expressed here are only those of the author and do not represent those of any organization or any individual with whom the author may be associated, professionally or personally.