Upgrading Django app to Python 3.10
At my day job we have a Django app with almost 500_000 lines of Python code that was written over the last decade. Ever since we migrated to Python 3.7 (almost seven months after Python 2 EOL 😓), we’ve tried to keep on top on Python upgrades. We migrated to 3.9 at the end of last year, skipping 3.8 completely. It was now time for us to jump on Python 3.10.
Installing Python 3.10
We get the latest and greatest versions of Python through the excellent deadsnakes PPA. Python 3.10 installs without a problem on the system, but pip is broken with Python 3.10 and Ubuntu 18.04:
$ pip install --upgrade pip
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 187, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/usr/lib/python3.10/runpy.py", line 146, in _get_module_details
return _get_module_details(pkg_main_name, error)
File "/usr/lib/python3.10/runpy.py", line 110, in _get_module_details
__import__(pkg_name)
File "/usr/lib/python3/dist-packages/pip/__init__.py", line 22, in <module>
from pip._vendor.requests.packages.urllib3.exceptions import DependencyWarning
File "/usr/lib/python3/dist-packages/pip/_vendor/__init__.py", line 73, in <module>
vendored("pkg_resources")
File "/usr/lib/python3/dist-packages/pip/_vendor/__init__.py", line 33, in vendored
__import__(modulename, globals(), locals(), level=0)
File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/__init__.py", line 77, in <module>
File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/packaging/requirements.py", line 9, in <module>
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 672, in _load_unlocked
File "<frozen importlib._bootstrap>", line 632, in _load_backward_compatible
File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/extern/__init__.py", line 43, in load_module
File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/pyparsing.py", line 943, in <module>
AttributeError: module 'collections' has no attribute 'MutableMapping'
Luckily a quick Google search found this GitHub thread with a solution:
curl -Ss https://bootstrap.pypa.io/get-pip.py | python3.10
Installing all dependencies
We use 290 of Python packages and they all installed successfully with Python 3.10 on first try 🎉
Get runserver to start
Now that we have Python and all the dependencies installed we can finally see if we can run our Django application. Unfortunately, running python manage.py runserver immediately failed with the following error:
Traceback (most recent call last):
File "/app/manage.py", line 22, in <module>
execute_from_command_line(sys.argv)
File "/home/user/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
utility.execute()
File "/home/user/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 363, in execute
settings.INSTALLED_APPS
File "/home/user/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 82, in __getattr__
self._setup(name)
File "/home/user/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 69, in _setup
self._wrapped = Settings(settings_module)
File "/home/user/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 170, in __init__
mod = importlib.import_module(self.SETTINGS_MODULE)
File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/app/__init__.py", line 3, in <module>
from client.celery import app as celery_app
File "/app/celery.py", line 13, in <module>
from sentry_sdk import configure_scope
File "/home/user/.local/lib/python3.10/site-packages/sentry_sdk/__init__.py", line 1, in <module>
from sentry_sdk.hub import Hub, init
File "/home/user/.local/lib/python3.10/site-packages/sentry_sdk/hub.py", line 8, in <module>
from sentry_sdk.scope import Scope
File "/home/user/.local/lib/python3.10/site-packages/sentry_sdk/scope.py", line 7, in <module>
from sentry_sdk.utils import logger, capture_internal_exceptions
File "/home/user/.local/lib/python3.10/site-packages/sentry_sdk/utils.py", line 870, in <module>
HAS_REAL_CONTEXTVARS, ContextVar = _get_contextvars()
File "/home/user/.local/lib/python3.10/site-packages/sentry_sdk/utils.py", line 840, in _get_contextvars
if not _is_contextvars_broken():
File "/home/user/.local/lib/python3.10/site-packages/sentry_sdk/utils.py", line 801, in _is_contextvars_broken
from eventlet.patcher import is_monkey_patched # type: ignore
File "/home/user/.local/lib/python3.10/site-packages/eventlet/__init__.py", line 17, in <module>
from eventlet import convenience
File "/home/user/.local/lib/python3.10/site-packages/eventlet/convenience.py", line 7, in <module>
from eventlet.green import socket
File "/home/user/.local/lib/python3.10/site-packages/eventlet/green/socket.py", line 4, in <module>
__import__('eventlet.green._socket_nodns')
File "/home/user/.local/lib/python3.10/site-packages/eventlet/green/_socket_nodns.py", line 11, in <module>
from eventlet import greenio
File "/home/user/.local/lib/python3.10/site-packages/eventlet/greenio/__init__.py", line 3, in <module>
from eventlet.greenio.base import * # noqa
File "/home/user/.local/lib/python3.10/site-packages/eventlet/greenio/base.py", line 32, in <module>
socket_timeout = eventlet.timeout.wrap_is_timeout(socket.timeout)
File "/home/user/.local/lib/python3.10/site-packages/eventlet/timeout.py", line 166, in wrap_is_timeout
base.is_timeout = property(lambda _: True)
TypeError: cannot set 'is_timeout' attribute of immutable type 'TimeoutError'
Looks like we were using an out of date version of eventlet and upgrading that fixed the issue. Once it was upgraded the error went away, but only to give way to the next one:
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/home/user/.local/lib/python3.10/site-packages/sentry_sdk/integrations/threading.py", line 69, in run
reraise(*_capture_exception())
File "/home/user/.local/lib/python3.10/site-packages/sentry_sdk/_compat.py", line 54, in reraise
raise value
File "/home/user/.local/lib/python3.10/site-packages/sentry_sdk/integrations/threading.py", line 67, in run
return old_run_func(self, *a, **kw)
File "/usr/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/home/user/.local/lib/python3.10/site-packages/django/utils/autoreload.py", line 64, in wrapper
fn(*args, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/django/core/management/commands/runserver.py", line 118, in inner_run
self.check(display_num_errors=True)
File "/home/user/.local/lib/python3.10/site-packages/django/core/management/base.py", line 419, in check
all_issues = checks.run_checks(
File "/home/user/.local/lib/python3.10/site-packages/django/core/checks/registry.py", line 76, in run_checks
new_errors = check(app_configs=app_configs, databases=databases)
File "/home/user/.local/lib/python3.10/site-packages/django/core/checks/urls.py", line 13, in check_url_config
return check_resolver(resolver)
File "/home/user/.local/lib/python3.10/site-packages/django/core/checks/urls.py", line 23, in check_resolver
return check_method()
File "/home/user/.local/lib/python3.10/site-packages/django/urls/resolvers.py", line 416, in check
for pattern in self.url_patterns:
File "/home/user/.local/lib/python3.10/site-packages/django/utils/functional.py", line 48, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/home/user/.local/lib/python3.10/site-packages/django/urls/resolvers.py", line 602, in url_patterns
patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
File "/home/user/.local/lib/python3.10/site-packages/django/utils/functional.py", line 48, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/home/user/.local/lib/python3.10/site-packages/django/urls/resolvers.py", line 595, in urlconf_module
return import_module(self.urlconf_name)
File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/app/urls.py", line 23, in <module>
from client.apps.superuser.views import queue_length
File "/app/apps/superuser/views.py", line 53, in <module>
from client.libs.integrations.locations_calendar_events.importer import (
File "/app/libs/integrations/locations_calendar_events/importer.py", line 28, in <module>
from client.libs.integrations.rbi.utils.common import (
File "/app/libs/integrations/rbi/utils/common.py", line 18, in <module>
from client.libs.integrations.rbi.parser import (
File "/app/libs/integrations/rbi/parser.py", line 9, in <module>
import agate
File "/home/user/.local/lib/python3.10/site-packages/agate/__init__.py", line 5, in <module>
from agate.aggregations import *
File "/home/user/.local/lib/python3.10/site-packages/agate/aggregations/__init__.py", line 22, in <module>
from agate.aggregations.count import Count # noqa
File "/home/user/.local/lib/python3.10/site-packages/agate/aggregations/count.py", line 5, in <module>
from agate.utils import default
File "/home/user/.local/lib/python3.10/site-packages/agate/utils.py", line 9, in <module>
from collections import OrderedDict, Sequence
ImportError: cannot import name 'Sequence' from 'collections' (/usr/lib/python3.10/collections/__init__.py)
agate package seems to be to blame this time. Upgrade and 🎉 we have runserver up and running!
Fixing the warnings on startup
Runserver now starts and we can already start interacting whith the app, but we have a huge amount of warnings on startup:
[vagrant@bfdf66b9faee /vagrant]$ python manage.py runserver
2022-08-12 12:07:42,662 client.apps.deployment_flags.singleton - Initializing feature flag service EXTRA => flag_service_provider: `clientServiceProvider`
2022-08-12 12:07:46,815 client.apps.deployment_flags.singleton - Initializing feature flag service EXTRA => flag_service_provider: `clientServiceProvider`
2022-08-12 12:07:46,870 django.utils.autoreload - Watching for file changes with StatReloader
Performing system checks...
<frozen importlib._bootstrap>:914: ImportWarning: ImportHookFinder.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: ImportHookFinder.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: ImportHookFinder.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: ImportHookFinder.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: ImportHookFinder.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: ImportHookFinder.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: ImportHookFinder.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: ImportHookFinder.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: ImportHookFinder.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: ImportHookFinder.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: ImportHookFinder.find_spec() not found; falling back to find_module()
There were screens and screens full of these warnings, but there only two different types of messages:
<frozen importlib._bootstrap>:914: ImportWarning: ImportHookFinder.find_spec() not found; falling back to find_module()
This one was easy to solve, a quick Google search revelead that this is a warning caused by the newrelic package. Upgrading to the latest version solved it 🎉
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
This one seems to be caused by an out of date six package. What makes matters worse is that certain packages (I’m looking at you botocore 👀) vendor their own six version so finding all the packages to update can be a bit annoying. I wrote a separate blog post that shows how to find them all.
Now the Django dev server starts without errors:
[vagrant@bfdf66b9faee /vagrant]$ python manage.py runserver
2022-08-12 12:17:28,363 django.utils.autoreload - Watching for file changes with StatReloader
Performing system checks...
System check identified no issues (13 silenced).
August 12, 2022 - 12:17:31 PM
Django version 3.2.15, using settings 'client.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
Getting all the tests to pass
Since clicking around the app was now working, it was time to run all the 6k+ tests that we have. First run and the results don’t seem that bad:
47 tests failed out of 6185
I remember the upgrade from 2.7 to 3.7 where we had over 4k failing tests 😅
It’s even better than it seems because 45 of these tests came from the us states package:
Traceback (most recent call last):
File "/root/client/client/libs/integrations/tests/rbi/test_importer.py", line 4970, in test_delete_an_ingredient_location_default_unit_of_measure
test_import_locations(self.location_data, self.managed_company)
File "/root/client/client/libs/integrations/tests/rbi/test_importer.py", line 242, in test_import_locations
location_success, location_error = _import_test_location(
File "/root/client/client/libs/integrations/tests/rbi/test_importer.py", line 205, in _import_test_location
"state": get_us_state_abbreviation(
File "/root/client/client/libs/integrations/rbi/utils/common.py", line 217, in get_us_state_abbreviation
state_ = us.states.lookup(state)
File "/usr/local/lib/python3.10/dist-packages/us/states.py", line 86, in lookup
val = jellyfish.metaphone(val)
TypeError: str argument expected
The package has a pull request open with a fix, but unfortunately the maintainer is not responsive so a new version wasn’t released yet. Really frustrating when this happens, but since we didn’t really need the package that much we just decided to remove it. Less dependencies is always better!
The other two test failures were:
Traceback (most recent call last):
File "/root/client/client/libs/tests/tests_dataclass_utils.py", line 96, in test_bad_attribute_in_nested_data
self.assertEqual(
AssertionError: "Coul[25 chars]ist: ParentList.__init__() got an unexpected k[17 chars]foo'" != "Coul[25 chars]ist: __init__() got an unexpected keyword argument 'foo'"
- Could not instantiate ParentList: ParentList.__init__() got an unexpected keyword argument 'foo'
? -----------
+ Could not instantiate ParentList: __init__() got an unexpected keyword argument 'foo'
and
Traceback (most recent call last):
File "/root/client/client/libs/integrations/tests/rbi/datastructures/tests_attributes.py", line 85, in test_wrong_keys_raises_type_error
self.assertEqual(str(exc.exception), expected_message)
AssertionError: "Could not instantiate NameTranslation: NameTranslation.__init__() got an unexpected keyword argument 'wrong_key'" != "Could not instantiate NameTranslation: __init__() got an unexpected keyword argument 'wrong_key'"
- Could not instantiate NameTranslation: NameTranslation.__init__() got an unexpected keyword argument 'wrong_key'
? ----------------
+ Could not instantiate NameTranslation: __init__() got an unexpected keyword argument 'wrong_key'
We were asserting the exception message string and the message has changed slightly in Python 3.10. I’ve decided to just fix the asserts.
Fin
That was it! There were a few gotchas, but not something that caused a huge amount of work and all the goodies in the new Python version are definitively worth it (woohooo match statement!). Now hopefully we don’t break anything when pushing this to prod. 🤞