Anže's Blog

Python, Django, and the Web

11 Aug 2022

Upgrading Django app to Python 3.10

At my day job we have a Django app with almost 500_000 lines of Python code that was written over the last decade. Ever since we migrated to Python 3.7 (almost seven months after Python 2 EOL 😓), we’ve tried to keep on top on Python upgrades. We migrated to 3.9 at the end of last year, skipping 3.8 completely. It was now time for us to jump on Python 3.10.

Installing Python 3.10

We get the latest and greatest versions of Python through the excellent deadsnakes PPA. Python 3.10 installs without a problem on the system, but pip is broken with Python 3.10 and Ubuntu 18.04:

$ pip install --upgrade pip
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 187, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/lib/python3.10/runpy.py", line 146, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/usr/lib/python3.10/runpy.py", line 110, in _get_module_details
    __import__(pkg_name)
  File "/usr/lib/python3/dist-packages/pip/__init__.py", line 22, in <module>
    from pip._vendor.requests.packages.urllib3.exceptions import DependencyWarning
  File "/usr/lib/python3/dist-packages/pip/_vendor/__init__.py", line 73, in <module>
    vendored("pkg_resources")
  File "/usr/lib/python3/dist-packages/pip/_vendor/__init__.py", line 33, in vendored
    __import__(modulename, globals(), locals(), level=0)
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/__init__.py", line 77, in <module>
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/packaging/requirements.py", line 9, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 672, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 632, in _load_backward_compatible
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/extern/__init__.py", line 43, in load_module
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/pyparsing.py", line 943, in <module>
AttributeError: module 'collections' has no attribute 'MutableMapping'

Luckily a quick Google search found this GitHub thread with a solution:

curl -Ss https://bootstrap.pypa.io/get-pip.py | python3.10

Installing all dependencies

We use 290 of Python packages and they all installed successfully with Python 3.10 on first try 🎉

Get runserver to start

Now that we have Python and all the dependencies installed we can finally see if we can run our Django application. Unfortunately, running python manage.py runserver immediately failed with the following error:

Traceback (most recent call last):
  File "/app/manage.py", line 22, in <module>
    execute_from_command_line(sys.argv)
  File "/home/user/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/home/user/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 363, in execute
    settings.INSTALLED_APPS
  File "/home/user/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 82, in __getattr__
    self._setup(name)
  File "/home/user/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 69, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/user/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 170, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/app/__init__.py", line 3, in <module>
    from client.celery import app as celery_app
  File "/app/celery.py", line 13, in <module>
    from sentry_sdk import configure_scope
  File "/home/user/.local/lib/python3.10/site-packages/sentry_sdk/__init__.py", line 1, in <module>
    from sentry_sdk.hub import Hub, init
  File "/home/user/.local/lib/python3.10/site-packages/sentry_sdk/hub.py", line 8, in <module>
    from sentry_sdk.scope import Scope
  File "/home/user/.local/lib/python3.10/site-packages/sentry_sdk/scope.py", line 7, in <module>
    from sentry_sdk.utils import logger, capture_internal_exceptions
  File "/home/user/.local/lib/python3.10/site-packages/sentry_sdk/utils.py", line 870, in <module>
    HAS_REAL_CONTEXTVARS, ContextVar = _get_contextvars()
  File "/home/user/.local/lib/python3.10/site-packages/sentry_sdk/utils.py", line 840, in _get_contextvars
    if not _is_contextvars_broken():
  File "/home/user/.local/lib/python3.10/site-packages/sentry_sdk/utils.py", line 801, in _is_contextvars_broken
    from eventlet.patcher import is_monkey_patched  # type: ignore
  File "/home/user/.local/lib/python3.10/site-packages/eventlet/__init__.py", line 17, in <module>
    from eventlet import convenience
  File "/home/user/.local/lib/python3.10/site-packages/eventlet/convenience.py", line 7, in <module>
    from eventlet.green import socket
  File "/home/user/.local/lib/python3.10/site-packages/eventlet/green/socket.py", line 4, in <module>
    __import__('eventlet.green._socket_nodns')
  File "/home/user/.local/lib/python3.10/site-packages/eventlet/green/_socket_nodns.py", line 11, in <module>
    from eventlet import greenio
  File "/home/user/.local/lib/python3.10/site-packages/eventlet/greenio/__init__.py", line 3, in <module>
    from eventlet.greenio.base import *  # noqa
  File "/home/user/.local/lib/python3.10/site-packages/eventlet/greenio/base.py", line 32, in <module>
    socket_timeout = eventlet.timeout.wrap_is_timeout(socket.timeout)
  File "/home/user/.local/lib/python3.10/site-packages/eventlet/timeout.py", line 166, in wrap_is_timeout
    base.is_timeout = property(lambda _: True)
TypeError: cannot set 'is_timeout' attribute of immutable type 'TimeoutError'

Looks like we were using an out of date version of eventlet and upgrading that fixed the issue. Once it was upgraded the error went away, but only to give way to the next one:

Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/home/user/.local/lib/python3.10/site-packages/sentry_sdk/integrations/threading.py", line 69, in run
    reraise(*_capture_exception())
  File "/home/user/.local/lib/python3.10/site-packages/sentry_sdk/_compat.py", line 54, in reraise
    raise value
  File "/home/user/.local/lib/python3.10/site-packages/sentry_sdk/integrations/threading.py", line 67, in run
    return old_run_func(self, *a, **kw)
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/django/utils/autoreload.py", line 64, in wrapper
    fn(*args, **kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/django/core/management/commands/runserver.py", line 118, in inner_run
    self.check(display_num_errors=True)
  File "/home/user/.local/lib/python3.10/site-packages/django/core/management/base.py", line 419, in check
    all_issues = checks.run_checks(
  File "/home/user/.local/lib/python3.10/site-packages/django/core/checks/registry.py", line 76, in run_checks
    new_errors = check(app_configs=app_configs, databases=databases)
  File "/home/user/.local/lib/python3.10/site-packages/django/core/checks/urls.py", line 13, in check_url_config
    return check_resolver(resolver)
  File "/home/user/.local/lib/python3.10/site-packages/django/core/checks/urls.py", line 23, in check_resolver
    return check_method()
  File "/home/user/.local/lib/python3.10/site-packages/django/urls/resolvers.py", line 416, in check
    for pattern in self.url_patterns:
  File "/home/user/.local/lib/python3.10/site-packages/django/utils/functional.py", line 48, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/home/user/.local/lib/python3.10/site-packages/django/urls/resolvers.py", line 602, in url_patterns
    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
  File "/home/user/.local/lib/python3.10/site-packages/django/utils/functional.py", line 48, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/home/user/.local/lib/python3.10/site-packages/django/urls/resolvers.py", line 595, in urlconf_module
    return import_module(self.urlconf_name)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/app/urls.py", line 23, in <module>
    from client.apps.superuser.views import queue_length
  File "/app/apps/superuser/views.py", line 53, in <module>
    from client.libs.integrations.locations_calendar_events.importer import (
  File "/app/libs/integrations/locations_calendar_events/importer.py", line 28, in <module>
    from client.libs.integrations.rbi.utils.common import (
  File "/app/libs/integrations/rbi/utils/common.py", line 18, in <module>
    from client.libs.integrations.rbi.parser import (
  File "/app/libs/integrations/rbi/parser.py", line 9, in <module>
    import agate
  File "/home/user/.local/lib/python3.10/site-packages/agate/__init__.py", line 5, in <module>
    from agate.aggregations import *
  File "/home/user/.local/lib/python3.10/site-packages/agate/aggregations/__init__.py", line 22, in <module>
    from agate.aggregations.count import Count  # noqa
  File "/home/user/.local/lib/python3.10/site-packages/agate/aggregations/count.py", line 5, in <module>
    from agate.utils import default
  File "/home/user/.local/lib/python3.10/site-packages/agate/utils.py", line 9, in <module>
    from collections import OrderedDict, Sequence
ImportError: cannot import name 'Sequence' from 'collections' (/usr/lib/python3.10/collections/__init__.py)

agate package seems to be to blame this time. Upgrade and 🎉 we have runserver up and running!

Fixing the warnings on startup

Runserver now starts and we can already start interacting whith the app, but we have a huge amount of warnings on startup:

[vagrant@bfdf66b9faee /vagrant]$ python manage.py runserver
2022-08-12 12:07:42,662 client.apps.deployment_flags.singleton - Initializing feature flag service EXTRA => flag_service_provider: `clientServiceProvider`
2022-08-12 12:07:46,815 client.apps.deployment_flags.singleton - Initializing feature flag service EXTRA => flag_service_provider: `clientServiceProvider`
2022-08-12 12:07:46,870 django.utils.autoreload - Watching for file changes with StatReloader
Performing system checks...

<frozen importlib._bootstrap>:914: ImportWarning: ImportHookFinder.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: ImportHookFinder.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: ImportHookFinder.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: ImportHookFinder.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: ImportHookFinder.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: ImportHookFinder.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: ImportHookFinder.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: ImportHookFinder.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: ImportHookFinder.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: ImportHookFinder.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:914: ImportWarning: ImportHookFinder.find_spec() not found; falling back to find_module()

There were screens and screens full of these warnings, but there only two different types of messages:

<frozen importlib._bootstrap>:914: ImportWarning: ImportHookFinder.find_spec() not found; falling back to find_module()

This one was easy to solve, a quick Google search revelead that this is a warning caused by the newrelic package. Upgrading to the latest version solved it 🎉

<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()

This one seems to be caused by an out of date six package. What makes matters worse is that certain packages (I’m looking at you botocore 👀) vendor their own six version so finding all the packages to update can be a bit annoying. I wrote a separate blog post that shows how to find them all.

Now the Django dev server starts without errors:

[vagrant@bfdf66b9faee /vagrant]$ python manage.py runserver
2022-08-12 12:17:28,363 django.utils.autoreload - Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (13 silenced).

August 12, 2022 - 12:17:31 PM
Django version 3.2.15, using settings 'client.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

Getting all the tests to pass

Since clicking around the app was now working, it was time to run all the 6k+ tests that we have. First run and the results don’t seem that bad:

47 tests failed out of 6185

I remember the upgrade from 2.7 to 3.7 where we had over 4k failing tests 😅

It’s even better than it seems because 45 of these tests came from the us states package:

Traceback (most recent call last):
  File "/root/client/client/libs/integrations/tests/rbi/test_importer.py", line 4970, in test_delete_an_ingredient_location_default_unit_of_measure
    test_import_locations(self.location_data, self.managed_company)
  File "/root/client/client/libs/integrations/tests/rbi/test_importer.py", line 242, in test_import_locations
    location_success, location_error = _import_test_location(
  File "/root/client/client/libs/integrations/tests/rbi/test_importer.py", line 205, in _import_test_location
    "state": get_us_state_abbreviation(
  File "/root/client/client/libs/integrations/rbi/utils/common.py", line 217, in get_us_state_abbreviation
    state_ = us.states.lookup(state)
  File "/usr/local/lib/python3.10/dist-packages/us/states.py", line 86, in lookup
    val = jellyfish.metaphone(val)
TypeError: str argument expected

The package has a pull request open with a fix, but unfortunately the maintainer is not responsive so a new version wasn’t released yet. Really frustrating when this happens, but since we didn’t really need the package that much we just decided to remove it. Less dependencies is always better!

The other two test failures were:

Traceback (most recent call last):
  File "/root/client/client/libs/tests/tests_dataclass_utils.py", line 96, in test_bad_attribute_in_nested_data
    self.assertEqual(
AssertionError: "Coul[25 chars]ist: ParentList.__init__() got an unexpected k[17 chars]foo'" != "Coul[25 chars]ist: __init__() got an unexpected keyword argument 'foo'"
- Could not instantiate ParentList: ParentList.__init__() got an unexpected keyword argument 'foo'
?                                   -----------
+ Could not instantiate ParentList: __init__() got an unexpected keyword argument 'foo'

and

Traceback (most recent call last):
  File "/root/client/client/libs/integrations/tests/rbi/datastructures/tests_attributes.py", line 85, in test_wrong_keys_raises_type_error
    self.assertEqual(str(exc.exception), expected_message)
AssertionError: "Could not instantiate NameTranslation: NameTranslation.__init__() got an unexpected keyword argument 'wrong_key'" != "Could not instantiate NameTranslation: __init__() got an unexpected keyword argument 'wrong_key'"
- Could not instantiate NameTranslation: NameTranslation.__init__() got an unexpected keyword argument 'wrong_key'
?                                        ----------------
+ Could not instantiate NameTranslation: __init__() got an unexpected keyword argument 'wrong_key'

We were asserting the exception message string and the message has changed slightly in Python 3.10. I’ve decided to just fix the asserts.

Fin

That was it! There were a few gotchas, but not something that caused a huge amount of work and all the goodies in the new Python version are definitively worth it (woohooo match statement!). Now hopefully we don’t break anything when pushing this to prod. 🤞