Recently, I have stumbled upon an apparently strange situation in which a non-empty Python package was empty after it got imported. Let’s take a close look at that issue because it may happen to anyone.

Our Package

The package is very simple:

foo/
└── __init__.py

It’s name (foo) does not collide with anything else in the system, and the __init__.py file contains just the following line:

x = 1

Let’s Import It

Let’s go into the directory in which foo is located and import it:

$ python3
>>> import foo

Great, the package was imported. Let’s try to access x:

>>> foo.x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'foo' has no attribute 'x'

Huh?

Investigation

That AttributeError is definitely unexpected. We know for sure that the foo package has attribute x, defined in its __init__.py file. Let’s see what does Python know about the module:

>>> print(foo)
<module 'foo' (namespace)>

Notice that (namespace) part. Instead, we would expect something like this:

>>> print(foo)
<module 'foo' from '/path/to/foo/__init__.py'>

What does namespace mean? To refresh your memory, Python 3 (more specifically, Python 3.3 or later) defines two types of packages: regular packages and namespace packages. Regular packages are the traditional ones that have existed in Python since its beginning. They are typically implemented as a directory containing an __init__.py file. On the other hand, namespace packages may have no physical representation on disk, and specifically have no __init__.py file. The rationale behind them is to allow developers to split sub-packages and modules within a single package across multiple, separate distribution packages. For example, you may have separate packages bar, bar-plugin-x, and bar-plugin-y that can be imported as follows:

import bar
import bar.plugins.x
import bar.plugins.y

Notice that the imports make it look like that everything is part of a single package (bar), but in reality, there are three separate namespace packages in play.

Why Is Foo a Namespace Package When There Is an __init__.py File?

Readers with a keen eye may have noticed that our foo package got marked as a namespace one even in the presence of an __init__.py file. Why is that? In my case, the user who was importing the package did not have sufficient permissions to access the contents of the foo directory. This can easily happen if you either install a package under user A and then run it under user B, or when the permissions on the directory or __init__.py file get messed up.

Why Has Python Raised AttributeError?

Because it was unable to read the __init__.py file, which contained the definition of x. The package was imported correctly, but it was pretty much empty.

Why Has Python Not Raised PermissionError?

Usually, when you try to import a module that you do not have sufficient permissions to, Python helpfully raises PermissionError:

>>> import baz
PermissionError: [Errno 13] Permission denied: '/path/to/baz.py'

Why has Python not raised PermissionError in our case, where the user who imports the package was unable to read the contents of the foo directory? To answer this question, we will need to take a look at the nitty-gritty details of Python’s import system.

Let’s start with a tip. If you run python3 with -v, its output will be more verbose. In our case, Python will print the following output when importing foo (notice the first line):

# possible namespace for /path/to/foo
import 'foo' # <_frozen_importlib_external._NamespaceLoader object at 0x7f949e7a4550>

Now, let’s take a look at how Python decides whether a package is regular or namespace. In Lib/importlib/_bootstrap_external.py, there is the following piece of code in FileFinder.find_spec():

1439   # Check if the module is the name of a directory (and thus a package).
1440   if cache_module in cache:
1441       base_path = _path_join(self.path, tail_module)
1442       for suffix, loader_class in self._loaders:
1443           init_filename = '__init__' + suffix
1444           full_path = _path_join(base_path, init_filename)
1445           if _path_isfile(full_path):             # <-----
1446               return self._get_spec(...)
1447       else:
...
1450           is_namespace = _path_isdir(base_path)   # <-----
...
1459   if is_namespace:
1460       _bootstrap._verbose_message('possible namespace for {}', base_path)

On line 1445, Python checks if path/to/foo/__init__.py is a file by calling _path_isfile() on it. This function is defined as follows:

def _path_isfile(path):
    return _path_is_mode_type(path, 0o100000)

def _path_is_mode_type(path, mode):
    try:
        stat_info = _path_stat(path)
    except OSError:                     # <-----
        return False
    return (stat_info.st_mode & 0o170000) == mode

Since we do not have permissions to read the contents of path/to/foo, _path_stat() raises PermissionError. However, PermissionError inherits from OSError, and so the exception gets swallowed by the except block. Thus, _path_is_mode_type() returns False, and so the path does not qualify for a regular package. Then, after the for loop on line 1442 exhausts all the possible __init__ suffixes (such as .py), it goes to the else clause on line 1447. In there, it checks if the package is a namespace one by calling _path_isdir() on path/to/foo, which simply checks if the path is a directory. It is, so it sets is_namespace to True, and the package is considered to be a namespace one. Then, on line 1460, the verbose message that we saw earlier gets printed.

Conclusion

When the contents of an imported package seem off, apart from checking that you have imported the correct package, also verify whether you have sufficient permissions, both to the package and its subdirectories and files.

Discussion

Apart from comments below, you can also discuss this article on /r/programming.