How It Works

slothy can feel like it's magic, but the trick is super simple.

It all started with a dictionary

Python scopes are essentially dictionaries.
Most of the times they can be accessed via locals() and globals().

Define a variable fooSee foo in locals()See foo in globals()

>>> foo = 1

>>> foo = 1
>>> locals()
{'__annotations__': {},
'__builtins__': <module 'builtins' (built-in)>,
'__doc__': None,
'__loader__': <class '_frozen_importlib.BuiltinImporter'>,
'__name__': '__main__',
'__package__': None,
'__spec__': None,
'foo': 1}

Note

This includes a bunch of things Python automatically assigns to the module-level variables to work correctly. Unsurprisingly, they are dunders and we typically don't need to know they are there.

>>> foo = 1
>>> locals()
{'__annotations__': {},
'__builtins__': <module 'builtins' (built-in)>,
'__doc__': None,
'__loader__': <class '_frozen_importlib.BuiltinImporter'>,
'__name__': '__main__',
'__package__': None,
'__spec__': None,
'foo': 1}
>>> # In this case it's the same as locals(),
>>> # because we're at the module level.
>>> globals()
{'__annotations__': {},
'__builtins__': <module 'builtins' (built-in)>,
'__doc__': None,
'__loader__': <class '_frozen_importlib.BuiltinImporter'>,
'__name__': '__main__',
'__package__': None,
'__spec__': None,
'foo': 1}

---

At the module level (outside any functions/classes), locals() and globals() are the same dictionary.

We can reference a global variable from inside a module-level function, and that will first look up the locals() dictionary from there and then globals() without us having to do it manually:

Call a functionLook up xResolve xReturn x

Given the code below, we call our func first.

x = "whatever"

def func():
    return x

print(func())

Once func is called, x is requested:

x = "whatever"

def func():
    return x

print(func())

Python looks here (locals()):

x = "whatever"

def func():
    return x

print(func())

Then here (globals()):

x = "whatever"

def func():
    return x

print(func())

and... >>>

...finally sees it here, in globals():

x = "whatever"

def func():
    return x

print(func())

Now x can be returned.

x = "whatever"

def func():
    return x  # found it!

print(func())  # outputs 'whatever'

---

But globals() isn't where the lookup ends: as you might have already noticed, we often use names that aren't inside globals() nor any subsequent locals(). That is, why does globals work in the first place?!

>>> globals
<built-in function globals>
>>> "globals" in globals()
False
>>> # How on earth is that possible?

Well, there's another scope that they probably don't want you to know about.
It's called the built-in scope and it correlates with the builtins module.

What is interesting, those built-ins are typically mounted through the __builtins__ variable that you could have noticed before:

>>> foo = 1
>>> locals()
{'__annotations__': {},
'__builtins__': <module 'builtins' (built-in)>,
'__doc__': None,
'__loader__': <class '_frozen_importlib.BuiltinImporter'>,
'__name__': '__main__',
'__package__': None,
'__spec__': None,
'foo': 1}

In CPython, changing the value of __builtins__ can change the actual built-in scope entirely for all subsequent frames, i.e. the entire call stack starting from the moment you override the __builtins__ variable.

This is what slothy initially relied on: it captured undefined names (since built-ins is the last scope checked) from a program and then looked it up in "lazy import declarations" to import them on demand.

However, that didn't work outside CPython, because it's not in the standard.

So, what slothy does instead is...

Intercepting dictionary access

From the documentation on mappings:

A mapping object maps hashable values to arbitrary objects. Mappings are mutable objects. There is currently only one standard mapping type, the dictionary. (...) A dictionary’s keys are almost arbitrary values. Values that are not hashable (...) may not be used as keys. Values that compare equal (such as 1, 1.0, and True) can be used interchangeably to index the same dictionary entry.

From the documentation on hashable objects:

An object is hashable if it has a hash value which never changes during its lifetime (it needs a __hash__() method), and can be compared to other objects (it needs an __eq__() method). Hashable objects which compare equal must have the same hash value.

Which makes sense with regard to the previous quote:

>>> hash(1)
1
>>> 1 == 1.0
True
>>> hash(1.0)
1
>>> 1 == True
True
>>> hash(True)
1

Hashability makes an object usable as a dictionary key and a set member, because these data structures use the hash value internally.

Knowing that, let's see what happens if we try to insert a custom key to a dictionary.
The new, "special" key will:

• log every __hash__ and __eq__ call,
• be equal to every string of value "special".

>>> class SpecialKey:
...     def __eq__(self, other):
...         print("__eq__ called")
...         return "special" == other
...     def __hash__(self):
...         print("__hash__ called")
...         return hash("special")

And we are good to go!

Create a dictionaryInsert the special keyCheck against invalid keyCheck against "special"Check against a colliding key

Let's create the dictionary we will experiment on.

>>> # Create a dictionary.
>>> d = {}

Let's insert the key.

>>> # Create a dictionary.
>>> d = {}
>>>
>>> # Insert out special key that equals "special".
>>> d[SpecialKey()] = 1
__hash__ called

Notice that __hash__ was called.

Just to make sure, we will try to get a key that can't exist in that dictionary.

>>> # Create a dictionary.
>>> d = {}
>>>
>>> # Insert out special key that equals "special".
>>> d[SpecialKey()] = 1
__hash__ called
>>>
>>> # Try to get an inexisting key.
>>> d[object()]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: <object object at 0xc0ffee04bee4>

The reason for the dictionary to know this is an invalid key is that there is no value under a key with the same hash as that of the requested object().

Hash-based lookup is what makes dictionaries (and hash tables in general) so fast.

We intended the special key to be interoperable with a string "special". Let's try it then.

>>> # Create a dictionary.
>>> d = {}
>>>
>>> # Insert out special key that equals "special".
>>> d[SpecialKey()] = 1
__hash__ called
>>>
>>> # Try to get an inexisting key.
>>> d[object()]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: <object object at 0xc0ffee04bee4>
>>>
>>> d["special"]
__eq__ called
1

That works! Notice that __eq__ was called.

You might ask: why? If hashable objects which compare equal must have the same hash value, why would we have to additionally check their equality? >>>

Well, imagine an object that very randomly has the same hash as the string "special".

>>> # Create a dictionary.
>>> d = {}
>>>
>>> # Insert out special key that equals "special".
>>> d[SpecialKey()] = 1
__hash__ called
>>>
>>> # Try to get an inexisting key.
>>> d[object()]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: <object object at 0xc0ffee04bee4>
>>>
>>> d["special"]
__eq__ called
1
>>>
>>> class RandomlyCollidingKey:
...     colliding = True
...     def __hash__(self):
...         return hash("special")
...     def __eq__(self, other):
...         # All RandomlyCollidingKey objects have this attribute
...         # set to True.
...         return bool(getattr(other, "colliding", None))
...
>>> d[RandomlyCollidingKey()]
__eq__ called
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: <__main__.RandomlyCollidingKey object at 0xfeeddeadbeef>
>>> d[RandomlyCollidingKey()] = 2
__eq__ called
>>> d["special"]
__eq__ called
1
>>> d[RandomlyCollidingKey()]
__eq__ called
2

What __eq__ ultimately gives us is the possibility of maintaining distinct keys with colliding hashes in the same dictionary.

---

In conclusion, Python dictionaries let us map objects with possibly colliding hashes to values and retrieve them quickly, making sure we get (and set) what we intend.

Since arbitrary objects participate in the process of dictionary key resolution, slothy does its magic by modifying the first step to operate on a slightly different dictionary:

Our experimentslothy

>>> # Create a dictionary.
>>> d = {}

>>> # Create a dictionary.
>>> d = locals()

You get the point?

slothy essentially uses the "special key" idea and implements behavior that dynamically imports items on their first actual lookup inside your locals().

These keys are created on every import inside slothy context managers that temporarily patch __import__ in your built-in scope: the function without which the import statement wouldn't work.

This was How It Works. Congrats for making it this far! 🎉