Even a Feature That You Do Not Use Can Bite You

By Petr Zemek in Software Engineering 19 Comments

Let’s have a look at a simple piece of Python code that I have both accidentally written and seen in code reviews which does an entirely different thing than expected.

Pop Quiz

What does the following code do when being run in Python 3.6 or newer?

ages = {}
ages['John']: 42
print(ages['John'])

Options:

  1. Prints 42.
  2. Fails with SyntaxError on line 2.
  3. Fails with KeyError on line 3.

Correct Answer

The correct answer is number three.

Why?

The issue is on the following line:

ages['John']: 42

This statement doesn’t actually do anything (it is a no-op). What we have meant to write is this:

ages['John'] = 42

Hmm, fair enough. However, why doesn’t the original code raise SyntaxError? The reason is that it is actually valid Python code that uses variable annotations, which is a feature introduced in Python 3.6. In the previous versions of Python, the code would have raised SyntaxError.

In short, variable annotations are meant to allow programmers to provide so-called type hints. These hints can be then used by your IDE, documentation generator, or type checker, such as mypy. For example, the following line says that the type of variable x should be int:

x: int

Then, when you write

x = 'Hey there!'

a type checker may warn you that there is a type mismatch. As a side note, apart from variables, type hints may also be given when writing functions (function annotations):

def transmogrify(v: [int]) -> int:
    ...

Anyway, let’s get back to our original line:

ages['John']: 42

You may be wondering: “But the left-hand side is not a variable! And the right hand side is not a type!”. You are right, but that does not matter. According to the Annotating expressions section, the target of the annotation can be any valid assignment target. The section also gives the following example:

d = {}
d['b']: int  # Annotates d['b'] with int.

So, we can see that our original line

ages['John']: 42

says that we annotate ages['John'] with 42.

As for the fact that 42 is not a type, this also does not matter as the right-hand side of an annotation can be any valid Python expression, so you can even write

x: print(1 + 1)

This will print 2 when evaluating the code and annotate x with None (the result of print()).

Discussion

Apart from comments below, you can also discuss this post at /r/Python.

19 Comments

    • If I remember correctly, I accidentally wrote it when changing

      ages = {
    'John': 42
}

      to

      ages = {}
if condition:
    ages['John']: 42

      Then, I wondered why the dictionary did not get updated when condition was True…

      Reply

  2. I was looping in my DynamoDb results and building HTML.

    Reformatting, I bumped a loop in too deeply, and all heck broke loose in the output.

    Reply

  4. im definitely a little concerned that ages['John']: 42 doesn’t throw a syntax error, but im also intrigued by the idea of type hinting.

    we didn’t they just only allow a limited set of “options” after the colon? like, why not allow: x: int but not x: 42

    Reply

  6. I get what you want to say with this article, but isn’t this like complaining about this in c or cpp (the pitfall probably we all fell in at some point):

    #include <stdio.h>
int main(void) {
  int i = 0;
  if (i = 1) {
    printf("nay\n");
  } else {
    printf("yay\n");
  }
}

    with default configuration cc or gcc won’t even warn you about the assignment inside the if statement.
    In fact what happens is that it will assign the variable and then evaluate it (therefore in this example we will get “nay” as output, as i is 1 when evaluating it).

    You may not use this feature but it is a completely viable option.
    It’s not a syntax error, it’s just a semantic error, but that is up to you to decide.

    Reply

    • Thank you for the comment. I do not think that this is the same as in C or C++, both assignment and comparison have been in the language since their beginnings, and you cannot realistically opt-out of using them. In contrast, Python’s annotations are relatively new and are marketed as “you do not have to use them if you don’t want to”. Well, even if you do not want to use them, you may still get burned.

      Anyway, I did not intend to complain, even if the tone of the post may sound like I did. I just want to raise awareness about this potential pitfall and have a handy link I can give to people when they encounter this kind of issue.

      Reply

      • Ok, fair point. I wasn’t aware that type annotations are advertised in such a way.

        I dug a bit into that topic and found PEP 484 and PEP 526 which point out that evaluating the types is not a part of this module as python would still be a dynamically typed language

        i: int
i = 1
i = 'abc'

        This code would (as expected) not result in any errors. Type checking has to get handled by third party tools.
        With that in mind your post is of even more value, as they seem to have no intention in changing this behaviour at all (what is understandable if you are aware of the decision)

        That was bad phrasing on my side. I never saw your post as a complaint. I just saw the similarities to the complaint we all were going through at some point with the assignment in c (or similar) when we were starting to code.

        Reply

    • I do not think that the issue is in the same category. If type annotations are advertised not to change program behavior, at least line 2 in the original example should raise KeyError. But it doesn’t, so annotations in fact do change behavior.

      Here is what OCaml would do in the other posted example:

      # let launch () = Printf.printf “launch missile”; true;;
      val launch : unit -> bool =
      # [|1|].(0) == 1 || launch();;
      – : bool = true
      # ([|1|].(0) : int) == 1 || launch();;
      – : bool = true

      This is sane behavior. In no way should a type annotation change program semantics.

      I have been programming Python for quite some time and had not been aware of this issue (I’d call it a bug). So thanks Petr Zemek for pointing it out.

      Reply

      • Hmm, I think it’s a bit of a stretch to complain about a “change [of] program semantics” in a case that involves 

        KeyError

        , 

        SyntaxError

        , etc.

        For example, the following code will “break” if the 

        foo

        module ever gets a 

        bar

        function added:

        import foo

try:
  foo.bar
  import sys; sys.exit(1)
except:
  print('hello world')

        I think this article’s complaint of invalid syntax becoming valid is essentially the same thing, but at the language level rather than the library level.

        Reply

        • The fact that KeyError is involved is a red herring. The example where OCaml does the one and only right thing and Python fails miserably is a clearer demonstration of altering program semantics.

          It is simply a bug in Python.

          Reply

  7. I’m designing a programming language that borrows some syntax from Python. As a design principle, I avoid overloading the same syntax to mean unrelated things in different contexts.

    So, in my language, you can write

       ages = {"John": 42}

    But the ‘:’ syntax is now reserved for this purpose. As a result, type hints must use a different operator, so I chose to use ‘::’.

    I also avoid using different syntax to mean the same thing in different contexts. So I use exactly the same syntax for conditional statements and conditional expressions, which in my language happens to be:

        if (condition) consequent else alternate

    Python, for some reason, uses

        consequent if condition else alternate

    for conditional expressions, and

        if condition:
       consequent
    else
       alternate

    for conditional statements. The order of operands is different, and that inconsistency makes it harder to remember.

    The point I’m trying to make is that you can adopt a set of design principles for designing a programming language, which help you avoid creating inconsistencies and pitfalls for your users. If somebody is reading this post to learn about a pitfall in Python that can be avoided in future languages, then this is my approach for avoiding these kinds of pitfalls.

    Reply

Leave a Comment.