thoughts and perspective
by Luis Artola

This article explores a couple of many Python flaws that can let developers incur in very subtle bugs that can be difficult and expensive to uncover. One is the lack of scope for variable declarations. The other is the lack of compiler support to detect shadow declarations. This gives a brief insight into the eternal debate of highly-dynamic interpreted languages versus strongly-typed compiled languages.

The problem

While reviewing some piece of code written in Python, I came across a very interesting bug. The code goes something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
def f():
    # ...
    values = [ items[i:i+3] for i in xrange(10) ]
    # ...
    # ...
    for pkg, ver, os in values:
        # ...
        # ...
        for row in rows:
            # ...
            # ...
            if item[i] == values:
                dosomethingwith(row)

Did you see the problem? The if statement is testing item[i] instead of item[row], which is the actual index variable in the immediately enclosing for loop. Very easy to incur, especially when programming under
pressure or time constraints.

Variable i was declared in the list comprehension statement at the beginning of the function. However, because Python lacks variable scope, the programmer was able to use i much further below. Needless to say, the
application exhibited some strange behavior under certain circumstances.

The problem here is that the statement:

if item[i] == values:

is really equivalent to:

if item[9] == values:

because the list comprehension iterated from 0 to 9, i.e. xrange(10).

Lack of scope

There are many other things to criticize in this code, but this particular issue speaks to a much larger flaw in Python, the lack of scope.

Compiled languages provide consistent scope support for variable declarations, in particular local variables. In the example above, variable i was declared and used inside a list comprehension iteration.

Had Python supported local scope, the interpreter could have flagged the error in the if statement with a NameError exception. This could have saved the application from having erratic behavior and many hours of debugging, which are not cheap.

Strongly-typed compiled languages like C++ do provide support for scope. For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
#include <iostream>

int main( int argc, char* argv[] )
{
    // First declaration
    float value = 1.23f;
    std::cout << "value: " << value << ", type: " << typeid( value ).name() << std::endl;

    // Inner local scope
    for ( int i = 0; i < 2; i++ ) {
        long value = i;
        std::cout << "value: " << value << ", type: " << typeid( value ).name() << std::endl;
    }

    // Back to original scope
    std::cout << "value: " << value << ", type: " << typeid( value ).name() << std::endl;
    return 0;
}

Compiling and running this code:

g++ -o shadow shadow.cpp
./shadow

Produces this output:

value: 1.23, type: f
value: 0, type: l
value: 1, type: l
value: 1.23, type: f

Notice how the inner scope correctly handles the change of data type and intention for variable value.

The equivalent in Python looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Initial declaration
value = 1.23
print 'value: %s, type: %s' % ( value, type( value ).__name__ )

# Inner local scope
for i in xrange(2):
    value = i
    print 'value: %s, type: %s' % ( value, type( value ).__name__ )

# Back to original scope
print 'value: %s, type: %s' % ( value, type( value ).__name__ )

Running the code produces this output:

value: 1.23, type: float
value: 0, type: int
value: 1, type: int
value: 1, type: int

Notice how the last line points out how variable value has lost its type and original intention after leaving the inner scope of the for loop.

Shadow declarations

This phenomenon is known as shadow declarations. They are easy to encounter, especially by less experienced programmers and/or in code that has very long function bodies. It’s aggravated by the lack of compiler support to detect
them.

For example, in C++, a programmer can rely on the -Wshadow option to detect such problems ahead of time. In stricter development environments, -Werror can be used to prevent such code from even going through the linking phase.

Compiling the C++ example above with stricter options:

g++ -o shadow shadow.cpp -Wshadow -Werror

Produces this output:

cc1plus: warnings being treated as errors
shadow.cpp: In function ‘int main(int, char**)’:
shadow.cpp:11: warning: declaration of ‘value’ shadows a previous local
shadow.cpp:6: warning: shadowed declaration is here

And no executable is generated.

Conclusion

For some, the highly-dynamic nature of Python is something to praise and brag about. Sure, it has some benefits, to which I will dedicate other articles. However, when such features get in the way and let programmers incur in subtle bugs like this, it really speaks to the lack of more fundamental and better support either by the language itself or the tool set.

2010.07.25 | | |

1 Comment to “Python flaws: scope and shadows”