Debugging in Python

As a programmer, one of the first things that you need for serious program development is a debugger.

Python has a debugger, which is available as a module called pdb (for “Python DeBugger”, naturally!). Unfortunately, most discussions of pdb are not very useful to a Python newbie — most are very terse and simply rehash the description of pdb in the Python library reference manual. The discussion that I have found most accessible is in the first four pages of Chapter 27 of the Python 2.1 Bible.

So here is my own personal gentle introduction to using pdb. It assumes that you are not using any IDE — that you’re coding Python with a text editor and running your Python programs from the command line.

Some Other Debugger Resources

  • For information on the IDLE interactive debugger, see the IDLE documentation
  • For information on the Wing IDE debugger, see the Wing IDE documentation. There is a chapter on the Wing debugger, and another chapter on advanced debugging techniques which covers debugging code launched outside of Wing and/or on another host. Thanks to Stephan Deibel of Wing for updating this information.

Getting started — pdb.set_trace()

To start, I’ll show you the very simplest way to use the Python debugger.

1. Let’s start with a simple program, epdb1.py.

# epdb1.py -- experiment with the Python debugger, pdb
a = "aaa"
b = "bbb"
c = "ccc"
final = a + b + c
print final

2. Insert the following statement at the beginning of your Python program. This statement imports the Python debugger module, pdb.

import pdb

3. Now find a spot where you would like tracing to begin, and insert the following code:

pdb.set_trace()

So now your program looks like this.

# epdb1.py -- experiment with the Python debugger, pdb
import pdb
a = "aaa"
pdb.set_trace()
b = "bbb"
c = "ccc"
final = a + b + c
print final

4. Now run your program from the command line as you usually do, which will probably look something like this:

PROMPT> python epdb1.py

When your program encounters the line with pdb.set_trace() it will start tracing. That is, it will (1) stop, (2) display the “current statement” (that is, the line that will execute next) and (3) wait for your input. You will see the pdb prompt, which looks like this:

(Pdb)

Execute the next statement… with “n” (next)

At the (Pdb) prompt, press the lower-case letter “n” (for “next”) on your keyboard, and then press the ENTER key. This will tell pdb to execute the current statement. Keep doing this — pressing “n”, then ENTER.

Eventually you will come to the end of your program, and it will terminate and return you to the normal command prompt.

Congratulations! You’ve just done your first debugging run!

Repeating the last debugging command… with ENTER

This time, do the same thing as you did before. Start your program running. At the (Pdb) prompt, press the lower-case letter “n” (for “next”) on your keyboard, and then press the ENTER key.

But this time, after the first time that you press “n” and then ENTER, don’t do it any more. Instead, when you see the (Pdb) prompt, just press ENTER. You will notice that pdb continues, just as if you had pressed “n”. So this is Handy Tip #1:

If you press ENTER without entering anything, pdb will re-execute the last command that you gave it.

In this case, the command was “n”, so you could just keep stepping through the program by pressing ENTER.

Notice that as you passed the last line (the line with the “print” statement), it was executed and you saw the output of the print statement (“aaabbbccc”) displayed on your screen.

Quitting it all… with “q” (quit)

The debugger can do all sorts of things, some of which you may find totally mystifying. So the most important thing to learn now — before you learn anything else — is how to quit debugging!

It is easy. When you see the (Pdb) prompt, just press “q” (for “quit”) and the ENTER key. Pdb will quit and you will be back at your command prompt. Try it, and see how it works.

Printing the value of variables… with “p” (print)

The most useful thing you can do at the (Pdb) prompt is to print the value of a variable. Here’s how to do it.

When you see the (Pdb) prompt, enter “p” (for “print”) followed by the name of the variable you want to print. And of course, you end by pressing the ENTER key.

Note that you can print multiple variables, by separating their names with commas (just as in a regular Python “print” statement). For example, you can print the value of the variables a, b, and c this way:

p a, b, c

When does pdb display a line?

Suppose you have progressed through the program until you see the line

final = a + b + c

and you give pdb the command

p final

You will get a NameError exception. This is because, although you are seeing the line, it has not yet executed. So the final variable has not yet been created.

Now press “n” and ENTER to continue and execute the line. Then try the “p final” command again. This time, when you give the command “p final”, pdb will print the value of final, which is “aaabbbccc”.

Turning off the (Pdb) prompt… with “c” (continue)

You probably noticed that the “q” command got you out of pdb in a very crude way — basically, by crashing the program.

If you wish simply to stop debugging, but to let the program continue running, then you want to use the “c” (for “continue”) command at the (Pdb) prompt. This will cause your program to continue running normally, without pausing for debugging. It may run to completion. Or, if the pdb.set_trace() statement was inside a loop, you may encounter it again, and the (Pdb) debugging prompt will appear once more.

Seeing where you are… with “l” (list)

As you are debugging, there is a lot of stuff being written to the screen, and it gets really hard to get a feeling for where you are in your program. That’s where the “l” (for “list”) command comes in. (Note that it is a lower-case “L”, not the numeral “one” or the capital letter “I”.)

“l” shows you, on the screen, the general area of your program’s souce code that you are executing. By default, it lists 11 (eleven) lines of code. The line of code that you are about to execute (the “current line”) is right in the middle, and there is a little arrow “–>” that points to it.

So a typical interaction with pdb might go like this

  • The pdb.set_trace() statement is encountered, and you start tracing with the (Pdb) prompt
  • You press “n” and then ENTER, to start stepping through your code.
  • You just press ENTER to step again.
  • You just press ENTER to step again.
  • You just press ENTER to step again. etc. etc. etc.
  • Eventually, you realize that you are a bit lost. You’re not exactly sure where you are in your program any more. So…
  • You press “l” and then ENTER. This lists the area of your program that is currently being executed.
  • You inspect the display, get your bearings, and are ready to start again. So….
  • You press “n” and then ENTER, to start stepping through your code.
  • You just press ENTER to step again.
  • You just press ENTER to step again. etc. etc. etc.

Stepping into subroutines… with “s” (step into)

Eventually, you will need to debug larger programs — programs that use subroutines. And sometimes, the problem that you’re trying to find will lie buried in a subroutine. Consider the following program.

# epdb2.py -- experiment with the Python debugger, pdb
import pdb

def combine(s1,s2):      # define subroutine combine, which...
    s3 = s1 + s2 + s1    # sandwiches s2 between copies of s1, ...
    s3 = '"' + s3 +'"'   # encloses it in double quotes,...
    return s3            # and returns it.

a = "aaa"
pdb.set_trace()
b = "bbb"
c = "ccc"
final = combine(a,b)
print final

As you move through your programs by using the “n” command at the (Pdb) prompt, you will find that when you encounter a statement that invokes a subroutine — the final = combine(a,b) statement, for example — pdb treats it no differently than any other statement. That is, the statement is executed and you move on to the next statement — in this case, to print final.

But suppose you suspect that there is a problem in a subroutine. In our case, suppose you suspect that there is a problem in the combine subroutine. What you want — when you encounter the final = combine(a,b) statement — is some way to step into the combine subroutine, and to continue your debugging inside it.

Well, you can do that too. Do it with the “s” (for “step into”) command.

When you execute statements that do not involve function calls, “n” and “s” do the same thing — move on to the next statement. But when you execute statements that invoke functions, “s”, unlike “n”, will step into the subroutine. In our case, if you executed the

final = combine(a,b)

statement using “s”, then the next statement that pdb would show you would be the first statement in the combine subroutine:

def combine(s1,s2):

and you will continue debugging from there.

Continuing… but just to the end of the current subroutine… with “r” (return)

When you use “s” to step into subroutines, you will often find yourself trapped in a subroutine. You have examined the code that you’re interested in, but now you have to step through a lot of uninteresting code in the subroutine.

In this situation, what you’d like to be able to do is just to skip ahead to the end of the subroutine. That is, you want to do something like the “c” (“continue”) command does, but you want just to continue to the end of the subroutine, and then resume your stepping through the code.

You can do it. The command to do it is “r” (for “return” or, better, “continue until return”). If you are in a subroutine and you enter the “r” command at the (Pdb) prompt, pdb will continue executing until the end of the subroutine. At that point — the point when it is ready to return to the calling routine — it will stop and show the (Pdb) prompt again, and you can resume stepping through your code.

You can do anything at all at the (Pdb) prompt …

Sometimes you will be in the following situation — You think you’ve discovered the problem. The statement that was assigning a value of, say, “aaa” to variable var1 was wrong, and was causing your program to blow up. It should have been assigning the value “bbb” to var1.

… at least, you’re pretty sure that was the problem…

What you’d really like to be able to do, now that you’ve located the problem, is to assign “bbb” to var1, and see if your program now runs to completion without bombing.

It can be done!

One of the nice things about the (Pdb) prompt is that you can do anything at it — you can enter any command that you like at the (Pdb) prompt. So you can, for instance, enter this command at the (Pdb) prompt.

(Pdb) var1 = "bbb"

You can then continue to step through the program. Or you could be adventurous — use “c” to turn off debugging, and see if your program will end without bombing!

… but be a little careful!

[Thanks to Dick Morris for the information in this section.]

Since you can do anything at all at the (Pdb) prompt, you might decide to try setting the variable b to a new value, say “BBB”, this way:

(Pdb) b = "BBB"

If you do, pdb produces a strange error message about being unable to find an object named ‘= “BBB” ‘. Why???

What happens is that pdb attempts to execute the pdb b command for setting and listing breakpoints (a command that we haven’t discussed). It interprets the rest of the line as an argument to the b command, and can’t find the object that (it thinks) is being referred to. So it produces an error message.

So how can we assign a new value to b? The trick is to start the command with an exclamation point (!).

(Pdb)!b = "BBB"

An exclamation point tells pdb that what follows is a Python statement, not a pdb command.

The End

Well, that’s all for now. There are a number of topics that I haven’t mentioned, such as help, aliases, and breakpoints. For information about them, try the online reference for pdb commands on the Python documentation web site. In addition, I recommend Jeremy Jones’ article Interactive Debugging in Python in O’Reilly’s Python DevCenter.

I hope that this introduction to pdb has been enough to get you up and running fairly quickly and painlessly. Good luck!

—Steve Ferg

About these ads

25 thoughts on “Debugging in Python

  1. it’s so easy in perl, without entering any source code. why not in python?

  2. These days, if you are on Linux, you should definitely check out pudb. You’ll get the console-deployment advantages of pdb with a sane UI.

    • pudb is cross platform!

      Also, I concur that it is fabulous, and uses pretty much the same controls as described above, so you can use it more-or-less based on the instructions from this blog post.

  3. winpdb (http://winpdb.org/) deserves special mention. It is graphical (& command-line) python debugger which offers a number of exceptional features. For example, it can break into threads without the need to set a breakpoint. In spite of it’s name, it is cross-platform (using wxPython for the GUI). It’s a much smaller download than the full-featured IDEs.

  4. Excellent tutorial. Thank you.It will be awesome if you could write a “mid level one” with more commands. Loved the writing, I’ll sure point people here when they ask me.

    One note you may want to add — the list command doesn’t reset. That is, hitting “l” will give you the 11 lines, and “l” again will give you the next 11 lines.

    It could throw you off if you have something like “l l” because you forgot the name of some var but you are now in the next 11 lines.

    Sadly there is little to be done to “fix” this http://stackoverflow.com/questions/1318676/in-pdb-how-do-you-reset-the-list-l-command-line-count

    anyway awesome tutorial.

  5. > If you press ENTER without entering anything, pdb will re-execute the last command that you gave it.

    Hey Marge! I just DOUBLED my productivity!

    Great tutorial, much appreciated.

  6. Nice post.
    I would recommend using ipdb instead of pdb. This gives you the same features as well as syntax highlightning and completion.

  7. You left out one of the best ad-hock troubleshooting ways to figure out what caused an exception post mortem.

    Run:

    python -i your_script.py
    

    After the exception you will be left at the python interpreter prompt staring at your exception.

    Type:

    import pdb
    pdb.pm
    

    And you will be put into the context of the stack of the last exception. This means you can print/examine the local variables at the point of failure after the failure has occurred without having to change a line of code or import pdb in advance.

    Another method that requires a little bit of preparation is to put the import pdb and pdb.set_trace() in a signal handler trap. That way you can do a kill -SIGNAL PID or if the signal you trap is INT you can just Ctrl-C to get dropped into the debugger at any point. The signal handler technique is good for testing release candidates and released code because the signal handler doesn’t create any runtime overhead.

    Signal handler example. Debugger starts with Ctrl-C:

    import signal
    def  int_handler(signal, frame):
            import pdb
            pdb.set_trace(frame)
    signal.signal(signal.SIGINT, int_handler)
    

    Put that at the top of your script and you can start debugging your script at any point by type Ctrl-C. Resume programe execution by typing exit at the Pdb prompt.

  8. “it’s so easy in perl, without entering any source code. why not in python?”

    Although not mentioned in this tutorial that’s certainly possible by invoking the pdb module on start-up, e.g. like this:

    python -m pdb myscript.py
    
  9. Hi, first of all, thanks a lot for a great tutorial. :)

    I’m debugging a program and I have a question. I’m using both the “!” and the global directive to change the value of a global variable that’s causing the code to blow-up. Important note: the code is run from a CD, so I have no permanent access to the source.

    localpath = u’/some//path’ # notice the double slash, that’s the issue

    !global localpath; localpath=u’/some/path’

    pp localpath

    u’/some//path’

    So, can someone tell me why this is happening?

    • A lot depends on the context. For example, here is a small chunk of code:

      import pdb
      localpath = u'/some//path'
      def main():
           #localpath = u'/some//path'  # uncommenting this = problem
           xm = 123
           pdb.set_trace()
           print("Hello,world!")
      main()
      

      Your commands would work fine with this script. But if you uncomment the line that is commented out, then localpath is a local (not global) variable, and you can’t reset it.

      I’ve tried resetting the “xm” variable (which is local to the main() function) from within a pdb debugging session, and I can’t. I don’t understand why not. It looks like a bug (or a limitation) in pdb to me.

      I’ve posted a question on comp.lang.python

      http://groups.google.com/group/comp.lang.python/browse_thread/thread/faa4e0e152e77a93#

    • The u and d debug commands are your friends and can answer your question as a frivolity.

      • u moves up the call stack all the way to the global level if called enough.
      • d moves back down the stack.

      As you move up and down the stack the local scope changes to that level in the call stack. the l (lower-case L) command returns information appropriate to the stack.

      I find it useful to start an interactive Python session by typing “python” alone at a command prompt and then playing with different variations of the code until I am comfortable that the code does what I think it does.

      I like to write dense code and often bump up against the edges of idioms and have to deal realistically with the language lawyers to write reliable code.

  10. Oustanding, thanks. Our build code is written in Python and even though I used to admin it, I had never needed the debugger. Well, in this case, the code I was adding tested fine outside our environment but was getting a wrong value from somewhere during run-time, and I needed to use the debugger on the running process in our enviornment. I had no idea how to do this, but once I found your page I had the answers I needed in less time than it took me to write this note.

    Thanks.

  11. very nice! let me mention a trap. if when executing the program you redirect standard output, you will never see the pdb prompt.

  12. It looks like the Python documentation needs a Debugging HOWTO. How about we write it? :) If you’d like to contribute your nice tutorial, we can work on a new document on bugs.python.org.

  13. I have started to work on the missing debugging HOWTO in the official documentation: http://bugs.python.org/issue12913 Everyone is welcome to give feedback in that bug report. I’ll try to find time in the coming months and put it in the docs of Python 2.7, 3.2 and 3.3.

  14. Thanks for a really great introductory tutorial. Very helpful, and more good stuff in the comments too!

    I want to also add that with good tests, you almost never need to use a debugger – maybe once every couple of years. This is because there are fewer bugs to begin with, but also because getting to the right point in your program might require setting the value of several variables or input data. If you’re only going to be doing it once, a debugger is probably easier than writing a test. But if you’re going to be doing it twice or more, writing a test is generally easier than using a debugger. And if you have a bug, you need to be writing a test anyhow. So pretty soon, you just get in the habit of writing a test from the outset.

    Knowing how to use pdb is still very useful though for those occasional times when you simply cannot figure out what is going on.

  15. Great tutorial; maybe needs ‘refreshing’ for current Pythons eg 2.7 that do not need the ‘p’ command to evaluate / print expressions ??

    QUESTION: probably related to ‘phillip’ s comment above:

    – I am trying to debug an app that makes use of MLT & has a ‘GUI’ interface; where occasionally it appears to go into an infinite loop: the app GUI locks as well as the whole desktop…(only way out is Ctrl-Alt-F2 to open a root console, & kill the process… top shows Python is using about 8% of the CPU and very gradually increasing memory footprint..).

    The app also outputs ‘status’ messages (corresponding to key / button clicks etc) to the console …

    I have used

    python -m –trace app > python_trace.log 2>&1

    to capture a log of ‘where the app is stuck’ … I can see source lines of one of the inner modules that are ‘suspicious’ (ie logical analysis of the code indicates there are fail cases present).

    SO then I try

    python -m pdb app

    & setting a break-point on a line I can see is executed in the loop (after deleting the corresponding .pyc file); I have to use the full path name to the inner_module.py file.

    The breakpoint is APPARENTLY not hit .. meaning apps seems to run fine, and can eventually hit the lock-up, but no breakpoint is triggered (only .way out is as described above)

    SO then I add

    import pdb
    pdb.set_trace()

    at the same point, and start the app normally (ie type ‘app’ at the command line).
    The app stops at the expected point & pdp prompt appears like:

    (Pdb) on_frmMain_key_press_event
    (Pdb)

    (NOTE the on_frmMain_key_press_event message is the first status message printed by the app without the pdb.set_trace() statement after it has drawn the first GUI screen..)

    But I cannot type more than 1 character & the pdb console (ONLY) locks … but I can switch to another terminal window (same desktop, same user) and kill the process….at which point all the chars/commands that I was trying to type into the pdb console appear along with the message ‘Killed’ (expected due to kill -9 from root).

    ? how to proceed ? is ‘pudb’ the right tool ? (have not checked it out yet…); or is there a way from the root console to start pdb & ‘attach’ to the app that was started at the ‘user’ console like you can with gdb ?? (so hopefully pdb IO is not blocked on the same channel as the app IO…)

    thanks..

    • Try winpdb. It allows for threaded apps and seamless remote disconnected debugging at the console and with the gui fronend. pudb is console only and is not as tightly written performance and stability wise. At least not the last time I tried to use it which was several years ago.

  16. Pingback: Kenny Chowdhary · Python for Scientific Computing! (installing numpy, scipy, and matplotlib from source)

  17. Pingback: Python debugging | Some Things Are Obvious

  18. Having read this I believed it was very informative. I appreciate you spending some time and effort to put this information together. I once again find myself personally spending a lot of time both reading and leaving comments. But so what, it was still worthwhile!

  19. I seldom leave a response, but i did a few searching and wound up here Debugging in Python | Python
    Conquers The Universe. And I actually do have a couple of questions for you if
    it’s allright. Is it only me or does it look like a few of these comments come across as if they are coming from brain dead individuals? :-P And, if you are posting at additional sites, I would like to follow everything fresh you have to post. Would you list of every one of all your social sites like your linkedin profile, Facebook page or twitter feed?

Comments are closed.