Ismail Badawi

Unintended StopIteration in a Generator

Sometimes, if I have a generator that I happen to know is non-empty, and I want to get at the first element, I’ll write code like this:

output = next(f for f in os.listdir(dir) if f.endswith('.o'))

In theory, the intended meaning of this code is something like this:

outputs = [f for f in os.listdir(dir) if f.endswith('.o')]
assert len(outputs) > 0 # or maybe assert len(outputs) == 1
output = outputs[0]

These two pieces of code are similar, but differ in one important way – if the assumption is wrong (i.e. there is a bug in the program), then the second will raise an AssertionError, while the first will raise a StopIteration exception. If this code happens to be inside a generator, maybe like this:

def outputs(dirs):
  for dir in dirs:
    yield next(f for f in os.listdir(dir) if f.endswith('.o'))

Then while an AssertionError would correctly bubble up to the caller, a StopIteration exception would instead only prematurely signal that the generator is exhausted, and it wouldn’t be possible in general for the caller to tell that something has gone wrong – it’s likely that the program would just keep running and produce wrong results, making the bug potentially much less straightforward to track down.

So while using next for this purpose is cute, its behavior in cases like this might catch you off guard. If your intention is to communicate an assumption you’re making, you’re probably better off using assert, even if it’s slightly more long-winded.

Read-only File in Writable Directory

This is a small gotcha about file permissions I ran into recently.

Let’s say there is a file on which you don’t have write permission:

$ ls -l file
-rw-r--r--  1 notyou  somegroup  68 25 Mar 21:12 file

You can’t write to this file:

$ echo "some text" > file
bash: file: Permission denied

However, let’s say this file is in a directory on which you do have write permission (assuming here that you are in somegroup):

$ ls -ld .
drwxrwxr-x  2 notyou  somegroup  68 25 Mar 21:12 dir

Now even though you can’t modify the file, write permission on the directory lets you remove the file and write a new file in its place:

$ rm file
$ echo "some text" > file

Depending on the situation, this can be just as good. You don’t even have to do this manually – if you’re using vim, then :w! will automatically do this if the file is not writable.

I encountered a system at work that would inspect the contents of certain files to decide whether to allow certain dangerous operations. These files had clearly defined owners that you were meant to get in touch with to ask for approval – if approval was granted, they would edit the relevant files for you, so that you could go ahead and do what you needed to do. But since the files were set up in this way – read-only in a group-writable directory – in practice anyone could edit them, bypassing whatever restrictions were in place.

If you’re relying on a file being read-only, be mindful of the permissions set on any parent directories.

A Bug Caused by Using 0 Instead of NULL

This is a quick post about a bug I ran into at work which turned out to be caused by passing a literal 0 instead of NULL to a function. Here’s a small program reproducing it:

When Optimizations Hide Bugs

The past few months at work I’ve been working with a large legacy codebase, mostly written in C. In order to debug problems as they come up, I wanted to use gdb – but it seemed like the program was only ever compiled with optimizations turned on (-O2), which can make using gdb a frustrating experience, as every interesting value you might want to examine has been optimized out.

After grappling with the build system to pass -O0 in all the right places (a surprisingly difficult task), I found that the program did not link with optimizations turned off. Once I got around that, I ran into a crash in some basic functionality, easily reproducible at -O0, but not -O2. This post contains two tiny contrived programs reproducing those issues.

The Compositional Nature of Vim

I use vim. I’ve used vim since I started programming; the very first program I wrote – hello world in C, following along a tutorial – was typed out in vim, inside a cygwin environment on Windows. Naturally, at first it was hard and intimidating. I didn’t know how to do anything, least of all edit text. I learned about insert mode and normal mode. I learned about navigating using hjkl, and deleting the current line with dd, and saving and quitting with :wq, and for a long time that was it.

An Obscure Bug Story

It’s common to be asked a question like “what’s the hardest bug you’ve debugged?” at job interviews in tech. This post is about the bug I usually describe. The snag is that it’s quite involved and I don’t actually understand it all the way through – there are one or two aspects to it I often hand-wavingly gloss over. The hope was that by writing it out and fact checking it I’d have a better handle on it; this is what came out.

Writing a Code Coverage Tool

Disclaimer: I’m not quite sure who the audience is for this. I guess it’s describing a fun little project I put together, but it’s also written kind of like a tutorial, so you can maybe follow along. I don’t think it’s particularly beginner-friendly, though. Some knowledge of Java is assumed, but not much. The code is available on github.

Code coverage is a software metric that measures how much, and which parts, of the source code of a program were exercised in a given execution of that program. There are many different flavors of coverage data, for example tracking which lines or statements were executed, which functions were called, which branches or control flow paths were taken. In this post, we’ll walk through writing a simplistic coverage collection tool for Java.