Although I’m employing property-based testing for several years already, I keep on being surprised about its ability to find obscure bugs before they occur in production. I’m using the Hypothesis Python package regularily in software projects and lately it helped me to catch a very simple but still somehow surprising bug while working on tmtoolkit:

I had a simple function that converted Windows line breaks to UNIX line breaks. Something like:

def linebreaks_win2unix(text):
    return text.replace('\r\n', '\n')

What could go wrong with such a simple function? Let’s write a property-based test with Hypothesis that generates strings with maximum length of 20 characters from an alphabet of a, b, c, space, carriage return \r and line feed \n. We only check for the property that after conversion, there shouldn’t be any Windows line breaks \r\n left in the converted string.

from hypothesis import given, strategies as st

@given(text=st.text(alphabet=list('abc \r\n'), max_size=20))
def test_linebreaks_win2unix(text):
    assert '\r\n' not in linebreaks_win2unix(text)

If you (like me) didn’t give much thought into the problem, because it seemed to simple, you’ll be surprised that the test fails:

Falsifying example: test_linebreaks_win2unix(
    text='\r\r\n',
)

    @given(text=st.text(alphabet=list('abc \r\n'), max_size=20))
    def test_linebreaks_win2unix(text):
>       assert '\r\n' not in linebreaks_win2unix(text)
E       AssertionError: assert '\r\n' not in '\r\n'

On second thought, though, it is clear why this happens: If you have a string that contains \r\r\n, only the last two characters will be translated to \n which in the end leads to the string \r\n so that the result string still contains a Windows line break. A possible solution would be to perform the replacements iteratively:

def linebreaks_win2unix(text):
    while '\r\n' in text:
        text = text.replace('\r\n', '\n')

    return text

It’s natural not to think about such issues, because when thinking about which input such a function could receive, you’d imagine files with lines that end with \r\n. You wouldn’t probably come up with edge cases like a file that contains \r\r\n. But it is these edge cases that make programs fail, not the “regular” inputs. Property-based testing helps to find such edge cases.