The Art of Reporting Bugs
I've always intended to write from the end-user's perspective about the art of testing software and reporting bugs. End-user testing is fundamental to the iterative process of developing quality software. The more technical approach for testing software would certainly discuss engineering tools and standards which are widely practiced by development teams. This piece is addressed to end-users who use desktop/laptop software daily and occasionally run into an annoying, unexpected behavior or worse, an application crash. It happens.
Independent developers depend upon feedback and bug reports to verify then validate their bug fixes. Obviously not all bugs are showstoppers or system crashes; they might be cosmetic in nature or a typo in the documentation, an unclear menu command or even a confusing feature that is poorly implemented or poorly explained. These are just a few of the many issues we expect to deal with as software developers. And for us, customer service, philosophically speaking, is a product feature that is necessary for delivering more value. And many times, customer service begins with the bug report and problem feedback. Why is my software crashing today and can you help me right now?
Over the years, we have come to know many of our users while furiously tracking down a bug fix or making a feature improvement. Typically the exchange begins with a request from us for more information about the bug or problem being reported. No one expects a perfectly documented bug report but some of our users have come close on their first feedback email.
So what are some of the best practices we recommend when reporting bugs? As common sense as any checklist can be, there are still gaps in the reporting process but these guidelines are intended to help remind anyone reporting a bug that process and completeness matter. Even the most experienced developers and testers will slip up or forget important details that are necessary for testing and validating a new bug report or even doing a regression test to validate a bug fix. But at the end of the day, it's still an art in terms of creating an effective bug report that communicates so clearly as to inform the developer without any ambiguity or obfuscation.
A Simple Bug Reporting Checklist
#1 - Is the unexpected behavior or application hang/crash reproducible?
#2 - What are the exact steps for reproducing the observed bug?
#3 - What is your environment (system software version, application version, hardware)
#4 - What other applications were running at the time of the event?
#5 - Create a screen recording with voice narration of the issue (QuickTime Player feature)
#6 - Send the Console log or the crash report (modern OS systems now report automatically)
#7 - Send a System Report (found in the About This Mac menu)
And finally, Mike, our CTO says to think of Sherlock Holmes. The smallest detail may seem insignificant but could be the most important clue.