More thoughts on Tester Bias and Perception

A reader asked for more information on bias. Everyone has biases. Bias is addressed in psychology, sociology, experiments (testing), law, math, and many other fields. Wikipedia addresses it with postings such as:
General –
Cognitive (psychology)
Math stat’s –
Defined on Google web as
1. 1.
prejudice in favor of or against one thing, person, or group compared with another, usually in a way considered to be unfair.
“there was evidence of bias against foreign applicants”
synonyms: prejudice, partiality, partisanship, favoritism, unfairness, one-sidedness…

So testers have bias, but as professional experimenters, we need to understand the concept and potential risk. Now there are whole web sites, sections of classes, and materials dealing with testing bias. It is large subject dealing with human nature. For testing the big areas to be aware of his the Cognitive bias and perspective issues. To experience these, watch this video:

Now in my classes if people are focused on counting the passes and haven’t seen the concept before, they miss the person on the bear suit. Sometimes people are counting the passes and see the bear, but this is minority. The implications on testing are:
1. No observer (of a test) is perfect, so things will be missed, e.g. a bug.
2. By being focused on one task, the likelihood of missing something increases.
3. Once tester know they have bias, they stand some chance of “seeing” the bear.
4. If we are testing (i.e. checking) for success of a requirement as a task, we may miss bugs and not fully test our product (this is why one thing I like to do is attack based testing). We call “Confirmation Bias” in testing (we know the code works) and causes many testers to test (check) only requirements in what some of call “green light” test cases.
There are videos on the subject and forms of bias, but I hope you get the idea. Bias can be a big risk for testing. The understanding by a tester that what is thought of and perceived are not the pure factual truth is important. I talk with many smart people, who tell me “I remember what happened and know what truth is”. I know this to be impossible in myself and others. It is irrational judgment. Anaïs Nin says: “We don’t see things as they are, we see them as we are.” I can use math, techniques, and training to overcome this in part, but it will always exist.
Consider why some people say developers should not test their own code??? They have bias. They know their code. It works. They run a few tests which show this (confirmation bias). They miss bugs. Now this does not say developers should not test their own code. They should. Some programmers will even do a good job at it because they know they are biased, but someone else should also test the code. We do this with editor in writing books or articles. Even testers should have someone independent looking at their tests, inputs, and results. The number of independent check has to be balanced. For example, on my book I had many independent checkers (editors). On my blog posting, I have one (sometimes after the fact). On my emails I have none. My emails have more bugs than the others, but the book still has some bugs (hope I can find those in revision 2). So in testing and dealing with bias, the amount of effort to compensate for bias is a function of risk. Is the code like my emails (a bug has only minimal impact) or is the code mission critical (like my book). The later probably wants more testing and work to overcome bias
Overcoming Bias
Here is my list of “tricks” which can help with bias. It is not complete, but gets you started.
1. Identify you are biased (belief pattern) and seek “independent” second sets of eyes. Convince yourself that attention blindness strongly affects you. Avoid complacent “if it’s there I will see it” attitude.
2. Watch biasing the second set of eyes (don’t say “the code works, but check it for me anyway)
3. Apply attack based and exploratory testing as well as traditional scripted or automated tests
4. Take a break (leave for an hour or day), then come back and look at the problem (test, code, whatever) with “fresh” perspective by doing things such as: a second reading, read backwards, read out loud, explain it to others, etc.
5. Test in multiple passes, switching your attention to different parts of the screen, or different qualities of it (example: performance on one pass, correct answer on the next, watch other areas of the screen on third, etc.)
6. Diversify your testing eyes by using paired testing, external beta testing, bug bash events, etc.
7. Beware of testing that relies exclusively on narrow oracles, “monkey” procedures, or “total” test automation.
8. Augment visual testing with automatic (tools), e.g. sounds, probes that signal trouble, capture play back tools, data analysis, etc..
9. Use different test techniques, approaches and basis…………
a. Math based testing: Combinatorial testing, statistics, random, etc.
b. Model based testing: Build a model, have the model checked, use the model in testing….
c. Risk based testing: what scary things should we test for?
d. How many test techniques do you know and use (3, 10, 50, more)?
Hope this helps.


Sticky Minds interview and information filters

So as part of “PR” on a book one does interviews and presentations. Such information sources have data and tie to the book. Here is one that just came out this week.

So this raises the question of information overload in my mind. I sometimes feel like I get to much information, e.g. books, magazines, web pages, emails, tweets, blogs, forums, white papers, presentations, webinars, radio, TV, my dog barking, my wife, my family,…………

We all filter these. In the case of several of these information flows listed above, I filter at the wrong time, e.g. my dog cries to go out, I ignore, she poops in the house.

So be careful with filters and bias. We all have them. Do you even know the filters and bias you have? A good tester thinks about these things. You need good filters. You need to know when to turn the filters on and off as a tester. If you don’t know about information filters and bias as a tester, you should do some more reading on these subjects.

I hope you don’t filter this blog and interview. I am still trying to understand better ways to filter the information age. Let me know if you have some good ones. I’d like to improve filters for email, tweets, and forums.

Book is Here!

After years of hard work, my dream has been achieved. FedEx delivered copies of my book from CRCPress!  (Order your copy from your favorite bookseller.)  Now, I get to think about all the things I’d like to change in the book and I would like to begin work on other books.

Testing, as a profession, is about constantly learning and thinking. Hopefully I’ll last other 10 years. Thanks to all who helped me or will help me.