Introduction
A reader asked for more information on bias. Everyone has biases. Bias is addressed in psychology, sociology, experiments (testing), law, math, and many other fields. Wikipedia addresses it with postings such as:
General – http://en.wikipedia.org/wiki/Bias
Cognitive (psychology) http://en.wikipedia.org/wiki/Cognitive_bias
Math stat’s – http://en.wikipedia.org/wiki/Bias#In_statistics
Defined on Google web as
noun
1. 1.
prejudice in favor of or against one thing, person, or group compared with another, usually in a way considered to be unfair.
“there was evidence of bias against foreign applicants”
synonyms: prejudice, partiality, partisanship, favoritism, unfairness, one-sidedness…
So testers have bias, but as professional experimenters, we need to understand the concept and potential risk. Now there are whole web sites, sections of classes, and materials dealing with testing bias. It is large subject dealing with human nature. For testing the big areas to be aware of his the Cognitive bias and perspective issues. To experience these, watch this video:
Now in my classes if people are focused on counting the passes and haven’t seen the concept before, they miss the person on the bear suit. Sometimes people are counting the passes and see the bear, but this is minority. The implications on testing are:
1. No observer (of a test) is perfect, so things will be missed, e.g. a bug.
2. By being focused on one task, the likelihood of missing something increases.
3. Once tester know they have bias, they stand some chance of “seeing” the bear.
4. If we are testing (i.e. checking) for success of a requirement as a task, we may miss bugs and not fully test our product (this is why one thing I like to do is attack based testing). We call “Confirmation Bias” in testing (we know the code works) and causes many testers to test (check) only requirements in what some of call “green light” test cases.
http://en.wikipedia.org/wiki/Confirmation_bias
Thoughts
There are videos on the subject and forms of bias, but I hope you get the idea. Bias can be a big risk for testing. The understanding by a tester that what is thought of and perceived are not the pure factual truth is important. I talk with many smart people, who tell me “I remember what happened and know what truth is”. I know this to be impossible in myself and others. It is irrational judgment. Anaïs Nin says: “We don’t see things as they are, we see them as we are.” I can use math, techniques, and training to overcome this in part, but it will always exist.
Consider why some people say developers should not test their own code??? They have bias. They know their code. It works. They run a few tests which show this (confirmation bias). They miss bugs. Now this does not say developers should not test their own code. They should. Some programmers will even do a good job at it because they know they are biased, but someone else should also test the code. We do this with editor in writing books or articles. Even testers should have someone independent looking at their tests, inputs, and results. The number of independent check has to be balanced. For example, on my book I had many independent checkers (editors). On my blog posting, I have one (sometimes after the fact). On my emails I have none. My emails have more bugs than the others, but the book still has some bugs (hope I can find those in revision 2). So in testing and dealing with bias, the amount of effort to compensate for bias is a function of risk. Is the code like my emails (a bug has only minimal impact) or is the code mission critical (like my book). The later probably wants more testing and work to overcome bias
Overcoming Bias
Here is my list of “tricks” which can help with bias. It is not complete, but gets you started.
1. Identify you are biased (belief pattern) and seek “independent” second sets of eyes. Convince yourself that attention blindness strongly affects you. Avoid complacent “if it’s there I will see it” attitude.
2. Watch biasing the second set of eyes (don’t say “the code works, but check it for me anyway)
3. Apply attack based and exploratory testing as well as traditional scripted or automated tests
4. Take a break (leave for an hour or day), then come back and look at the problem (test, code, whatever) with “fresh” perspective by doing things such as: a second reading, read backwards, read out loud, explain it to others, etc.
5. Test in multiple passes, switching your attention to different parts of the screen, or different qualities of it (example: performance on one pass, correct answer on the next, watch other areas of the screen on third, etc.)
6. Diversify your testing eyes by using paired testing, external beta testing, bug bash events, etc.
7. Beware of testing that relies exclusively on narrow oracles, “monkey” procedures, or “total” test automation.
8. Augment visual testing with automatic (tools), e.g. sounds, probes that signal trouble, capture play back tools, data analysis, etc..
9. Use different test techniques, approaches and basis…………
a. Math based testing: Combinatorial testing, statistics, random, etc.
b. Model based testing: Build a model, have the model checked, use the model in testing….
c. Risk based testing: what scary things should we test for?
d. How many test techniques do you know and use (3, 10, 50, more)?
Hope this helps.
breakingembeddedsoftware 