There will be more like this and we should learn from the mistakes of the past, but do we?
As I read Matt’s article, I am struck by how many of the things he cites as being things I’ve heard before. Further many of these are not technical failure, but what I would call management issues. We talk about better skills in testing and development. Maybe, we need to educate people better in school what is possible with computers. Some people expect to much from software, but as the young generations (who grew up with computers) come into leadership, hopefully we’ll have less of these events. Unfortunately, I really expect such situations are human nature, so we will always have them.
So my writings and teaching are in part aimed to give teams the testing tools to get information that might avoid such articles. I have minimal insight to specific companies, but the ones I have worked with want to avoid such press and legal actions. Many managers and leaders of companies working with embedded software devices do not always seem to see the importance of good testing, including approaches such as attack based testing (what my book is about). I was lucky because many places I work valued testing as part of their information gathering. They ran scared and needed as much info as possible. True, you can not test everything, but embedded and mobile software can be complex and simple testing only partially works for complexity.
I wish everyone to balance basic check tests and more advanced attacks. The bugs cited in the article match the general error taxonomy history I have worked on for years and confirm the validity of many of the attack patterns in the book.
Introduction
A reader asked for more information on bias. Everyone has biases. Bias is addressed in psychology, sociology, experiments (testing), law, math, and many other fields. Wikipedia addresses it with postings such as:
General – http://en.wikipedia.org/wiki/Bias
Cognitive (psychology) http://en.wikipedia.org/wiki/Cognitive_bias
Math stat’s – http://en.wikipedia.org/wiki/Bias#In_statistics
Defined on Google web as
noun
1. 1.
prejudice in favor of or against one thing, person, or group compared with another, usually in a way considered to be unfair.
“there was evidence of bias against foreign applicants”
synonyms: prejudice, partiality, partisanship, favoritism, unfairness, one-sidedness…
So testers have bias, but as professional experimenters, we need to understand the concept and potential risk. Now there are whole web sites, sections of classes, and materials dealing with testing bias. It is large subject dealing with human nature. For testing the big areas to be aware of his the Cognitive bias and perspective issues. To experience these, watch this video:
Now in my classes if people are focused on counting the passes and haven’t seen the concept before, they miss the person on the bear suit. Sometimes people are counting the passes and see the bear, but this is minority. The implications on testing are:
1. No observer (of a test) is perfect, so things will be missed, e.g. a bug.
2. By being focused on one task, the likelihood of missing something increases.
3. Once tester know they have bias, they stand some chance of “seeing” the bear.
4. If we are testing (i.e. checking) for success of a requirement as a task, we may miss bugs and not fully test our product (this is why one thing I like to do is attack based testing). We call “Confirmation Bias” in testing (we know the code works) and causes many testers to test (check) only requirements in what some of call “green light” test cases. http://en.wikipedia.org/wiki/Confirmation_bias
Thoughts
There are videos on the subject and forms of bias, but I hope you get the idea. Bias can be a big risk for testing. The understanding by a tester that what is thought of and perceived are not the pure factual truth is important. I talk with many smart people, who tell me “I remember what happened and know what truth is”. I know this to be impossible in myself and others. It is irrational judgment. Anaïs Nin says: “We don’t see things as they are, we see them as we are.” I can use math, techniques, and training to overcome this in part, but it will always exist.
Consider why some people say developers should not test their own code??? They have bias. They know their code. It works. They run a few tests which show this (confirmation bias). They miss bugs. Now this does not say developers should not test their own code. They should. Some programmers will even do a good job at it because they know they are biased, but someone else should also test the code. We do this with editor in writing books or articles. Even testers should have someone independent looking at their tests, inputs, and results. The number of independent check has to be balanced. For example, on my book I had many independent checkers (editors). On my blog posting, I have one (sometimes after the fact). On my emails I have none. My emails have more bugs than the others, but the book still has some bugs (hope I can find those in revision 2). So in testing and dealing with bias, the amount of effort to compensate for bias is a function of risk. Is the code like my emails (a bug has only minimal impact) or is the code mission critical (like my book). The later probably wants more testing and work to overcome bias
Overcoming Bias
Here is my list of “tricks” which can help with bias. It is not complete, but gets you started.
1. Identify you are biased (belief pattern) and seek “independent” second sets of eyes. Convince yourself that attention blindness strongly affects you. Avoid complacent “if it’s there I will see it” attitude.
2. Watch biasing the second set of eyes (don’t say “the code works, but check it for me anyway)
3. Apply attack based and exploratory testing as well as traditional scripted or automated tests
4. Take a break (leave for an hour or day), then come back and look at the problem (test, code, whatever) with “fresh” perspective by doing things such as: a second reading, read backwards, read out loud, explain it to others, etc.
5. Test in multiple passes, switching your attention to different parts of the screen, or different qualities of it (example: performance on one pass, correct answer on the next, watch other areas of the screen on third, etc.)
6. Diversify your testing eyes by using paired testing, external beta testing, bug bash events, etc.
7. Beware of testing that relies exclusively on narrow oracles, “monkey” procedures, or “total” test automation.
8. Augment visual testing with automatic (tools), e.g. sounds, probes that signal trouble, capture play back tools, data analysis, etc..
9. Use different test techniques, approaches and basis…………
a. Math based testing: Combinatorial testing, statistics, random, etc.
b. Model based testing: Build a model, have the model checked, use the model in testing….
c. Risk based testing: what scary things should we test for?
d. How many test techniques do you know and use (3, 10, 50, more)?
Hope this helps.
So as part of “PR” on a book one does interviews and presentations. Such information sources have data and tie to the book. Here is one that just came out this week.
StickyMinds http://www.stickyminds.com/s.asp?F=S18047_ART_2
So this raises the question of information overload in my mind. I sometimes feel like I get to much information, e.g. books, magazines, web pages, emails, tweets, blogs, forums, white papers, presentations, webinars, radio, TV, my dog barking, my wife, my family,…………
We all filter these. In the case of several of these information flows listed above, I filter at the wrong time, e.g. my dog cries to go out, I ignore, she poops in the house.
So be careful with filters and bias. We all have them. Do you even know the filters and bias you have? A good tester thinks about these things. You need good filters. You need to know when to turn the filters on and off as a tester. If you don’t know about information filters and bias as a tester, you should do some more reading on these subjects.
I hope you don’t filter this blog and interview. I am still trying to understand better ways to filter the information age. Let me know if you have some good ones. I’d like to improve filters for email, tweets, and forums.
After years of hard work, my dream has been achieved. FedEx delivered copies of my book from CRCPress! (Order your copy from your favorite bookseller.) Now, I get to think about all the things I’d like to change in the book and I would like to begin work on other books.
Testing, as a profession, is about constantly learning and thinking. Hopefully I’ll last other 10 years. Thanks to all who helped me or will help me.
Certification, certificates, degrees, demonstrated skill, practice (jobs). When I read a resume, I see these things. Each provides a data point about the person. Each can be used as a filter in hiring. Next, many people looking to hire have interview questions (even standardized ones). I asked a few of these after I read the resume. But to me when I was looking to hire someone, I looked for a quality that is hard to see on resumes or in standard questions. I want to see passion for the job I am offering. I believe my thinking is that passion about doing the work was more important and that passion would lead to the person seeking knowledge to perform well in the job.
Well, the book is starting to ship and be available this week. Check out your favorite online book seller for “Software Tests Attacks to Break Mobile and Embedded Devices” from CRCPress.
FYI I will be presenting a free webinar based on my recent book on software test attacks to find bugs in mobile and embedded software. There are parts that are applicable to many mobile and embedded systems.
Finally if you have some time (and money) I will be teaching at STARWEST in two weeks (google “STARWEST +conference” to find the details and how to register.
Well as my book talks about I am a big fan of combinatorial testing (you can google on that and look for the tool from NIST called ACT). It is possible to use math to test a large space, such as numerous platforms, in a very scientific approach.
So we were at Softec Asia (software testing) last week. It is a regional conference but there seemed to be a lot of interest in software testing. The interest started at “how do I get started”, but interestingly there was also interest in “getting testing to the next level”. I think the certifications may be a start for some testers, but to be a good tester and working at higher levels, much more is needed. So the questions would be: 1) what skills are needed to become a good; 2) how does a tester work on developing these; 3) does a tester need skills outside of “traditional” test training; and 4) what are the resources to help?
I think I know the answer to item 3. I have always felt a tester needs many skills other than testing, e.g. system and software engineering, hardware experience, soft science skills, science skills, and even management. I have spent 55 years of training and effort to learn these (starting in pre-school).
The answers to the other question are not clear to me. The industry may have some of these in work. We see standards, groups like AST and ISTQB, training companies, traditional college, and OJT. Of these OJT may be best, but it seems like we need more. Further what we have, e.g. standards, books of knowledge, training, and certification, have not stood the test of usage and time.
So anyone working in testing, using books, following standards, getting training and/or seeking certification should view each of these with a scientific eye (scientific method). Evolution and survival of the fit will take place.
Our books is being printed. Now all that is left is to get it on the shelf in early Oct. We will be talking and writing about it plus we need to get the web site for it up. When the site is ready (in work now), I’ll blog about that. The world of mobile and embedded changes quickly so the web site will contain the latest and greatest.
breakingembeddedsoftware 