On the Noble Art of Quality Assurance

So you want to be a Software Tester?

Do you have a knack for figuring things out? A knack for figuring things out that goes beyond the obvious, even to the point of figuring things out that the developer didn't figure out when he was writing the code? And yet, do you feel it is time to elevate that 'knack' to a whole new level? Do you want to take charge of the test process and elevate it to an art form?

Then Welcome to the Noble Art of Quality Assurance!

The goal of Quality Assurance in a software organization is simple: perfecting the software? That means: perfecting the developers, perfecting the processes by which software is developed, and ultimately perfecting the relationship that the software (and the company behind it) has with the end user: the customer.

There are many today that do not comprehend the difference between Software Quality Assurance and Software Test. They may deem that 'Quality Assurance Engineer' is simply a more elaborate title placed upon the Software Tester, or they may consider that the Quality Assurance Engineer is simply a Software Tester who is also schooled in the use of various tools for reporting of test metrics.

Both of these concepts touch upon the truth, but they also both entirely miss it.

The use of metrics with respect to testing, development, test coverage, bug finding, and so on, are only a tool that the QA Engineer uses on his path to helping mature his company's development processes. The end goal must always remain in sight: help mold this company's products into something the customers can depend on. Help elevate the quality of workmanship that goes into the product. Quality of workmanship is the path to a quality product, a dependable product that the customer will thank you for, both by continued patronage and word-of-mouth promotion.

In a company that makes proper use of the Quality Assurance role, the Quality Assurance Engineer is able to use both title and tools to elevate the quality of workmanship. The company gives the QA Engineer the authority to hold up a release, if needed, in order to ensure the highest quality product is released. The company gives the QA Engineer the authority to hold developers accountable to fulfillment of requirements, and overall meeting of the project vision.

This is not an 'authority' that the QA Engineer abuses, or uses lightly. It is a serious thing to hold up a software release. It can ultimately result in lost revenue. But just as serious is the authority to release a product 'on time' even if it is not ready. Not only will such a release result in lost revenue, but what's more, lost credibility.

It is thus through the use of various tools in reporting test coverage and other quality metrics that the QA Engineer provides 'Assurance' to the project management, and various other project stakeholders, on the release-ability of the product. The QA Engineer must become 'master' of both the company development processes, and the tools which enable him to provide some level of assurance in a product.

To speak of the role of Quality Assurance like this is to underscore that it is one of the most important roles in a Software Organization. The QA Engineer must not merely elevate the process of software testing to an art form, but must elevate the entire process of software development to an art form. In the end it is the artwork itself (a finely crafted piece of software) that provides the entire team that worked on it with the satisfaction of having perfected something and brought it out into popular usage. And the QA Engineer can sleep at night, knowing that he has in no small way helped accomplish this.

Test?

Has The Seattle School District ( http://seattletimes.nwsource.com/html/education/2003906580_bus27m.html ) ever heard the word "Test" before?

How much you want to bet this software project was something that they insisted on being released on time (when school started) even if it wasn't ready yet?

Compromise leads to chaos!

~ QA Guru

You haven't got anything to sell but quality

I'm reading numerous books on quality as I prepare for my ASQ Certification and I really liked the quote I found below:

Your customers are in a perfect position to tell you about quality, because that's all they're really buying. They're not buying a product. They're buying your assurances that their expectations for that product will be met.

And you haven't really got anything else to sell them but those assurances. You haven't really got anything else to sell but quality.

(Guaspari, in Kan)

Regards!

Basil the Bugsmasher

Model-Based Testing

Some great articles by Harry Robinson:

http://www.geocities.com/model_based_testing/shoestring.htm

http://www.geocities.com/model_based_testing/intelligent.pdf

and the model-based testing web site in general:

http://www.geocities.com/model_based_testing/

Model-based testing is nothing new, and was basically first eluciated by Chin-Kuei Cho back in 1980 (An Introduction to Software Quality Control) and elaborated on in 1987 (Quality Programming: Developing and Testing Software with Statistical Quality Control).

The basic idea is fairly simple. You cannot test every path through a program, nor all inputs in all configurations for all possible outputs, because it is often (usually) infinite. You therefore have to model the processes of the software, and choose what specific inputs and configurations you want to use.

There are four basic methods of selecting sample test data:

1) Regular Sampling (each sample has an equal chance of being selected)
2) Weighted sampling (some regular samples mixed with some specially chosen samples that have more interesting characteristics)
3) Boundary Sampling (selecting boundaries for all samples)
4) Invalid Sampling (selecting some out-of-range samples to make sure the program correctly handles them.

I usually use all four. But I would usually run two separate tests: one with Invalid Sampling (a stress test) and one with Weighted Sampling that includes (Regular Samples, Boundary Samples, plus some other specially chosen samples.)

An interesting article by IEEE on "Why Software Fails"
http://www.spectrum.ieee.org/sep05/1685

IT pros say they cannot stop every threat

I ran across this excellent article at "SearchSecurity.com" today:

http://searchsecurity.techtarget.com/originalContent/0,289142,sid14_gci1168007,00.html?track=NL-102&ad=543470


They say:

Security incidents can slip right past an IT shop amid a merger, tight staffing or when technology deployments outpace an enterprise's ability to keep up. In a recent survey, some IT professionals admitted this is exactly the scenario they're dealing with.

And this is precisely the scenario that the world of computer users is faced with constantly in the software they use. Companies are always in a rush to adapt new technologies with little or no thought toward Quality Assurance (which can and should include security vulnerability assessment).

In all, 28% of respondents said they have "little or no confidence" that they've detected all significant security breaches in the past year. Meanwhile, 26% rated their current IT environment as more vulnerable than it was the year before.

That doesn't surprise me either. Even on a small scale I've experienced lately a management imperative of "just get it out there" - without planning or scheduling any quality control or test. Being the evil QA guy I am, I've recently even poked my nose into things that were not in mo domain and pointed out to management that certain vulnerabilities were blantantly obvious. But I still wasn't asked to fully QA the thing, even after spending 10 minutes and pointing out dramatic flaws.

I've also spent a lot of time lately testing on the fringes of various open source projects, and it is frankly appaling how many bugs and potential security vulnerabilities are present.

One wonders what we are setting ourselves up for in this computer crazed society where corporations have such lack of discretion.

Regards,
Basil

Good Reference Articles

The following are good reference articles:

"Doing it right the first time" doesn't solve the problems inherent in Quality:
Write First Time is Wrong