Software Testing

As you will learn in Chapter 2, software doesn’t just magically appear there’s usually a planned, methodical development process used to create it.

Figure 1.2. The cost to fix bugs can increase dramatically over time.

Time When Bug Is Found

The costs are logarithmic that is, they increase tenfold as time increases. A bug found and fixed during the early stages when the specification is being written might cost next finds it, the cost could easily be thousands or even millions of dollars.

As an example of how this works, consider the Disney Lion King case discussed earlier. The root cause of the problem was that the software wouldn’t work on a very popular PC platform. If, in the early specification stage, someone had researched what PCs were popular and specified that the software needed to be designed and tested to work on those configurations, the cost of that effort would have been minimal. If that didn’t occur, a backup would have been for the software testers to collect samples of the popular PCs and verify the software on them. They would have found the bug, but it would have been more expensive to fix because the software would have to be debugged, fixed, and retested. The development team could have also sent out a preliminary version of the software to a small group of customers in what’s called a beta test. Those customers, chosen to represent the larger market, would have likely discovered the problem. As it turned out, however, the bug was completely missed until many thousands of CD-ROMs were created and purchased. Disney ended up paying for telephone customer support, product returns, replacement CD-ROMs, as well as another debug, fix, and test cycle. It’s very easy to burn up your entire product’s profit if serious bugs make it to customer.

What Exactly Does a Software Tester Do?

You’re now seen examples of really nasty bugs. You know what the definition of a bug is, and you know how costly they can be. By now it should be pretty evident what a tester’s goal is:

The goal of a software tester is to find to bugs:

You may run across product teams who want their testers to simply confirm that the software works, not to find bugs. Reread the case study about the Mars Polar Lander, and you’ll see why this is the wrong approach. If you’re only testing things that should work and setting up your tests so they’ll pass, you will miss the things that don’t work. You will miss the bugs.

If you’re missing bugs, you’re costing your project and your company money. As a software tester you shouldn’t be content at just finding bugs you should think about how to find them sooner in the development process, thus making them cheaper to fix.

The goal of a software tester is to find bugs and find them as early as possible.

But, finding bugs, even finding them early, isn’t enough. Remember the definition of a bug. You, the software tester, are the customer’s eyes, the first one to see the software. You speak for the customer and must seek perfection.

This goal of a software tester is to find bugs, find them as early as possible, and make sure they get fixed.

This final definition is very important. Commit it to money and refer back to it as you learn the testing techniques discussed throughout the rest of this book.

It’s important to note that “fixing” a bug does not necessarily imply correcting the software. It could mean adding a comment in the user manual or providing special training to the customer. It could require changing the statistics that the marketing group advertise or even postponing the release of the buggy feature. You’ll learn throughout this book that although you’re seeking perfection and making sure that the bugs get fixed, that there are practical realities to software testing. Don’t get caught in the dangerous spiral of unattainable perfection.

What Makes a Good Software Tester?

In the movie star Trek II: the Wrath of Khan, Spock says, “As a matter of cosmic history, it has always been easier to destroy than to create.” At first glance, it may appear that a software tester’s job would be easier than a programmer’s. breaking code and finding bugs must surely be easier than writing the code in the first place. Surprisingly, it’s not. The methodical and disciplined approach to software testing that you’ll learn in this book requires the same hard work and dedication that programming does. It involves very similar skills, and although a software tester doesn’t necessarily need to be a full-fledged programmer, having that knowledge is a great benefit.

Today, most mature companies treat software testing as a technical engineering profession. They recognize that having trained software testers on their project teams and allowing them to apply their trade early in the development process allows them to build better quality software. Unfortunately, there are still a few companies that don’t appreciate the challenge of software testing and the value of well-done testing effort. In a free market society, these companies usually aren’t around for long because the customers speak with their wallets and choose not to buy their buggy products. A good test organization (or the lack of one) can make or break a company.

Here’s a list of traits that most software testers should have:

 They are explorers. Software testers aren’t afraid to venture into unknown situations. They love to get a new piece of software, install it on their PC, and see what happens.

 They are troubleshooters. Software testers are good at figuring out why something doesn’t work. They love puzzles.

 They are relentless. Software testers keep trying. They may see a bug that quickly vanishes or is difficult to re-create. Rather than dismiss it as a fluke, they will try every way possible to find it.

 They are creative. Testing the obvious isn’t sufficient for software testers. Their job is to think up creative and even off-the-wall approaches to find bugs.

 They are (mellowed) perfectionists. They strive for perfection, but they know when it becomes unattainable and they’re okay with getting as close as they can.

 They exercise good judgment. Software testers need to make decisions about what they will test, how long it will take, and if the problem they’re looking at is really a bug.

 They are tactful and diplomatic. Software testers are always the bearers of news. They have to tell the programmers that their baby is ugly. Good software testers know how to do so tactfully and professionally and know how to work with programmers who aren’t always tactful and diplomatic.

 They are persuasive. Bugs that testers find won’t always be viewed as severe enough to be fixed. Testers need to be good at making their points clear, demonstrating why the bug does indeed need to be fixed, and following through on making it happen.

In addition to these traits, having some education in software programming is a big bugs. As you’ll see in Chapter 6. “Examining the Code,” knowing how software is written can give you a different view of where bugs are found, thus making you a more efficient and effective tester. It can also help you develop the testing tools discussed in Chapter 15, “Automated Testing and Test Tools.”

Lastly, if you’re an expert in some non-computer field, your knowledge can be invaluable to a software team creating a new product. Software is being written to do just about everything today. Your knowledge of teaching, cooking, airplanes, carpentry, medicine, or whatever would be a tremendous help finding bugs in software for those areas.

Summary

Software testing is a critical job. With the size and complexity of today’s software, it’s imperative software testing be performed professionally and effectively. Too much is at risk. We don’t need more defective computer chips, crashed system, or stolen credit card numbers.

In the following chapters of Part I “The Big Picture,” you’ll learn more about the big picture of software development and how software testing fits in. This knowledge is critical to helping you apply the specific test techniques covered in the remainder of this book.

Quiz

These quiz questions are provided for your further understanding. See Appendix A, “Answers to Quiz Questions,” for the answers but don’t peek!

1: In the Year 2000 bug example, did Dave do anything wrong?

2: True or False: It’s important what term your company or team calls problem in its software.

3: What’s wrong with just testing that a program works as expected?

4: How much more does it cost to fix a bug found after the product is released than it does from very start of the project?

5: what’s the goal of a software tester?

6: True or False: A good tester relentlessly strives for perfection.

7: Give several reasons why the product specification is usually the largest source of bugs in a software product.

Chapter 2. The Software Development Process

IN THIS CHAPTER

 Product Components

 Software Project Staff

 Software Development Lifecycle Models

To be an effective software tester, it’s important to have least a high-level understanding of the overall process used to develop software. If you write small programs as a student hobbyist, you’ll find methods you use are much different from what big companies use to develop software. The creation of a new software product may involve dozens, hundreds, even thousands of team members all playing different roles an working together under tight schedules. The specific of what these people do, how they interact, and how they make decisions are all part of the software development process.

The goal of this chapter isn’t to teach you everything about the software development process that would take an entire book! The goal is to give you can overview of the all the pieces that go into a software product and a look at a few of the common approaches in use today. With this knowledge you’ll have a better understand of how best to apply the software testing skills you learn in the later chapters of this book.

The highlights of this chapter include

 What major components go into a software product

 What different people and skills contribute to a software product

 How software progress from an idea to a final product

Product Components

What exactly is a software product? Many of us think of it as simply a program that we download from the Internet or install from DVD that runs on our computer. That’s a pretty good description, but in reality, many hidden pieces go into making that software. There are also many pieces that “come in the box” that are often taken for granted or might even be ignored. Although it may be easy to forget about all those parts, as a software tester, you need to be aware of them, because they’re all testable pieces and all have bugs.

What Effort Goes Into a Software Product?

First, look at what effort goes into a software product. Figure 2.1 Identifies a few of the abstract pieces that you may not have considered.

Figure 2.1 A lot of hidden effort goes into a software product.

So what are all these things, besides the actual code, that get funneled into the software? At first glance they probably seem much less tangible than the program listing a programmer creates. And they definitely aren’t something that can be viewed directly from the product’s CD-ROM. But, to paraphrase a line from an old spaghetti sauce commercial, “they’re in there.” At least, they should be.

The term used in the software industry to describe a software product component that’s created and passed on to someone else is deliverable. The easiest way to explain what all deliverables are organize them into major categories.

Software is written to fulfill some need that a person or group of people has. Let’s call them the customer. To properly fill that need, the product development team must find out what the customer wants. Some teams simply guess, but most collect detailed information in the form of surveys, feedback from previous versions of the software, competitive product information, magazine reviews, focus groups, and numerous other methods, some formal, some not. All this information is then studied, condensed, and interpreted to decide exactly what features the software product should have.

The result of the customer requirements studies is really just raw data. It doesn’t describe the proposed product, it just confirms whether it should (or shouldn’t) be created and what features the customers want. The specifications take all this information plus any unstated but mandatory requirements and truly define what the product will be, what it will do, and how it will look.

The format of specifications varies greatly. Some companies specially those developing products for the government, aerospace, financial, and medical industries use a very rigorous process with many checks and balance. The result is an extremely detailed and thorough specification that’s locked down, meaning that it can’t change except under very extreme conditions. Everyone on the development team knows exactly what they are creating.

The most of specifications varies greatly. Some companies especially those developing products for the government, aero space, financial, and medical industries use a very rigorous process with many checks and balances. The result is an extremely detailed and thorough specification that’s locked down, meaning that it can’t change except under very extreme conditions. Every on the development team knows exactly what they are creating.

There are development teams, usually ones creating software for less-critical applications, who produce specifications on cocktail napkins, if they create them at all. This has the distinct advantage of being very flexible, but there’s lot of risk that not everyone is “on the same page.” And, what the product finally becomes isn’t know until released.

Schedules

A key part of a software product is its schedule. As a project grows in size and complexity, with many pieces and many people contributing to the product, it becomes necessary to have some mechanism to track its progress. This could range from simple task lists to Gantt charts (see Figure 2.2) to detailed tracking of every minute task with project management software.

Figure 2.2. A Gantt chart is a bar that shows a project’s tasks against a horizontal timeline.

The goals of scheduling are to know which work has been completed, how much work is still left to do, and when it will be finished.

One common misconception is that when a programmer creates a program, he simply sits down and starts writing code. That may happen in some small, informal software shops, but for anything other than the smallest programs, there must be a design process to plan out how the software will be written. Think about this book, which required an outline before the first words were typed, or a building, which has blueprints drawn the first concrete is poured. The same planning should happen with software.

The documents that programmers create vary greatly depending on the company, the project, and the team, but their purpose is to plan and organize the code that is to be written.

Here is a list of a few common software design documents:

 Architecture. A document that describes the overall design of the software, including descriptions of all the major pieces and how they interact with each other.

 Data Flow Diagram. A formalized diagram that shows how data moves through a program. It’s sometimes referred to as a bubble chart because it’s drawn by circles and lines.

 State Transition Diagram. Another formalized diagram that breaks the software into basic states, or conditions, and shows the means for moving from one state to the next.

 Flowchart. The traditional means for pictorially describing a program’s logic. Flowcharting isn’t very popular today, but when it’s used, writing the program code from a detailed flowchart is a very simple process.

 Commented Code. There’s an old saying that you may write code once, but it will be read by someone at least 10 times. Properly embedding useful comments in the software code itself is extremely important, so that programmers assigned to maintain the code can more easily figure out what it does and how.

Test documentation is discussed in detail in Chapters 1720 but is mentioned here because it’s integral to what makes up a software product. For the same reasons that programmers must plan and document their work, software testers must as well. It’s not unheard of for a software test team to create more deliverables than the programmers.

Here’s a list of the more important test deliverables:

 The test plan describes the overall method to be used to verify that the software meets the product specification and the customer’s needs. It includes the quality objectives, resource needs, schedules, assignments, methods, and so forth.

 Test cases list the specific items that will be tested and describe the detailed steps that will be followed to verify the software.

 Bug reports describe the problems found as the test cases are followed. These could be done on paper but are often tracked in a database.

 Test tools and automation are described in detail in Chapter 15, “Automated Testing and Test Tools.” If you team is using automated methods to test your software, the tools you use, either purchased or written in-house, must be documented.

 Metrics, statistics, and summaries convey the progress being made as the test work progresses. They take the form of graphs, chart, and written reports.

What Parts Make Up a Software Product?

So far in this chapter you ‘re learned about the effort that goes into creating a software product. It’s also important to realize that when the product is ready to be boxed up and shipped out the door, it’s not just the code that gets delivered. Numerous supporting parts go along with it (see Figure 2.3). Since all these parts are seen or used by the customer, they need to be tested too.

Figure 2.3. The software CD-ROM is just one of the many pieces that make up a software product.

It’s unfortunate, but these components are often overlooked in the testing process. You’ve surely attempted to use a product’s built-in help file and found it to be not so helpful it or worse just plain wrong. Or, maybe you’ve checked the system requirements on a sticker on the side of a software box only find out after you bought it that the software didn’t work on your PC. These seem like simple things to test, but no one probably even gave them a second look before the product was okayed for release. You will.

Later in this book you’ll learn about these non-software pieces and how to properly test them. Until then, keep this list in mind as just a sampling of what more there is to a software product than just the code:

Help files User’s manual

Samples and examples Labels and stickers

Product support info Icons and art

Error messages Ads and marketing material

Setup and installation Readme file

Software Project Staff

Now that you know what goes into a software product and what ships with one, it’s time to learn about all the people who create software. Of course, this varies a great deal based on the company and the project, but for the most part the roles are the same, it’s just the titles that are different.

The following lists, in no particular order, the major players and what they do. The most common names are given, but expect variations and additions:

 Project managers, program managers, or produces drive the project from beginning to end. They’re usually responsible for writing the product spec, managing the schedule, and making the critical decisions and trade-offs.

 Architects or system engineers are the technical experts on the product team. They’re usually very experienced and therefore are qualified to design the overall systems architecture or design for the software. They work very closely with the programmers.

 Programmers, developers, or coders design and write software and fix the bugs that are found. They work closely with the architects and project managers to create the software. Then, they work closely with the project managers and testers to get the bugs fixed.

 Testers or QA (Quality Assurance) Staff are responsible for finding and reporting problems in the software product. They work very closely with all members of the team as they develop and run their tests, and report the problems they find. Chapter 21, “Software Quality Assurance,” thoroughly cover the differences between software testing and software quality assurance tasks.

 Technical writers, user assistance, user education, manual writers, or illustrators create the paper and online documentation that comes with a software product.

 Configuration management or builder handles the process of pulling together all the software written by the programmers and all the documentation created by the writers and putting it together into a single package.

As you can see, several groups of people contribute to a software product. On large teams there may be dozens or hundreds working together. To successfully communicate and organize their approach, they need a plan, a method for getting from point A to point B. that’s what the next section is about.

A running joke in the computer industry is that three things should never be seen in the process of being created: laws, sausage, and software. Their creation process is so messy and disgusting that it’s best to just wait and see the final result. That may or may not be totally true, but with most old sayings, there is a grain of truth behind the words. Some software is developed with the rigor and discipline of a fine craftsman, some software with tightly controlled chaos, and other software is stuck together with duct tape and chewing gum. Usually, in the end, it’s apparent to the customer what process was used. The process used to create a software product from its initial conception to its public release is known as the software development lifecycle model.

As discussed previously, there are many different methods that can be used for developing software, and no model is necessarily the best for a particular project. There are four frequently used models, with most others just variations of these:

 Big-Bang

 Code-and-Fix

 Waterfall

 Spiral

Each model has its advantages and disadvantages. As a tester, you’ll likely encounter them all and will need to tailor your test approach to fit the model being used for your current project. Refer to these model description as you read the rest of this book and think about how you would apply the various testing techniques you learn under each of them.

Big-Bang Model

One theory of the creation of the universe is the big-bang theory. It states that billions of years ago, the universe was created in a single huge explosion of nearly infinite energy. Everything that exists is the result of energy and matter lining up to produce this book, DVDs, and Bill Gates. If the atoms didn’t line up just right, these things might all be just quivering masses of goop.

The big-bang model for software development shown in Figure 2.4 follows much the same principle. A huge amount of matter (people and money) is put together, a lot of energy is expended often violently and out comes the perfect software product…or it doesn’t.

Figure 2.4. The big-bang model is by far the simplest method of software development.

The beauty of the big-bang method is that it’s simple. There is little if any planning, scheduling, or formal development process. All the effort is spent developing the software and writing the code. It’s process that is used if the product requirements aren’t well understood and the final release date is completely flexible. It’s also important to have very flexible customers, too, because they won’t know what they’re getting until the very end.

Notice that testing isn’t shown in Figure 2.4. In most cases, there is little to no formal testing done under the big-bang model. If testing does occur, it’s squeezed in just before the product is released. It’s a mystery why testing is sometimes inserted into this model, but it’s probably to make everyone feel good that some testing was performed.

If you are called in to test a product under the big-bang model, you have both an easy and a difficult task. Because the software is already complete, you have the perfect specification the product itself. And, because it’s impossible to go back and fix things that are broken, your job is really just to report what you find so the customers can be told about the problems.

The downside is that, in the eyes of project management, the product is ready to go, so your work is holding up delivery to the customer. The longer you take to do your job and the more bugs you find, the more contentious the situation will become. Try to stay away from testing in this model.

Code-and-Fix Model

The code-and-fix model shown in Figure 2.5 is usually the one that project teams fall into by default if they don’t consciously attempt to use something else. It’s a step up, procedurally, from the big-bang model, in that it at least requires some idea of what the product requirements are.

Figure 2.5. The code-and-fix model repeats until someone gives up.

A wise man once said, “there’s never time to do it right, but there’s always time to do it over.” That pretty much sums up this model. A team using this approach usually starts with a rough idea of what they want, does some simple design, and then processed into a repeating cycle of coding, testing, and fixing bugs. At some point they decide that enough is enough and release the product.

As there’s very little overhead for planning and documenting a project team can show results immediately. For this reason, the code-and-fix model works very well for small projects intended to be created quickly and then thrown out shortly after they’re done, such as prototypes and demos. Even so, code-and-fix has been used on many large and well-known software products. If your word processor or spreadsheet software has lots of little bugs or it just doesn’t seem quite finished, it was likely created with the code-and-fix model.

Like the big-bang model, testing isn’t specifically called out in the code-and-fix model but does play a significant role between the coding and the fixing.

As a tester on a code-and-fix project, you need to be aware that you, along with the programmers, will be in a constant state of cycling. As often as every day you’ll be given new or updated releases of the software and will set off to test it. You’ll run your tests, report the bugs, and then get a new software release. You may not have finished testing the previous release when the new one arrives, and the new one may have new or changed features. Eventually, you’ll get a chance to test most of the features, find fewer and fewer bugs, and then someone (or the schedule) will decide that it’s time to release the product.

You will most likely encounter the code-and-fix model during your work as a software tester. It’s good introduction to software development and will help you appreciate the more formal methods.

The waterfall method is usually the first one taught in programming school. It’s been around forever. It’s simple, elegant, and makes sense. And, it can work well on the right project. Figure 2.6 shows the steps involved in this model.

Figure 2.6. The software development process flows from one step the next in the waterfall model.

A project using the waterfall model moves down a series of steps starting from an initial idea to a final product. At the end of each step, the project team holds a review to determine if they’re ready to move to the next step. If the project isn’t ready to progress, it stays at that level until it’s ready.

Notice three important things about the waterfall method:

 There’s large emphasis on specifying what the product will be. Note that the development or coding phase is only a single block!

 The steps are discrete; there’s no overlap.

 There’s no way to back up. As soon as you’re on a step, you need to complete the tasks for that step and then move on you can’t go back. [1]

[1] Variations of the waterfall model loosen the rules a bit, allowing some overlap of the steps and the ability to back up one step if necessary.

This may sound very limiting, and it is, but it works well for projects with a well-understood product definition and a disciplined development staff. The goal is to work out all the unknowns and nail down all the details before the first line of code is written. The drawback is that in today’s fast moving culture, with products being developed on Internet time, by the time a software product it so carefully thought out and defined, the original reason for its being may have changed.

Form a testing perspective, the waterfall model offers one huge advantage over the other models presented so far. Everything is carefully and thoroughly specified. By the time the software is delivered to test group, every detail has been decided on, written down, and turned into software. From that, the test group can create an accurate plan and schedule. They know exactly what they’re testing, and there’s no question about whether something is a feature or a bug.

But, with this advantage, comes a large disadvantage. Because testing occurs only at the end, a fundamental problem could creep in early on and not be detected until days before the scheduled product release. Remember from Chapter 1, “Software Testing Background,” how the cost of bugs increases over time? What’s needed is model that folds the testing tasks in earlier to find problems before they become too costly.

It’s not quite utopia, but the spiral model (see Figure 2.7) goes a long way in addressing many of the problems inherent with the other models while adding a few of its own nice touches.

Figure 2.7. The spiral model starts small and gradually expands as the project becomes better defined and gains stability.

The spiral model was introduced by Barry Boehm in 1986 in his Association for Computing Machinery (ACM) paper, “A Spiral of Software Development and Enhancement.” It’s used fairly often and has proven to be an effective approach to developing software.

The general idea behind the spiral model is that you don’t define everything in detail at the very beginning. You start small, define your important feature, try them out, get feedback from your customers, and then move on to the next level. You repeat this until you have your final product.

Each time around the spiral involves six steps:

1. Determine objectives, alternatives, and constraints.

2. Identify and resolve risks.

3. Evaluate alternatives.

4. Develop and test the current level.

5. Plan the next level.

6. Decide on the approach for the next level.

Built into the spiral mode is a bit of waterfall (the steps of analysis, design, develop, test), a bit of code-and-fix (each time around the spiral), and a bit of big-bang (look at it from the outside). Couple this with the lower costs of finding problems early, and you have a pretty good development model.

If you’re a tester, you’ll like this model. You’ll get a chance to influence the product early by being involved in the preliminary design phases. You’ll see where the project has come from and where it’s going. And, at the very end of the project, you won’t feel as rushed to perform all your testing at the last minute. You’re been testing all along, so the last push should only be a validation that everything is okay.

You now have an understanding of how software products are create both goes into them and the processes used to put them together. As you can see, there’s no definitive approach. The four models presented here are just examples. There are many others and lot of variations of these. Each company, each project, and each team will choose what works for them. Sometimes they will choose right, sometimes they will choose wrong. Your job as a software tester is to work the best you can in the development model you’re in, applying the testing skills you learn in the rest of this book to create the best possible.

Quiz

These quiz questions are provided for your further understanding. See Appendix A, “Answers to Quiz Questions,” for the answers but don’t peek!

在G J Myers的經典著作《軟件測試之藝術》（The Art of Software Testing）中，給出了軟件測試（Software Testing）的定義：“程序測試是為了發現錯誤而執行程序的過程”。這個定義，被業界所認可，經常被引用。除此之外，G J Myers還給出了與測試相關的三個重要觀點，那就是：

一個好的測試用例是在於它能發現至今未發現的錯誤；

一個成功的測試是發現了至今未發現的錯誤的測試。

實際上，這裏暗示了“軟件測試”在不同側面上的含義，也就決定了對軟件測試不同的定義和不同的理解，軟件測試的不同視野，概括為如下5類：

軟件測試的辨證論——正向思維和反向思維

軟件測試的風險論——測試是評估

軟件測試的經濟學觀點——為盈利而測試

軟件測試的標準論——驗證和確認

G.J.Myers所給出了測試定義——“程序測試是為了發現錯誤而執行程序的過程”，實際是一個狹義的概念，因為他認為測試是執行程序的過程，也就是傳統意義上的測試——在代碼完成後，通過運行程序來發現程序代碼或軟件系統中錯誤。但是，這種意義上的測試是不能在代碼完成之前發現軟件系統需求、發現設計上的問題，把需求、發現設計上的問題遺留到後期，這樣就會可能造成設計、編程的部分返工。增加軟件開發的成本、延長開發的週期等。需求階段和設計階段的缺陷產生的放大效應會加大。這非常不利於保證軟件質量。這種狹義論是受軟件開發瀑布模型影響。

正是為了更早地發現問題，所以將測試延伸到需求評審、設計審查活動中去，也就是將“軟件質量保證”的部分活動歸為測試活動。實際上，在軟件開發實際操作中，常常將軟件測試和質量保證——這兩種努力（efforts）合併起來。

延伸後的軟件測試，被認為是一種軟件測試的廣義概念。這就引出軟件測試的兩個概念“靜態測試”和“動態測試”，如 測試方法的辯證統一 （1）所述，這樣就由靜態測試和動態測試構成一個全過程的、完整的軟件測試，而且靜態測試顯得更為重要。

G.J.Myers的第2個觀點“測試是為了證明程序有錯，而不是證明程序無錯誤”，引出了軟件測試的另外一個爭論，軟件測試究竟是證明所有軟件的功能特性是正確的呢？還是其反向思維——對軟件系統進行各種試探和攻擊，找出軟件系統中不正常或不工作的地方呢？從我個人理解，這兩個方面都有一定道理，前者（證明所有軟件的功能特性是正確的）是從質量保證的角度來思考軟件測試，後者（證明程序有錯）從軟件測試的直接目標和測試效率來思考，兩者應該相輔相成。在後者的思想背景下，我們認為，測試不是為了證明所有的功能可以正常工作，恰恰相反，測試就是為了找出那些不能正常工作、不一致性的地方。也就是説，測試的一般工作就是發現缺陷 (detect bug)，即在軟件開發過程中，分析、設計與編碼等工作都是建設性的，而測試是帶有“破壞性”的工作。

對於不同的應用領域，兩者的比重是不一樣的，如國防、航天、銀行等軟件系統，承受不了任何系統失效，因為一次系統的失效完全有可能導致災難性的損失，所以強調前者以保證非常高的軟件質量。而一般的軟件服務應用則不同，強調後者，質量目標設置在“用户可接受水平”，不要國度追求質量，從而可以降低軟件開發成本。作者建議，在我們實際操作中，可以分階段實施不同的測試思想，在早期階段集中在“證明程序有錯”—— 發現Bug，後期集中在驗證所有特性是否正常工作——降低風險，見作者的另外一篇討論：測試執行中非常有效的策略

驗證軟件是驗證軟件是“工作的”，以正向思維，針對軟件系統的所有功能點，逐個驗證其正確性。其代表人物是軟件測試領域的先驅Dr. Bill Hetzel （代表論著《The Complete Guide to Software Testing》）。

證明軟件是“不工作的”，以反向思維方式，不斷思考開發人員理解的誤區、不良的習慣、程序代碼的邊界、無效數據的輸入以及系統的弱點，試圖破壞系統、摧毀系統，目標就是發現系統中各種各樣的問題。其代表人物就是上面多次提到的G.J.Myers。他強調，一個成功的測試必須是發現Bug Bug的測試，不然就沒有價值。

測試被定義為“對軟件系統中潛在的各種風險進行評估的活動”，這就是軟件測試的風險論。軟件測試自身的風險性是大家公認的，測試的覆蓋度不能做到100%。測試的這種風險定義一方面源於這層含義，另外軟件測試的標準有時不清楚，“軟件規格説明書（Specification/ Spec）”是其中的一個標準，但也不是唯一的，因為Spec中有些內容完全有可能是錯誤的。所以，我們常常強調軟件測試人員應該站在客户的角度去進行測試，除了發現程序中的錯誤，還要發現需求定義的錯誤、設計上的缺陷，可以針對Spec 去報Bug。但是，測試在大多數時間/情況下,是由工程師完成，而不是客户自己來做，所以又怎麼能保證工程師和客户想得一樣呢？

有人把開發比作打靶，目標明確，就是按照Spec 去實現系統的功能。而把測試比作撈魚，目標不明確，自己判斷哪些地方魚多，就去哪些地方撈；如果只撈大魚（嚴重缺陷），網眼就可以大些、撒網區域相對比較集中（測試點集中在主要功能-major features）。如果想把大大小小的魚撈上來，網眼就要小、普遍撒網，不放過任何一塊區域（測試點遍及所有功能——all features）。

在“風險”論的框架下，軟件測試可以被看作是一個動態的監控過程，對軟件開發全過程進行檢測，隨時發現不健康的徵兆，發現問題、報告問題，並重新評估新的風險，設置新的監控基準，不斷地持續下去，包括迴歸測試。這時，軟件測試可以完全看作是軟件質量控制的過程。

對應這種觀點，產生基於風險的測試策略，首先評估測試的風險，功能出問題的概率有多大？哪些是用户最常用的20%功能——Pareto原則（也叫80/20原則）？如果某個功能出問題，其對用户的影響有多大？然後根據風險大小確定測試的優先級。優先級高的測試，優先得到執行，一般來講，針對用户最常用的20%功能（優先級高）的測試會得到完全執行，而低優先級的測試（另外用户不經常用的80%功能）就不是必要的，如果時間或經費不夠，就暫時不做或少做。

“一個好的測試用例是在於它能發現至今未發現的錯誤”，體現了軟件測試的經濟學觀點。實際上，軟件測試經濟學問題至今仍是業界關注的問題之一。經濟學的核心就是要盈利，盈利的基礎就是要有一個清楚的商業性目標。同樣，商業性目標是否正確，直接決定了企業是否盈利的結果。多數情況下，軟件測試是在公司內的執行。正是公司的行為目的，決定了軟件測試含義或定義的經濟性一面。正如，對軟件質量的定義不僅僅局陷於“和客户需求的一致性、適用性”，而且要增加其它的要求——“預算內、按時發佈、易於維護”。

軟件測試也一樣，要儘快儘早地發現更多的缺陷，並督促和幫助開發人員修正缺陷。原因很簡單：平均而言，如果在需求階段修正一個錯誤的代價是1，那麼，在設計階段就是它的3～6倍，在編程階段是它的10倍，在內部測試階段是它的20～40倍，在外部測試階段是它的30～70倍，而到了產品發佈出去時，這個數字就是 40～ 1000倍。修正錯誤的代價不是隨時間線性增長，而幾乎是呈指數級增長的。

如果從標準論來看軟件測試，可以定義為軟件測試就是“驗證（Verification）”和“有效性確認（Validation）”活動構成的整體，即軟件測試 = V&V。

“驗證”是檢驗軟件是否已正確地實現了產品規格書所定義的系統功能和特性。驗證過程提供證據表明軟件相關產品與所有生命週期活動的要求（如正確性、完整性、一致性、準確性等）相一致。相當於，以Spec為標準進行軟件測試活動，驗證軟件產品和Spec的一致性。

“有效性確認”是確認所開發的軟件是否滿足用户真正需求的活動。相當於，保持對軟件需求定義、設計的懷疑，一切從客户出發，理解客户的需求，發現需求定義和產品設計中的問題。這主要通過各種軟件評審活動來實現。

需要説明的是，軟件測試的對象是產品（包括階段性產品，如市場需求説明書、產品規格説明書、技術設計文檔、數據字典、程序包、用户文檔等），而質量保證和管理的對象集中在軟件開發的標準、流程和方法等。