Testing Types

Testing Types
TABLE OF CONTENTS
1.0 General: 3
Module Objectives: 3
Module Structure: 3
2.0 Testing Types 4
Black Box Testing 4
White Box Testing: 5
3.0 Structural Vs Functional 7
Overview of Structural Testing 7
Overview of Functional Testing 7
Structural Vs Functional 8
4.0 Regression Testing 9
What is regression testing 9
Types of regression testing 9
How to select test cases for regression testing 9
Resetting the test cases for regression testing 11
How to conclude the results of a regression testing 12
Can we apply the regression test guidelines for patch/upgrade releases 13
How do I find out which test case to be executed for a particular defect fix 13
5.0 Performance Testing 14
What is Performance Testing? 14
Factors Influencing for Performance Test 15
Performance Testing Approach 16
Flow Diagram for Performance Engineering 17
Performance Testing Approach Summary 18
- Defines tasks. 18
Exemplar Scenario for Performance Testing 18
Tools for Performance Testing 21
6.0 Installation Testing 22
What Is Installation Testing 22
Flow Chart for Installation Testing 22
Who & How to do Installation Testing 23
What to Check in Installation Testing 23
7.0 Compatibility Testing 24
Process for Compatibility 25
Identification of Parameters: 25
Combination of Parameters to Test 26
Test Case Identification: 26
Setup Management 27
Ensuring Pre-requisites 27
Execution of Test: 27
What is Ghosting ? 29
8.0 Acceptance 30
Acceptance Criteria for Requirements Phase 30
Acceptance Criteria for Planning Phase 30
Acceptance Criteria for Design Phase 30
Acceptance Criteria for Execution Phase 30
9.0 Manual vs. Automation 31
Manual Testing 31
Automation Testing 31
Flow diagram to determine Test scenarios for Manual and Automation 33
Summary of Manual Vs Automated Testing 34
Why & When One Should Consider Automation 34
A Sensible Approach to Automation 36
10.0 Unit summary 37
Exercise 37

1.0 General:

Testing as such is a very wide domain, with its own complex and lengthy processes.
When a project is taken up for testing, deciding the kind of testing to be performed is a major task for the test engineers.
Typically the decision of “Kind of testing” , is either decided by the Management or the client. In most of the real-time scenarios, client specifies the kind of testing to be done for the given project.
Depending on the testing requirement the project managers need to decide the testing strategy for the product, and prepare the schedule accordingly.
The test engineers need to be prepared for taking up any kind of testing on the product and hence needs to know the processes to be followed to perform the given type of testing.
Each type of testing requires a different approach to be adopted and the corresponding testing techniques need to be followed for the success of the project.
The following Module explains in detail the various kinds of testing and the techniques and processes for the same.
The participants are suggested to go through the module carefully and understand the concepts in detail through practice and discussion.
1.1 Module Objectives:
At the end of this module, you should:

 Be able to define various kinds of testing
 Understand various processes to be followed for different types of testing
 Understand the difference between Manual and automated testing approaches.

1.2 Module Structure:

SNO Topic Duration
1 Introduction to testing types, Black box Testing, white box Testing 5 hours
2 Structural VS Functional Testing 5 hours
3 Regression Testing 5 hours
4 Installation Testing 5 hours
5 Performance Testing 5 hours
6 Compatibility Testing 5 hours
7
Acceptance Testing 5 hours
8 Manual VS Automated Testing 5 hours
Total Duration 40 hrs
2.0 Testing Types
There are several types of software testing that are widely used in today's IT world. When a tester knows what type of testing is needed it greatly improves the test results thereby decreasing the number of defects identified.
Testing can be conducted in different kinds like Manual Testing, Automation Testing, Memory Testing, Performance Testing, Usability Testing, etc. The different kinds of testing are again broadly classified into two major sectors called the “Black Box Testing” and the “White Box Testing”. We shall look into some of the commonly used testing methodologies of White Box testing and Black Box testing here.
2.1 Black Box Testing
Black Box testing is the testing all the features and functionalities of the final product. It is also called as Closed Box Testing. The Tester has no information about the internal architecture or functional mechanism of the product. Black Box testing is a wide range and has many sub kinds
2.1.1 Acceptance Testing:
When the product is delivered for testing, it is first checked whether the product is in a functional condition to test. This is called Acceptance Testing. The Testers can generate an Acceptance Test Case, which can be run for Acceptance Testing.
2.1.2 Ad Hoc Testing
This method is also called Exploratory Testing where in no Test Case designs are used to test the product. The Tester goes by his intuition and creativity in finding out problems with the product.
2.1.3 Automation Testing:
This method of testing is done using third party tools. In this method, we automate, or run a particular Test Case for a specific number of times. This helps in finding out the consistent passing of the test case and also helps in finding our problems related to memory, Performance etc. More about this method is dealt in detail later.
2.1.4 Boundary Value Testing
Certain features will require inputs to be taken from the user, like Login name in a site. There will also be a lower end limit and higher end limit to the input value to be given. In this kind of testing, we check the feature using values just below or above the lower limit and the upper limit.
2.1.5 Compatibility Testing
Compatibility Testing confirms that the Product installed does not hinder the functionality of any other product installed on the machine. Products are said to be compatible with each other if they can share data between them or if they can simultaneously reside in the same computers memory. In short Compatibility testing checks that one product works with another.
2.1.6 Integration Testing
This testing is done when the different modules of the product have been integrated together. It is an orderly process, which is carried out until all the modules have been integrated into the system.

2.1.7 Manual Testing
That part of software testing that requires operator input, analysis, or evaluation.
2.1.8 Performance Testing
The tests might determine which modules execute most often or use the most computer time. Then those modules are re-examined and recorded to run more quickly.
2.1.9 Regression Testing
This process of testing is done when new features are added or when features have been modified to confirms that the features that have been functioning are still functioning and that defects which has been fixed still remain fixed.

2.1.10 Usability Testing
Usability testing is done to check the flow of the product and the user friendliness of the product. It helps in finding out whether the user is able to interact with the product and achieve his goal. People who are novice to the product usually do this kind of test, and the Testers and Developers study the user while he is doing the process.

2.1.11 Installation Testing
Here the installation media and different kinds of installation process on different Operating systems with different configurations are tested. What good is a product if it does not install properly on different computer configurations?
2.2 White Box Testing:
White box testing involves testing at the Code Level and is done usually at the coding stage. Here the inputs required for testing is fed to the program and the output is checked. The Tester does not know or pretends not to know how the program works. There are a lot of different kinds of White Box testing. Some of the major methods are mentioned below.
2.2.1 Boundary Value Testing
This method can be used as black box and as white box. The feature tested here is similar to Black Box testing as mentioned earlier. (See Black Box testing section)

2.2.2 Branch Testing
This testing is done to satisfy the coverage criteria such that for each decision point each possible branch be executed at least once. For example if the execution reaches a Case statement, this kind of testing should cover each of the Case conditions at least once.

2.2.3 Top-Down and Bottom-Up Testing
In the Top-Down testing the highest level modules are tested first whereas the Bottom -up testing, the lower modules are tested first, and then the higher level modules.

2.2.4 Hybrid Testing
A combination of top-down testing combined with bottom-up testing of prioritized or available components.
2.2.5 Incremental Testing
Under this each piece of a module is first tested separately. This testing makes it easy to pin down the cause of the error but to test it requires a special code. Each piece is individually tested with focus on it and is thoroughly tested.
2.2.6 Special Case Testing
This kind of testing can be conducted as both Black Box testing and White Box testing. Here testing is done by giving input values that seem likely to cause program errors like "0", "1", NULL, empty string.
2.2.7 Statement Testing
Testing that satisfies that each statement in a program is executed at least once during program testing. It is said, “An untested code is a hidden Bomb”. It is true in the sense that if it is not tested, we can never say whether it will function properly or throw up errors.

Some of the other testing methods and terminologies used in the Testing field are:
Alpha Testing, Beta Testing, Assertion Testing, Big Bang Testing, Design-Based Testing, Development Testing, Error Based Testing, Equivalence Testing, etc.



3.0 Structural Vs Functional
Testing Goal is to evaluate the system as a whole, not its parts
 Techniques can be structural or functional
 Techniques can be used in any stage that tests the system as a whole (acceptance, installation, etc.)
 Techniques not mutually exclusive
3.1 Overview of Structural Testing
Structural techniques

 Stress testing - Test larger-than-normal capacity in terms of transactions, data, users, speed, etc.
 Execution testing - Test performance in terms of speed, precision, etc.
 Recovery testing - Test how the system recovers from a disaster, how it handles corrupted data, etc.
 Operations testing - Test how the system fits in with existing operations and procedures in the user organization
 Compliance testing - Test adherence to standards
 Security testing - Test security requirements
3.2 Overview of Functional Testing
Functional techniques
 Requirements test - fundamental form of testing - makes sure the system does what it’s required to do
 Regression testing - Make sure unchanged functionality remains unchanged
 Error-handling testing - Test required error-handling functions (usually user error)
 Manual-support testing - Test that the system can be used properly - includes user documentation
 Intersystem handling testing - Test that the system is compatible with other systems in the environment
 Control testing - Test required control mechanisms
 Parallel testing - Feed same input into two versions of the system to make sure they produce the same output









3.3 Structural Vs Functional

STRUCTURAL TESTING TECHNIQUES
FUNCTIONAL TESTING TECHNIQUES

• “White box” testing
• Based on statements in the code
• Coverage criteria related to physical parts of the system
• Tests how a program/system does something
• “Black box” testing
• Based on input and output
• Coverage criteria based on behavior aspects
• Tests the behavior of a system or program





4.0 Regression Testing
4.1 What is regression testing
Regression testing is selective retesting of the system with an objective to ensure the bug fixes work and those bug fixes have not caused any un-intended effects in the system.
4.2 Types of regression testing
There are two types of regression testing that are proposed here even though it is not being practiced or popular.
A "final regression testing" is being done to validate the gold master builds and "Regression testing" being done to validate the product & failed test cases between system test cycles.
The final regression test cycle is conducted on an "unchanged build for a period of x days" or for a period, which was agreed as the "cook-time" for release. The product is continuously exercised for the complete duration of this cook-time. Some of the test cases are even repeated to find out whether there are failures in the final product that will reach the customer. All the bug fixes for the release should have been completed for the build used for the final regression test cycle. The final regression test cycle is more critical than any other type or phase of testing, as this is the only testing which ensures "the same build of the product that was tested reaches the customer".
A normal regression testing can use the builds for a period that is exactly needed for the test cases to be executed. However unchanged build is highly recommended for each cycle of regression testing.

4.3 How to select test cases for regression testing

It was found that some of the defects reported by customers in the past were due to last minute bug fixes creating side effects and hence selecting the test case for regression testing is really an art and not that easy.
The selection of test cases for regression testing
a. Requires knowledge on the bug fixes and how it affect the system
b. Includes the area of frequent defects
c. Includes the area which has undergone many/recent code changes
d. Includes the area which is highly visible to the users
e. Includes the core features of the product which are mandatory requirements of the customer

Selection of test cases for regression testing depends more on the criticality of bug fixes than the criticality of the defect itself. A minor defect can result in major side effect and a bug fix for an Extreme defect can have no or a minor side effect. So the test engineer needs to balance these aspects for selecting the test cases for regression testing.

When selecting the test cases we should not select more test cases, which are bound to fail and has no or less relevance to the bug fixes. You need to select more positive test cases than negative test cases for final regression test cycle as this may create some confusion and unexpected heat. It is also recommended that the regular test cycles before regression testing should have right mix of both positive and negative test cases. Negative test cases here I mean those test cases which are introduced newly with an intent to break the system.

It is noticed that several companies have "constant test cases set" for regression testing and they are executed irrespective of the number and type of bug fixes. Sometimes this approach may not find all side effects in the system and in sometimes it may be observed that the effort spend on executing test cases for regression testing can be minimized if some analysis is done to find out what test cases are relevant and what are not.
It is a good approach to plan and act for regression testing from the beginning of project before the test cycles. One of the ideas is to classify the test cases into various Priorities based on importance and customer usage. Here I am suggesting the test cases be classified into three categories;

• Priority-0 – Sanity test cases which checks basic functionality and are run for pre-system acceptance and when product goes thru major change. These test cases deliver a very high project value to both engineering dept and to customers.

• Priority-1 – Uses the basic and normal setup and these test cases deliver high project value to both engineering and to customers.

• Priority-2 – These test cases deliver moderate project value. Executed part of ST cycle and selected for regression testing on need basis.

There could be several right approaches to regression testing which needs to be decided on "case to case" basis;

• Case 1: If the criticality and impact of the bug fixes are LOW, then it is enough a test engineer selects a few test cases from TCDB and executes them. These test cases can fall under any Priority (0, 1 or 2).

• Case 2: If the criticality and the impact of the bug fixes are Medium, then we need to execute all Priority-0 and Priority-1 test cases. If bug fixes need additional test cases from Priority-2, then those test cases can also selected and used for regression testing. Selecting Priority-2 test cases in this case is desirable but not a must.

• Case 3: If the criticality and impact of the bug fixes are High, then we need to execute all Priority-0, Priority-1 and carefully selected Priority-2 test cases.

• Case 4: One can also go thru the complete log of changes happened (can be obtained from CM engineer) because of bug fixes and select the test cases to conduct regression testing. This is an elaborate process but can give very good results.








4.4 Resetting the test cases for regression testing
In a big product release involving several rounds of testing, it is very important to note down what test cases were executed with what build and related information. This is called test case result history. In many organizations not all types of testing and all test cases were repeated for each cycle. In such cases resetting the test cases become very critical for the success of regression testing. Resetting a test case is nothing but setting a flag called NOTRUN or EXECUTE AGAIN with zero base thinking.

RESET of test case, are not expected to be done often. Resetting of the test cases needs to be done with following considerations;
a. When there is a major change in the product
b. When there is a change in the build procedure which affect the product
c. Large release cycle where some test cases were not executed for a long time
d. You are in the final regression test cycle with a few selected test cases
e. Where there is a situation the expected results of the test cases could be quite different from previous cycles

When the above guidelines are not met, you may want to RERUN the test cases rather than resetting the results of the test cases. There are only few differences between RERUN and RESET states in test cases, either way the test cases are executed but in case of RESET one has to think zero base and expect different result than what was obtained in earlier cycles and therefore those test cases affect the completion rate of testing. In case of RERUN the management need not worry about completion rate as those test cases can be considered complete except for a formality check and are expected to give same results.

RESET is also decided based on how stable the functionalities are. If you are in Priority-1 and have reached a stage of comfort level on Priority-0 (say for example more than 95% pass rate) then you don't RESET Priority-0 test cases unless there is a major change. This is true with Priority-1 test cases when you are in Priority-2 test phase.

Pre-system test cycle phase
For pre-system acceptance only Priority-0 test cases are used. For each build that is entering the system test, the build number is selected and all test cases in Priority-0 are reset to NOT RUN. The system test cycle starts only if all pre-system test cases (Priority-0) pass. Test manager or CCB, can decide exceptions if any.

System test cycle – Priority-1 testing phase
After pre-system acceptance is over, Priority-1 test cases are executed. Priority-1 testing can use multiple builds. In this phase the test cases are RESET only if the criticality and impact of the bug fixes and feature additions are high. A RESET procedure during this phase may affect all Priority-0 and Priority-1 test cases and these test cases are reset to NOTRUN in TCDB.

System test cycle – Priority-2 testing phase
Priority-2 testing starts after all test cases in Priority-1 are executed with an acceptable pass % as defined in test plan. In this phase several builds are used. In this phase the test cases are RESET only if the criticality and impact of the bug fixes and feature additions are very high. A RESET procedure during this phase may affect Priority-0, Priority-1 and Priority-2 test cases.

In what way regression testing is related to above three phases?
Regression testing is normally done after Priority-2 testing or for the next release involving only few changes. Resetting test cases during the above phases are not called as regression testing as in my assumption regression comes into picture only after the product is stable. A testing for a release can be decided either by saying a regression testing is sufficient or we can do all phases of testing starting from Priority-0 to Priority-2.
A regression testing for a release can use test cases from all priorities (as mentioned before). A regression testing involving multiple priorities of test cases also needs the test cases executed in strict order i.e. Priority-0 test cases are executed first, Priority-1 next and Priority-2 test cases.

Why we need to RESET the test cases?
Regression testing uses good number of test cases, which would have been executed already and associated with some results and assumptions on the result. A RESET procedure makes them to NOTRUN so that it gives a clear picture about how much of testing is still remaining, and reflect the results of the regression testing on Zero base.

If test cases are not RESET, then the test engineers tend to report a completion rate and other results based on previous builds. This is because of the basic assumption that multiple builds can be used in each phase of the testing and a gut-feel that if something passed in the past builds, it will pass in future builds also. Regression testing doesn't go with an assumption that "Future is an extension of the past".
4.5 How to conclude the results of a regression testing

Regression testing uses only one build for testing (if not it is strongly recommended). It is expected that all 100% of those test cases pass using the same build. In situations where the pass % is not 100, the test manager can look at the previous results of the test case to conclude the expected result;
a. If the result of a particular test case was PASS using the previous builds and FAIL in the current build, then regression failed. We need to get a new build and start the testing from scratch after resetting the test cases.
b. If the result of a particular test case was a FAIL using the previous builds and a PASS in the current build, then it is easy to assume the bug fixes worked.
c. If the result of a particular test case was a FAIL using the previous builds and a FAIL in the current build and if there are no bug fixes for this particular test case, it may mean that the result of this test case shouldn't be considered for the pass %. This may also mean that such test cases shouldn't be selected for regression.
d. If the result of a particular test case is FAIL using the previous builds but works with a documented workaround and
a. if you are satisfied with the workaround then it should considered as PASS for both system test cycle and regression test cycle
b. If you are not satisfied with the workaround then it should be considered as FAIL for a system test cycle but can be considered as PASS for regression test cycle.





4.6 Can we apply the regression test guidelines for patch/upgrade releases

The regression guidelines are applicable for both cases where
a. You are doing a major release of a product, executed all system test cycles and planning a regression test cycle for bug fixes.
b. You are doing a minor release of a product (CSPs, patches …etc) having only bug fixes, and you are planning for regression test cycles to take care of those bug fixes.

There can be multiple cycles of regression testing that can be planned for each release, if bug fixes come in phases or to take care of some bug fixes not working with specific build.
4.7 How do I find out which test case to be executed for a particular defect fix

When failing a test case it is a good practice to enter the defect number(s) along so that you will know what test cases to be executed when a bug fix arrives. Please note that there can be multiple defects that can come out of a particular test case and a particular defect can affect more than one test case.

Even though, it is easy to do the mapping between test cases and defects using these mechanisms, the test cases that are to be executed for taking care of side effects of bug fixes, may remain as a manual process as this requires knowledge.

5.0 Performance Testing
5.1 What is Performance Testing?

Performance Testing is a critical component of entire Testing Process, It determines the actual operational boundaries and will stimulate the real world use of application.

Performance Testing can be divided into:
- Load Testing
- Stress Testing
- Scalability Testing

5.1.1 Load Testing
Load testing is subjecting a system to a statistically representative (usually) load, main reason for using Load is to support software reliability testing and in performance testing

Load Testing determines the systems behavior under various workloads.
Objective of how the systems components react as the workload are gradually increased.
Usual Outcome is determination of system performance
-- Throughput
-- Response Time
-- CPU Load
-- Memory Usage

5.1.2 Stress Testing
Stress tests are designed to confront programs with abnormal situations. ... Stress testing executes a system in a manner that demands resources in abnormal quantity, frequency, or volume.

 Testing in which a system is subjected to unrealistically harsh inputs or load with inadequate resources with the intention of breaking it
 Testing conducted to evaluate a system or component at or beyond the limits of its specified requirements. See also: boundary value.
 Stress testing is subjecting a system to an unreasonable load while denying it the resources needed to process that load (e.g., RAM, disc, interrupts, etc.)



5.1.3 Scalability Testing
Scalability testing evaluates the effect of adding additional hardware and / or software to distribute “work” among systems components
Tests can be performed in variety of configurations with such variables as network speed, number and type of servers/CPU’s memory etc.
By increasing application’s workloads can determine overall flexibility and ability to scale for workload growth.
5.2 Factors Influencing for Performance Test

(a) Speed - Does the application respond quickly enough for the intended users?
Speed - User Expectations
- Experience
- Psychology
- Usage
Speed - System Constraints
- Hardware
- Network
- Software
Speed - Costs
Speed can be expensive

(b) Scalability – Will the application handle the expected user load and beyond?
How many users…
- Before it gets “slow”?
- Before it stops working?
- Will it sustain?
How much data can it hold?
- Database capacity
- File Server capacity
- Back-up Server capacity
- Data growth rates

(c) Stability – Is the application stable under expected and unexpected user loads?
What happens if…
- If, there are more users than we expect?
- All the users do the same thing?
- A user gets disconnected?
- There is a Denial of Service Attack?
- The web server goes down?
- We get too many orders for the same thing?

(d) Confidence – Are you sure that users will have a positive experience on go-live day?
If you know what the performance is…
- You can assess risk.
- You can make informed decisions.
- You can plan for the future.
5.3 Performance Testing Approach
Evaluate System
- Determine performance requirements.
- Identify expected and unexpected user activity.
- Determine test and/or production architecture.
- Identify non-user-initiated (batch) processes.
- Identify potential user environments.
- Define expected behavior during unexpected circumstances.
Develop Test Assets
- Create Strategy Document.
- Develop Risk Mitigation Plan.
- Develop Test Data.
- Automated test scripts:
- Plan
- Create
- Validate
Baseline and Benchmarks
- Most important for iterative testing. Baseline (single user) for initial basis of comparison
- Benchmark (15-25% of expected user load) determines actual state at loads expected
to meet requirements.
Analyze Results
Most important & most difficult focuses on:
- Have the performance criteria been met?
- What are the bottlenecks?
- Who is responsible to fix those bottlenecks?
- Decisions.

Tune System
- Engineering only, highly collaborative with development team and highly iterative.
- Usually, performance engineer ‘supports’ and ‘validates’ while developers / administrators

Identify Exploratory Tests
- Engineering only, exploits known bottleneck.
- Assists with analysis & tuning.
- Significant collaboration with ‘tuners’.
- Not robust tests – quick and dirty, not often reusable or relevant after tuning is complete.

Validate Requirements
- Only after Baseline and/or Benchmark tests.
- These tests evaluate compliance with documented requirements.
- Often are conducted on multiple hardware/configuration variations.

Complete Engagement
Document: Package Test Assets:
- Actual Results - Scripts
- Tuning Summary - Documents
- Known bottlenecks not tuned - Test Data
- Other supporting information
- Recommendation


5.4 Flow Diagram for Performance Engineering


5.5 Performance Testing Approach Summary
- Ensures goals are accomplished.
5.6 - Defines tasks.
- Identifies critical decision points.
- Shortens testing lifecycle.
- Increases confidence in results.

5.7 Exemplar Scenario for Performance Testing
5.7.1 Pre-Project
1. How many users (human and system) need to be able to use the system concurrently?
a. What is the total user base?
b. What is the projected acceptance rate?
c. How are the users distributed across the day/week/month?
Scenario
Assuming evenly distributed billing cycles throughout the month, users spending about 15 min each viewing and paying their bill, and the site is generally accessed between 9AM EST and 6PM PST (15 hours). Calculated concurrent users, with following formula

(Total monthly users) / (30 days a month * 15 hours a day * 4
{Note: 60min / 15min per user} = daily average concurrent user load.
Normally test to 200% of daily average concurrent user load.

If, 1 million monthly users

1,000,000 / (30*15*4) = 556 concurrent users. (2,222 hourly users)
Recommend testing up to 1000 concurrent users (4,000 hourly users).

2. General performance objectives
a. Can service up to a peak of Customers an hour without losing customers
due to performance issues?
b. System stable and functional under extreme stress conditions?
3. What are the project boundaries, such as:
a. Are all bottlenecks resolved Or..
b. Best possible performance in months Or...
c. Continue tuning until goals are met Or...
d. Continue tuning until bottlenecks deemed "unfixable" for current release?

5.7.2 Pre-Testing
1. What are the specific/detailed performance requirements?
a. User Experience (preferred) - i.e. With 500 concurrent users, a user accessing
the site over a LAN connection will experience not more than 5 seconds to display
a small or medium bill detail report 95% of the time
b. Component Metric (not recommended) - i.e. Database server will keep memory
Usage under 75% during all tested user loads
2. Detailed description of the test and production environments.
a. Associated risks if environments are not identical
b. How to best mark up performance if not identical

3. What is the availability of system administrators/developers/architects?

4. What is the test schedule?
a. Can tests be executed during business hours?
b. Must tests be executed during off hours?
c. Must tests be executed during weekends?

5. Are system monitoring tools already installed/being used on the systems under test
a. Who will monitor/evaluate monitoring tools?

6. Are any other applications/services running on the systems to be tested.
a. Think about associated risks to shared environments, like memory, disk I/O,
drive space, etc.

7. What are of the types of tests/users/paths desired
a. Target for 80% of user activity
b. Always model system intensive activity

8. Based on answers, create Test Strategy Document









5.7.3 Test Design/Execution

1. Design tests to validate requirements
a. Create User Experience Tests
b. Generate loads and collect Component Metric data while under load.
2. Always Benchmark application in a known environment first
a. Architecture need not match
b. Benchmark tests do not need to follow user community model exactly
c. Benchmark tests should represent about 15% of the expected peak load
d. Look for problems .low hanging fruit.
e. Do not focus on maximizing User Experience, just look for show stopping bottlenecks
3. Benchmark in Prod environment if possible
a. Look for problems with code/implementation
b. Look for problems out of scope that client must fix to meet performance goals
(i.e. Network, Architecture, Security, Firewalls)
c. This benchmark must be identical to the benchmark conducted in the known environment (taking into account difference in Architecture)
d Do not focus on maximizing User Experience, just look for show stopping bottlenecks
4. Load Test
a. Iteratively test/tune/increase load
b. Start with about the same load as benchmark, but accurately depicting user
c. Do not tune until critical (primary) bottleneck cause is found do not tune symptoms, do not tune components that could be faster.
d. Re-test after each change to validate that it helped . if it didn.t, change it back and try the next most likely cause of the bottleneck. Make a note of the change in case it becomes the primary bottleneck later.
e. If bottlenecks are environment related (i.e. out of scope) document the bottleneck and present to PM for guidance. Stop testing until client/PM agree on approach.
f. *Note* If you need performance improved by 200% to reach the goal, tuning a method to execute in .2 seconds instead of .25 seconds will not fix the problem.

5.7.4 Results Reporting
1. Report test results related to stated requirements only.
a. Show summarized data validating requirements
b. If requirements are not met, show data as to why and what needs to be
fixed / who needs to fix it by when
c. Show areas of potential future improvement that are out of scope
d. Show summary of tuning/settings/configurations if it adds value
e. Be prepared to deliver formatted raw data as back-up


5.8 Tools for Performance Testing

 Microsoft’s Web application Stress Tool
 Cyrano’s Open STA
 Empirix’s E-Test Suite 6.0
 Radview’s WebLoad 5.0
 Rational Software’s Rational Robot
 Mercury Interactive’s Astra LoadTest 5.4
 Compuware’s QA Load 4.7
 Segue Software’s Silk Performer 5.0





























6.0 Installation Testing

6.1 What Is Installation Testing

Installation of product for its behavior at various adverse conditions, this type of testing is performed to ensure that all Install features and options function properly. It is also performed to verify that all necessary components of the application are, indeed, installed.

6.2 Flow Chart for Installation Testing







6.3 Who & How to do Installation Testing
 A person with in-depth knowledge of System should oversee the installation and testing.
 Start testing basic functions and functionality and gradually add levels of complexity at each successive stage.
 As you complete each test, document the results and verify them against the Installation Guide. Investigate any problems and resolved them.
 Allow to identify and resolve design concerns during testing rather than during deployment.

6.4 What to Check in Installation Testing
Perform several installation tests, which include installing the application on:
 Computers that have the minimum hard drive space required
 Computers that have the minimum RAM required
 Removable drives
 Drives other than the default drive
 CLEAN systems (configurations with no other software installed)
 DIRTY systems (configurations with other programs installed, i.e. anti-virus software, office software, etc.)
 Using More than One Machine to Run the Tests
We also test the entire user setup options (full, typical, and custom), navigational buttons (Next, Back, Cancel, etc.), and user input fields to ensure that they function properly and yield the expected result.

The Un installation of the product also needs to be tested to ensure that all data, executables, and DLL files are removed. The un-installation of the application is tested using DOS command line, Add/Remove programs menu, and through the manual deletion of files.











7.0 Compatibility Testing

Compatibility testing verifies that the product functions correctly on a wide variety of hardware, software, and network configurations. Tests are run on a matrix of platform hardware configurations including High End, Core Market, and Low End.

The requirement to have a product run on different platforms defines the need to test the product for compatibility. The challenges posed are laboratory setup requirements for the hardware, software version, installation and un installation procedures, etc. These issues need to be managed against the time and budget constraints in running the functional & GUI test cases for all possible combinations.

There are mainly three cases to address the different issues in compatibility testing:

Case Study1 A browser-based decision support system developed in Java, which should tested for compatibility against different browsers, web servers & operating systems.

Case Study2 The approach taken while testing the compatibility of variety of sensors, different versions of OPERATING SYSTEMS, browser, modem speeds, firewall settings & LAN settings for a web site, which uses fingerprints for authentication.

Case Study3 This addresses the testing of compatibility on different hardware and software.
A video mail application was tested for compatibility of OPERATING SYSTEMS, Browser &hardware such as Cameras and capture cards.

Software Compatibility Testing can be defined as verification whether the relevant software interacts with and shares information properly with other software and hardware combinations. Thus this test plays an important role in case any application/web site is required to run on different platforms with different software and hardware components. Basically Compatibility Testing answers to the questions like:

 Whether the selected software package will operate on user’s machines or operating systems?
 Whether it is compatible with other computer programs & existing data files?
 Which models of device and other manufacturers of the similar devices are the tested application is compatible?
 Which version of the software is compatible with application to be tested?






7.1 Process for Compatibility
The general process of compatibility testing is as explained below:
Planning:
This is the first step in compatibility testing. It includes:
 Identification of machines required for compatibility testing
 Inform the systems department for arranging the required operating systems, browsers and other hardware/software components.
 Prepare the formats of test report and defect report and submit the same to the customer for their comments/approval.
 Study the application and list down the queries to be confirmed with the customer.
 Allocate hardware & software combinations to team members.
7.2 Identification of Parameters:
The second step in the process of compatibility testing is identifying the parameters. Some of the important parameters are explained below:
Hardware:
One can consider different configuration possibilities for a standard Windows-based PC used in business. Most PCs are modular and built up from various system boards, component cards and other internal devices such as disk drives, CD-ROM drives, video, sound, modem and network cards and peripherals like printers, scanners, mice, keyboards, monitors, cameras, joysticks and other devices that plug into your system and operate externally to the PC.

Operating Systems:
Operating systems and how application interacts with it will affect the appearance and functionality of an application. The most commonly used platforms are DOS, Windows, Mac OS, UNIX, and VMS & Linux.
Operating systems with all required components should be properly installed on corresponding machines before proceeding with the compatibility testing.

Browsers:
Browser compatibility test is applicable for web-based applications. Commonly used browsers are IE & Navigator. Some of the features in the application may or may not run on different browsers. E.g.: Frames can be displayed in IE but can not displayed in some versions of Navigator & Marquee effect can not be seen in some versions of IE. The browser versions may or may not run on all the platforms. e.g: IE 4.01 cannot be loaded with Win2000 Server.

Web Servers:
A web sever is a program that using the client/server model and WWW’s HTTP servers the files that form web pages to web users. Popular web servers are IIS, Apache, I-planet, Novell and Lotus Domino etc. These different web servers are compatible with different platforms.
E.g.: IIS for Windows NT Server, Apache for Unix & Lotus Domino for OS/390.




Databases:
This parameter comes into the picture when a single product supports multiple RDBMS. As for every RDBMS, there will be changes in stored procedures, triggers etc. Hence it becomes essential to verify compatibility of an application with multiples RDBMS that it supports.

Plug-in:
Plug-in applications are programs that can be installed and used as part of the parent application thus enhancing the functionality. The plug-in software is recognized automatically by the parent application and its functionality is integrated into. Amongst popular plug-ins is Adobe’s Acrobat, which allows pdf files to be read on a browser.
7.3 Combination of Parameters to Test

Matrix should be formed with suitable combinations of the above mentioned parameters before starting for testing. However, the following points have to be considered in the course of preparing the matrix:

 Availability of hardware/software. If the same is not available then they can not be included even if it is specified in test requirements.
 Criticality of a particular parameter for the application to be tested. e.g. Is it required to consider modem speeds as a parameter in matrix?
 The relative importance of different values of a parameter. e.g. Is it required to consider lower versions of operating systems and browsers
 Priority to be given for crucial parameters and their values, if enough time is not available because of tight project schedule. e.g. Give priority to Primary OPERATING SYSTEM (Win NT4.0 sp4) than Secondary OPERATING SYSTEM (Win NT4.0 sp5)

7.4 Test Case Identification:

Generally compatibility test is executed only after completion of functional testing. Hence test cases for compatibility testing are to be selected from available functional test case suite.
Following points are to be considered while selecting such test cases:

 Test cases for core functionality shall be selected. i.e. Test cases indicated with showstopper and some of the ‘High’ prioritized test cases to be selected.
 Test cases selected shall be specific to the parameters indicated in matrix E.g. : File –Open / Print for OPERATING SYSTEM compatibility and Down load Plug In for Web server compatibility etc.
 The test cases resulting in defects in functional testing need to be selected for compatibility testing; as such a test case may not result in a defect with some other combinations of parameters.

7.5 Setup Management

After identifying the parameters and test cases the hardware/software required for compatibility testing shall be organized in a separate lab. Following points are to be considered while setting up the lab:

 Connect all identified machines with a separate network.
 Machines are equipped with necessary software / hardware combinations.
 All other applications & data should be removed from these machines as it will be subjected to frequent loading & unloading of operating systems.
 Database & web server shall be properly configured with application.
7.6 Ensuring Pre-requisites
Following pre-requisites have to be ensured before execution of a compatibility test
 Test Report format shall be ready & it shall be approved by customer if applicable.
 Defect report format shall be in place.
 Lab setup should be ready in all respects as indicated in setup management.
 The application should be functionally stable and performance acceptable on one configuration.
 The application to be tested should be properly installed, along with all hardware/software required by it.
7.7 Execution of Test:

Execution of test includes actual testing assuming that all things are available in place.
Operating systems and browses shall be loaded in proper sequence to avoid problems in changing the same.
 Selected test cases are to be executed as per matrix and results are to be entered in test report.
 Defects shall be properly identified & they shall be verified before reporting. If required screen shot shall be taken to support such defects.
 Rotation of hardware – software after each round.
 Closure report to be prepared indicating the activities performed during testing and test results analysis.








Following is the OS/Browser checklist which can be followed while performing Compatibility Testing

OS Browsers
IE 4 IE 5 IE 5.5 IE 6.0 NS 4.08 NS 4.51 NS 4.77 NS 6.0
Win95   
Win98SE      
Win ME     
WinNT4Enterprise sp6      
Win 2000 Server       
Win 2000 Advance Server      
Win 2000 Professional      
Win XP Professional      
Win XP Home      
Win 2003 Server      
Mac OS 8.6    
Mac OS 9.2    
Mac OS X   

Also, to be noted are a few differences between IE and NS browsers. Following are the same:

Microsoft Internet Explorer Netscape Navigator
About 80% of internet users use MSIE (Microsoft Internet Explorer). About 20% of internet users use Netscape.
MSIE is more stable and has less bugs. Netscape supports more standards than MSIE.
It has features that make allow people to make nice looking web pages (page transitions, better looking tables and horizontal rules, favicons.) But lacks support of some basic standards. Netscape tends to crash more often than Internet Explorer.
Other more extreme features (like the calendar and gradients in v5.5) make web pages easy to make and pretty but real web designers can't use them because they won't work on Netscape. Or designers are forced to make two versions of their site - one for MSIE and one for Netscape. Netscape lets you feel good because you aren't using Microsoft software.
Netscape's browser is essentially open source through mozilla.org.
Netscape and AOL have been teaming up on things like AOL instant messenger.
There seems to be fewer security problems with Navigator than MSIE.

Following are the tools which were used for ghosting Operating systems for compatibility testing:

1. VM Ware images created using this
2. Ghosting of images using ghost floppy
3. Ghosting of images using a Menu server where the images have already been store
4. Ghosting of images of Macintosh Operating system using Carbon Cloner.
7.8 What is Ghosting ?

Ghost imaging, using ghosting software is a method of converting the contents of a hard drive -- including its configuration settings and applications -- into an image, and then storing the image on a server or burning it onto a CD.
When contents of the hard drive are needed again, ghosting software converts the image back to original form.
Companies use ghost imaging when they want to create identical configurations and install the same software on numerous machines. For instance, if a company needs to dole out 100 laptops to its employees, then instead of manually setting configurations, and installing applications on each machine, ghosting software (usually contained on a floppy ) will retrieve the ghost image from the server, convert it into its original form, and copy it onto the laptop.














8.0 Acceptance
We have differentiated the Entry & Exit Criteria for each phase of testing
(a) Requirements Phase ( Study of the Project / Product)
(b) Planning Phase (Planning the Testing Activity)
(c) Design Phase (Creation of all Test Plans / Test Cases etc)
(d) Execution Phase (Execution of Test Cases based on testing specific needs , Bugs Reporting etc)
8.1 Acceptance Criteria for Requirements Phase
Phase Requirements Phase
Entry Criteria Customer-signed Statement of Work (SOW)

Exit Criteria Kickoff Meeting has occurred
Project Acceptance Criteria have been identified and documented
8.2 Acceptance Criteria for Planning Phase
Phase Planning Phase
Entry Criteria Project Acceptance Criteria
SOW - Customer-signed
Exit Criteria Completed Project Schedule (e.g. Microsoft Project) – for PMP
Identify Project Risks – for PMP
Creation of List of HW/SW and other Resources – for PMP
Project Management Plan (PMP)
8.3 Acceptance Criteria for Design Phase
Phase Design Phase
Entry Criteria Preparation Project Plan / Test Plan
Preparation of Test Cases / Test Scenarios
Exit Criteria Completion of Test Plans, Test Cases, Test Scenarios
8.4 Acceptance Criteria for Execution Phase
Life Cycle Phase System / Performance Testing Phase – Execution Phase
Entry Criteria Source Code – System Test Baseline
Base Lined System Test Plan
Properly configured Test Environment
Exit Criteria Execute Test Cases, Report Bugs, Perform Test Re-test for Bugs fixed

9.0 Manual vs. Automation
9.1 Manual Testing
Definition: That part of software testing that requires human input, analysis, or evaluation.

 In an ideal world, software testing is 100% automated. Testers simply push a button, the tests execute automatically, and manual effort is limited to reviewing the results of the automation.
 In reality, manual testing in always a part of any testing effort. During the initial phases of software development, manual testing is performed until the software and its user interface are stable enough that beginning automation makes sense. Often, aggressive development cycles and release dates do not allow for the time required to design and implement automated tests. Also, for one-time development efforts or products with short lifetimes, manual testing often is the only sensible option from time and budget standpoints.
 If the product is being tested through a GUI (graphical user interface), and your automation style is to write scripts (essentially simple programs) that drive the GUI, an automated test may be several times as expensive as a manual test.
 If you use a GUI capture/replay tool that tracks your interactions with the product and builds a script from them, automation is relatively cheaper. It is not as cheap as manual testing, though, when you consider the cost of recapturing a test from the beginning after you make a mistake, the time spent organizing and documenting all the files that make up the test suite, the aggravation of finding and working around bugs in the tool, and so forth. Those small "in the noise" costs can add up surprisingly quickly.
 If you’re testing a compiler, automation might be only a little more expensive than manual testing, because most of the effort will go into writing test programs for the compiler to compile. Those programs have to be written whether or not they’re saved for reuse.

9.2 Automation Testing

Definition: Automated tests execute a sequence of actions without human intervention. This approach helps eliminate human error, and provides faster results.

 Since most products require tests to be run many times, automated testing generally leads to significant labor cost saving over time. Typically a company will pass the break-even point for labor costs after just two or three runs of an automated test.
 Testing may seem like just a set of actions, but good testing is an interactive cognitive process. That’s why automation is best applied only to a narrow spectrum of testing, not to the majority of the test process.
 Humans are good at noticing oddities; they’re bad at painstaking or precise checking of results. If bugs lurk in the 7th decimal place of precision, humans will miss it, whereas a tool might not. Tools are not limited to looking at what appears on the screen; they can look at the data structures that lie behind it.
 The fact that humans can’t be precise about inputs means that repeated runs of a manual test are often slightly different tests, which might lead to discovery of a support code bug. For example, people make mistakes, back out, and retry inputs, thus sometimes stumbling across interactions between error-handling code and the code under test.
 Configuration testing argues for more automation. Running against a new OS, device, 3rd party library, etc., is logically equivalent to running with changed support code. Since you know change is coming, automation will have more value. The trick, though, is to write tests that are sensitive to configuration problems - to the differences between OSes, devices, etc. It likely makes little sense to automate your whole test suite just so you can run it all against multiple configurations.
 It’s annoying to discover a bug in manual testing and then find you can’t reproduce it. Probably you did something that you don’t remember doing. Automated tests rarely have that problem (though sometimes they’re dependent on parts of the environment that change without your noticing it). Rudimentary tracing or logging in the product can often help greatly - and it’s useful to people other than testers.
 An automated test suite can explore the whole product every day. A manual testing effort will take longer to revisit everything. So the bugs automation does find will tend to be found sooner after the incorrect change was made.





























9.3 Flow diagram to determine Test scenarios for Manual and Automation



9.4 Summary of Manual Vs Automated Testing

Pros Cons
Manual Testing  Test case creation is quick and inexpensive
 Better at simulating real world use.
 Better at analyzing test results and exploring new test cases.
 Best when test steps and results are not well – defined.
 Best when extensive analysis is required.
 Does not require a technically trained staff  Might not be consistent in rerunning test cases.
 Rerunning large volumes of test cases is expensive
Automated Testing  Can execute test cases unattended.
 Can cost-effectively run large volumes of test cases.
 Can cost – effectively rerun large volumes of test cases repetitively.  Test case creation is expensive.
 Test case creation is time consuming.
 Maintaining test cases can be expensive.
 Requires a highly technical staff.
 Rerunning old tests is not likely to find new bugs.
9.5 Why & When One Should Consider Automation
Definition: Automated tests execute a sequence of actions without human intervention. This approach helps eliminate human error, and provides faster results.
If testing is a means to the end of understanding the quality of the software, automation is just a means to a means.
Few reckless assumptions of automated testing

Reckless Assumption #1 – Testing is a “sequence of actions”
A more useful way to think about testing in as a sequence of interactions interspersed with evaluations. Some of those interactions are predictable, and some of them can be specified in purely objective terms. However, many others are complex, ambiguous, and volatile. Although it is often useful to conceptualize a general sequence of action that comprise a given test, if we try to reduce testing to a rote series of actions the result will be a narrow and shallow set of tests.
Manual testing, on the other hand, is a process that adapts easily to changes and can cope with complexity. Testing may seem like just a set of action, but good testing in an interactive cognitive process. That’s why automation is best applied only to a narrow spectrum of testing, not to the majority of the test process. If you set out to automate all the necessary test execution, you’ll probably spend a lot of money and time creating relatively weak tests that ignore many interesting bugs, and find many “problems” that turn out to be merely unanticipated correct behavior.

Reckless Assumption #2 – Testing means repeating the same actions over and over.
Once a specific test case is executed a single time, and no bug is found, there is little chance that the test case will ever find a bug, unless a new bug is introduced into the system. If there is variation in the test cases, though, as three is a greater likelihood of revealing problems both new and old. Variability is one of the great advantages of hand testing over script and playback testing.
Highly repeatable testing can actually minimize the chance of discovering all the important problems, for the same reason stepping in someone else’s footprints minimizes the chance of being blown up by land mine.

Reckless Assumption #3 – We can automate testing actions.
Some tasks that are easy for people are hard for computers. Probably the hardest part of automation is interpreting test results. For GUI software, it is very hard to automatically notice all categories of significant problems while ignoring the insignificant problems. The problem of automat ability is compounded by the high degree of uncertainty and change in a typical innovative software project.
Even if we have a particular sequence of operations that can in principle sequence of operations that can in principle be automated, we can only do so if we have an appropriate tool for the job. Information about tools is hard to come by, though, and the most critical aspects of a regression test tool are impossible to evaluate unless we create or review as industrial size test suite using the tool. Here are some of the factors to consider when selecting a test tool.
• Capability: Does the tool have all the critical features we need, especially in the area of test result validation and test suite management?
• Reliability: Does the tool work for long periods without failure, or is it full of bugs? Many test tools are developed by small companies that do a poor job of testing them.
• Capacity: Beyond the toy examples and demos, does the tool work without failure in an industrial environment? Can it handle large scale test suites that run for hours or days and involve thousands of scripts?
• Performance: Is the tool quick enough to allow a substantial saving in test development and execution time versus hand testing.
• Compatibility: Does the tools work with the particular technology that we need to test?
• Non-Intrusiveness: How well does the tool simulate an actual user? Is the behavior of the software under test the same with automation as without?

Reckless Assumption #4 – An automated test is faster, because it needs no human intervention.
All automated test suites require human intervention, if only to diagnose the results and fix broken tests. It can also be surprisingly hard to make a complex test suite run without a hitch. Common culprits are changes to the software being tested, memory problems, file system problems, network glitches, and bugs in the test tool itself.

Reckless Assumption #5 - Automation reduces human error.
Yes, some errors are reduced. Namely, the ones that humans make when they are asked carry out a long list of mundane mental and tactile activities. But other errors are amplified. Any bug that goes unnoticed when the master compare files are generated will go systematically unnoticed every time the suite is executed. Or an oversight during debugging could accidentally deactivate hundreds of tests.



Reckless Assumption #6 – We can quantify the costs and benefits of manual vs. automated testing.
The truth is, hand testing and automated testing are really two different processes, rather than two different ways to execute the same process. Their dynamics are different, and the bugs they tend to reveal are different. Therefore, direct comparison of them in terms of dollar cost or number of bugs found is meaningless.

Reckless Assumption #7 – Automation will not harm the test project.
The last most thorny of all the problems that we face in pursuing an automation strategy: it’s dangerous to automate something that we don’t understand. If we don’t get the test strategy clear before introducing automation, the result of test automation will be a large mass of test code that no one fully understands. As the original developers of the suite drift away to other assignments, and others take over maintenance, the suite gains a kind of citizenship in the test team. The maintainers are afraid to throw any old tests out, even if they look meaningless, because they might later turn out to be important. So, the suite continues to accrete new tests, becoming an increasingly mysterious oracle, like some old Himalayan guru or talking oak tree from a Disney movie. No one knows what the suite actually tests, or what it means for the product to “pass the test suite” and the bigger it gets, the less likely anyone will go to the trouble to find out.
9.6 A Sensible Approach to Automation
• Maintain a careful distinction between the automation and the process that it automates. The test process should be in a form that is convenient to review and those maps to the automation.
• Think of your automation as a baseline test suite to be used in conjunction with manual testing, rather than as a replacement for it.
• Carefully select your test tools. Gather experiences from other testers and organizations. Try evaluation versions of candidate tools before you buy.
• Put careful thought into buying or building a test management harness. A good test management system can really help make the suite more reviewable and maintainable.
• Assure that each execution of the test suite results in a status report that includes what tests passed and failed versus the actual bugs found. The report should also detail any work done to maintain or enhance the suite. I’ve found these reports to be indispensable source material for analyzing just how cost effective the automation is.
• Assure that the product is mature enough so that maintenance costs from constantly changing tests don’t overwhelm any benefits provided.

10.0 Unit summary
In this session we have learnt:
1. The following testing types
i. Structural VS Functional Testing
ii. Regression Testing
iii. Performance Testing
iv. Installation Testing
v. Compatibility Testing
vi. Acceptance Testing
vii. Manual VS Automation Testing

10.1 Exercise
Answer the following in Short.
2. List various types of testing under Black box and White box testing?
3. Draw the flow chart for installation testing?
4. Define and explain Performance Testing in few lines?
5. What is the purpose of compatibility testing and what is the approach for the same?
6. Given a product, how do you decided, the type of testing to be performed?