SE 333/433 Software Testing & Quality Assurance
Introduction :
Software Quality and Software Testing
1
Outline
• Introductions
• Syllabus
• Software Quality Assurance: Introduction • Software Quality
• Software Testing
• Road map
2
Overview
• What is software quality? • How can it be measured?
• How can it be measured before/after the software is delivered?
3
Quality ?
• Think of an everyday object • e.g. a chair
• How would you measure it’s “quality”?
• construction quality? (e.g. strength of the joints,…)
• aesthetic value? (e.g. elegance,…)
• fit for purpose? (e.g. comfortable,…)
• All quality measures are relative
• there is no absolute scale
• we can say A is better than B but it is usually hard to say how much better
4
Examples of Metrics from Everyday Life
• Working and living
• Cost of utilities for the month
• Cost of groceries for the month
• Amount of monthly rent per month
• Time spent at work each Saturday for the past month
• Time spent mowing the lawn for the past two times
• College experience
• Grades received in class last quarter
• Number of classes taken each quarter
• Amount of time spent in class this week
• Amount of time spent on studying and homework this week
• Number of hours of sleep last night
• Travel
• Time to drive from home to the airport
• Amount of miles traveled today
• Cost of meals and lodging for yesterday
5
What is Software Quality ?
• Conformance to requirements.
• Narrowest sense of software quality.
• Lack of bugs.
• High reliability (number of failures per n hours of operation).
6
What is Software Quality ?
• According to the IEEE, Software quality is:
1. The degree to which a system, component, or process
meets specified requirements.
2. The degree to which a system, component, or process meets customer or user needs or expectations.
7
Software Quality :
• Definition:
Conformance to explicitly stated functional and performance requirements, explicitly documented development standards, and implicit characteristics that are expected of all professionally developed software
• Three important points in this definition
• Explicit software requirements are the foundation from which quality is measured. Lack of
conformance to requirements is lack of quality
• Specific standards define a set of development criteria that guide the manner in which software is engineered.
• There is a set of implicit requirements that often goes unmentioned (e.g., ease of use). If software conforms to its explicit requirements but fails to meet implicit requirements, software quality is suspect
8
ISO 9126 Software Quality Factors
• Functionality
• The degree to which the software satisfies stated needs
• Reliability
• The amount of time that the software is available for use
• Usability
• The degree to which the software is easy to use
• Efficiency
• The degree to which the software makes optimal use of system resources
• Maintainability
• The ease with which repair and enhancement may be made to the software
• Portability
• The ease with which the software can be transposed from one environment to another
9
Key Quality Concepts
• Reliability
• designer must be able to predict how the system will
behave:
• completeness – does it do everything it is supposed to do? (e.g. handle all possible inputs)
• consistency – does it always behave as expected? (e.g. repeatability)
• robustness – does it behave well under abnormal conditions? (e.g. resource failure)
• Efficiency
• Use of resources such as processor time, memory, network bandwidth
10
Key Quality Concepts
• Maintainability
• How easy will it be to modify in the future? • perfective, adaptive, corrective
• Usability
• How easy is it to use?
11
How is Software Quality is measured?
• Metric:
• (IEEE) A quantitative measure of the degree to which a system, component, or process possesses a given attribute
• Purpose
• Aid in the evaluation of analysis and design models
• Provide an indication of the complexity of procedural designs and source code
12
Metrics for Object Oriented Design
• Number of children (i.e., subclasses)
• As the number of children of a class grows
• Reuse increases
• The abstraction represented by the parent class can be diluted by inappropriate children
• The amount of testing required will increase
13
Metrics for Object Oriented Design
• Coupling between object classes
• Measures the number of collaborations a class has with
any other classes
• Higher coupling decreases the reusability of a class
• Higher coupling complicates modifications and testing • Coupling should be kept as low as possible
14
Comment Percentage (CP)
• Number of commented lines of code divided by the number of non-blank lines of code
• Usually 20% indicates adequate commenting for C or Java code
• The higher the CP value the more maintainable the module is
15
Outline
• Introductions
• Syllabus
• Software Quality Assurance: Introduction • Software Quality
• Software Testing
• Road map
16
Software Testing: Overview
• Software Testing?
• Why Test?
• What Do We Do When We Test ?
• Understand basic techniques for software verification and
validation
• Analyze basics of software testing techniques
17
A Concrete Example…
Should start searching at 0, not 1
public static int numZero (int [ ] arr)
{ // Effects: If arr is null throw NullPointerException
// else return the number of occurrences of 0 in arr int count = 0;
for (int i = 1; i < arr.length; i++){Test 1[ 2, 7, 0 ]Expected: 1Actual: 1 if (arr [ i ] == 0) {count++; }}return count; }Error : i is 1, not 0, on the first iteration Symptoms: noneError: i is 1, not 0Error propagates to the variable countSymptoms: count is 0 at the return statementTest 2[ 0, 2, 7 ]Expected: 1Actual: 0 18 Myth BustersSoftware Testing19 of 98Myth #1 in Software TestingQ: What is the objective of software testing? A: Testing is to show that there are noerrors/bugs/defects in the software. Fact: No!! The main objective of testing is to discover defects. Testing is a destructive activity.21 of 98Myth #2 in Software TestingQ: What is the objective of software testing?A: Testing is to ensure that the software does what it is supposed to do. Fact: Only partly true. Testing is also to ensure the software does not do what it is not supposed to do.23 of 98Myth #3 in Software TestingQ: How challenging is software testing? A: Testing is easier than design andimplementation. Fact: Must consider all possible scenarios. Implied and unstated requirements and threats. Must be imaginative and creative.25 of 98Myth #4 in Software TestingQ: How challenging is software testing? A: Testing is an extremely creative andintellectually challenging task.27 of 98 The Term Bug• Bug is used informally• Defect• Fault• Problem • Error• Incident • Anomaly • Variance• Failure• Inconsistency• Product Anomaly• Product Incidence• Feature 28Failures• deviation of the observed behavior of a system from its specification, i.e., its expected behavior.• Failures can only be determined with respect to the specifications.• Failures are concerned with the observed behavior and outcome of the system. • Failures areDefects• Defects are• flaws in a system that can cause the systemto fail to perform its required function• e.g. an incorrect condition or statement.• Defects are concerned with specific parts or components of the system.• Defects are synonymous with faults Errors• Errors are• human actions that result in a fault ordefect in the system.• Errors are concerned with the underlying causes of the defects.• Errors are synonymous with mistakes. The Relations among Failures, Defects, and Errors• A human being makes an error (mistake)• can occur in design, coding, requirements, even testing.• An error can lead to a defect (fault)• can occur in requirements, design, or program code.• If a defect in code is executed, a failure may occur.• Failures only occur when a defect in the code is executed. • Not all defects cause failures all the time.• Defects occur because human beings are fallible• Failures can be caused by environmental conditions as well.The Relations among Failures, Defects, and Errors• The terms error, failure and defect have different meaning when testing. Especially in using JUnit. In this case:• Test Case Verdicts • Pass• The test case execution was completed• The function being tested performed as expected• Fail• The test case execution was completed• The function being tested did not perform as expected• Error• The test case execution was not completed, due to an unexpected event, exceptions, or improper set up of the test case, etc. Failures vs. Defects: A Simple Example• For any integer n, square (n) = n*n. int square (int x) {return x*2; }A defectsquare (3) = 6A failureFailures vs. Defects: A Simple Example• For any integer n, square (n) = n*n. int square (int x) {return x*2; }A defectsquare (2) = 4Correct result Not a failureSoftware Testing• Software testing is• the process of executing a program (or parts of a program) withthe intention of finding defects• The purpose of testing• to find defects.• to discover every conceivable weakness in a software product. 1. Software testing ≠ Debugging.2. Software testing ≠ Quality assuranceSoftware Testing vs. Quality Assurance (QA)• Testing is necessary, but not sufficient for quality assurance • Testing contributes to improve quality by identifying problems.• Quality assurance sets the standards for the team/organization to build better software.Software is a Skin that Surrounds Our Civilization 38Spectacular Software Failures• NASA’s Mars lander: September 1999, crashed due to a units integration fault• Ariane 5 explosion : Very expensiveAriane 5: exception-handling bug : forced self destruct on maiden flight (64-bit to 16-bit conversion: about 370 million $ lost)39 Northeast Blackout of 2003508 generating units and 256 power plants shut downAffected 10 million people in Ontario, CanadaAffected 40 million people in 8 US statesFinancial losses of $6 Billion USD The alarm system in the energy management system failed due to a software error and operators were not informed of the power overload in the system40Costly Software Failures !• NIST report, “The Economic Impacts of Inadequate Infrastructure for Software Testing” (2002)➢Inadequate software testing costs the US alone between $22 and $59 billion annually• Huge losses due to web application failures➢Financial services : $6.5 million per hour (just in USA!) ➢Credit card sales applications : $2.4 million per hour (in USA) 41Discussion … • Have you heard of other software bugs? • In the media?• From personal experience?• Does this embarrass you as a (future) software engineer?42Cost of Not Testing Poor Program Managers might say:“ Testing is too expensive.” • Testing is the most time consuming and expensive part of software development• Not testing is even more expensive• If we do not have enough testing effort early, the cost of testing increases 43Testing Goals • The Major Objectives of Software Testing:- Detect errors (or bugs) as much as possible in a given timeline.- Demonstrate a given software product matching its requirement specifications.- Validate the quality of a software testing using the minimum cost and efforts.• Testing can NOT prove product works 100%44Testing Overview • Who tests• Programmers• Testers/Req. Analyst • Users• What is tested• Unit Code testing• Functional Code testing• Integration/system testing • User interface testing• How (test cases designed) – Intuition– Specification based (black box) – Code based (white-box) 45Exhaustive Testing is Hard• Number of possible test cases (assuming 32 bit integers)• 232 232 = 264• Do bigger test sets help?• Test set {(x=3,y=2), (x=2,y=3)}will detect the error• Test set {(x=3,y=2),(x=4,y=3),(x=5,y=1)} will not detect the error although it has more test cases• It is not the number of test cases• But, if T1 T2, then T1 will detect every fault detected by T2int max(int x, int y) { if (x > y)
return x;
else
return x;
}
18446744073709551616 possibilities
46
Exhaustive Testing is Hard
• Assume that the input for the max procedure was an integer array of size n
• Number of test cases: 232 n
• Assume that the size of the input array is not bounded
• Number of test cases:
47
Generating Test Cases Randomly
bool isEqual(int x, int y)
{
if (x = y)
z := false;
else
z := false;
return z; }
0.00000000023283064365386962890625
• If we pick test cases randomly it is unlikely that we will pick a case where x and y have the same value
• If x and y can take 232 different values, there are 264 possible test cases. In 232 of them x and y are equal
• probability of picking a case where x is equal to y is 2-32
• It is not a good idea to pick the test cases randomly (with uniform distribution) in this case
• So, naive random testing is pretty hopeless too
48
Mutation Testing
1. Induce small changes to the program: mutants
2. Find tests that cause the mutant programs to fail: killing
mutants
3. Failure is defined as different output from the original program
4. Check the output of useful tests on the original program • Example program and mutants
if (x > y)
z = x – y;
else
z = 2 * x;
49
if (x > y) if (x >= y) z = x – y;
z = x + y;
z = x – m; else
z = 2 * x;
Types of Testing
• Unit (Module) testing
• testing of a single module in an isolated environment
• Integration testing
• testing parts of the system by combining the modules
• System testing
• testing of the system as a whole after the integration phase
• Acceptance testing
• testing the system as a whole to find out if it satisfies the requirements specifications
50
Information
Project Proposal :
I will create empty groups in D2L where you can self-enroll.
I will create a project discussion for those who are looking for members, or want to join a group can post it there
Choose your partner(s) for the project and enroll in an empty group in D2L
51
Outline
• Introductions
• Syllabus
• Software Quality Assurance: Introduction • Software Quality
• Software Testing
• Road map
52
Reviews
There are no reviews yet.