Software Construction & Design 1
The University of Sydney Page 1
Software Design and
Construction 2
SOFT3202 / COMP9202
Advanced Testing
Techniques (2)
School of Information Technologies
Dr. Basem Suleiman
The University of Sydney Page 2
Copyright Warning
COMMONWEALTH OF AUSTRALIA
Copyright Regulations 1969
WARNING
This material has been reproduced and communicated to
you by or on behalf of the University of Sydney
pursuant to Part VB of the Copyright Act 1968 (the
Act ).
The material in this communication may be subject
to copyright under the Act. Any further copying or
communication of this material by you may be the
subject of copyrightprotection under
the Act.
Do not remove this notice.
The University of Sydney Page 3
Agenda
Testing Techniques
Black-box, white-box and gray-box testing
Fuzzing
Code/Test Coverage
System Testing
User Acceptance Testing
The University of Sydney Page 4
Advanced Testing Types
The University of Sydney Page 5
Black-box Testing
No programming and software knowledge
Carried by software testers
May lead to poor coverage
Can be applied unit, integration, system and acceptance testing
Executable
software
code
Test
Case
Input
Test
Case
Output
The University of Sydney Page 6
Black Box Testing Example
Test planned without knowledge of the code
Based only on specification or design
E.g., given a function that computes sign (x+y)
f(X,Y)
x
y
sign(x+y)
The University of Sydney Page 7
White-box Testing
Verifies the internal structures of a source code (structural testing)
Test cases are designed based on code understanding and programming
skills (by software developers)
Unit, integration and system testing
Can help detecting many errors, but not unimplemented/missing
requirements
Test
Case
Input
Test
Case
Output
The University of Sydney Page 8
Grey-box Testing
Tests are designed based on the internal structures and algorithms
Test cases from high-level and detailed application documentation
To design more informed test cases (better test coverage)
Tests are executed from outside
Black-box testing
The University of Sydney Page 9
Black, White or Grey?
.
The University of Sydney Page 10
Black-box Testing Test Cases
Equivalence Class Value for
month
Value for
year
Months with 31 days, non-leap years 7 (July) 1901
Months with 31 days, leap years 7 (July) 1904
Months with 30 days, non-leap years 6 (June) 1901
Months with 30 days, leap year 6 (June) 1904
Months with 28 or 29 days, non-leap year 2 February 1901
Months with 28 or 29 days, leap year 2 February 1904
Equivalence Class Value for
month
Value
for year
Leap years divisible by 400 2 (February) 2000
Non-leap years divisible by
100
2 (February) 1900
Non-positive invalid month 0 1291
Positive invalid months 13 1315
The University of Sydney Page 11
Exercise White-box Testing
Based on the provided code, design test cases for
the method getNumDaysInMonth()
Hint: Consider various paths in the code
The University of Sydney Page 12
White-box Testing
Exercising all possible paths through the code at least once, most faults will
trigger failures
Paths identification requires understanding the source code and the data
structure
Diagrams (e.g., flow graph, activity diagram) may be utilized to visualize all
paths
Test cases that each edge in the activity diagram is traversed at least once
Condition select inputs both the true and false branches
The University of Sydney Page 13
White-box Testing Visualizing Paths
.
The University of Sydney Page 14
White-box Testing Test Cases
Test case Path
(year = 0, month = 1) {throw1}
(year = 1901, month = 1) {n=32 return}
(year = 1901, month = 2) (year = 1904, month = 2) {n=28 return}
(year = 1901, month = 4) {n=30 return}
(year = 1901, month = 0) {throw2}
The University of Sydney Page 15
White-box Testing Compare
Test case Path
(year = 0, month = 1) {throw1}
(year = 1901, month = 1) {n=32 return}
(year = 1901, month = 2) (year = 1904, month = 2) {n=28 return}
(year = 1901, month = 4) {n=30 return}
(year = 1901, month = 0) {throw2}
Equivalence Class Value for
month
Value for
year
Months with 31 days, non-leap years 7 (July) 1901
Months with 31 days, leap years 7 (July) 1904
Months with 30 days, non-leap years 6 (June) 1901
Months with 30 days, leap year 6 (June) 1904
Months with 28 or 29 days, non-leap year 2 February 1901
Months with 28 or 29 days, leap year 2 February 1904
Equivalence Class Value for
month
Value
for year
Leap years divisible by 400 2 (February) 2000
Non-leap years divisible by
100
2 (February) 1900
Non-positive invalid month 0 1291
Positive invalid months 13 1315
The University of Sydney Page 16
White-box Testing Compare
Both extensively test cases of February calculations
No test cases for years divisible by 100 path testing with isLeapYear()
Path testing exercise a path in the program (e.g., numDays=32)
Omission and missing requirements cannot be detected in path testing
Failure to handle the non-leap year 1900
The University of Sydney Page 17
Code (Test) Coverage
The University of Sydney Page 18
Code (Test) Coverage
A metric that measures the extent to which a source code has been
executed by a set of tests
White-box testing
Usually measured as percentage, e.g., 70% coverage
Different criteria to measure coverage
E.g., method, statement, loop
The University of Sydney Page 19
Coverage Criteria
Coverage Criteria Description
Method How many of the methods are called, during the tests
Call How many of the potential method calls are executed, during the tests
Statement How many statements are exercised, during the tests
Branch (decision) How many of the possible outcomes of the branches have occurred during
the tests
ConditionHas each separate condition within each branch been evaluated to both
true and false
Condition/decision
coverage
Requires both decision and condition coverage be satisfied
Loop Each loop executed zero times, once and more than once
Data flow coverage Each variable definition and its usage executed
The University of Sydney Page 20
Coverage Criteria Method
Examine the code. Given the following
test cases, calculate the method
coverage
month = 7, year = 1901
month 0, year = 2002
month = 2, year = 1901
month = 2, year = 1904
Discuss
The University of Sydney Page 21
Coverage Criteria Statement
Examine the code snippet.
Calculate the statement
coverage after executing each
of the following test cases:
year = 1901
year = 1904
Discuss
The University of Sydney Page 22
Coverage Criteria Branch
Examine the code snippet.
Identify test cases that will
ensure that each branch is
exercised at least once
Discuss
The University of Sydney Page 23
Coverage Criteria Condition
Examine the code snippet.
Calculate the condition
coverage after executing each
of the following test cases:
year = 2000
year = 2018
Discuss
The University of Sydney Page 25
Other Coverage Criteria
Path coverage
Has every possible route through a given part of the code been executed
Entry/exit coverage
Has every possible call and return of the function been executed
State coverage
Has every state in a FSM (finite-state-machine) been reached and explored
Other
The University of Sydney Page 26
Visualizing Paths Recall
.
The University of Sydney Page 27
Weather Station Case Study Revisit
The University of Sydney Page 28
State Coverage
Each state in a finite-state-machine
been exercised reached and explored
Testing states of WeatherStation using
state model
Identify sequences of state transitions to
be tested
Define event sequences to force these
transitions
Examples:
Shutdown Running Shutdown
Configuration Running Testing
Transmitting Running
Many others
The University of Sydney Page 29
Which Coverage Criteria?
Some coverage criteria are implied in other ones
Path coverage implies decision, statement and entry/exit coverage
Decision coverage implies statement
Full coverage might be infeasible
Module with n decisions
Loops can result in infinite number of paths
The University of Sydney Page 30
Coverage Target
What coverage should one aims for?
Software criticality determines coverage level
Extremely high coverage for safety-critical (dependable) software
Government/standardization organizations
E.g., European corporation for space standardization (ESS-E-ST-40C)
100% statement and decision coverage for 2 out of 4 criticality levels
The University of Sydney Page 31
Coverage Tests Quality
Coverage percentage might confusingly be linked to a quality target
Achieving certain coverage target not necessarily useful
e.g., software to be deployed when 80% coverage is achieved
High coverage numbers can be attained but probably by test smells
TDD can help but not sufficient to get good test
Understanding different types of coverage criteria can effectively help
in identifying effective test cases
https://martinfowler.com/bliki/TestCoverage.html
The University of Sydney Page 32
Fuzzy Testing (Fuzzing)
The University of Sydney Page 33
Fuzz Testing (Fuzzing)
Automated software testing techniques that verifies the softwarebehaviour
using invalid, unexpected or random data
Random testing or monkey testing
Infinite monkey theorem
a monkey hitting keys at random on a typewriter keyboard for an infinite
amount of time will almost surely type any given text, such as the complete
works of William Shakespeare
https://en.wikipedia.org/wiki/Infinite_monkey_theorem
The University of Sydney Page 34
Fuzz Testing
The Fuzzer would generate particular sequence of inputs that might
trigger a crash/fault
Typically input domain is structured
Effective fuzzing should generate inputs
Valid enough to create unexpected behaviours
invalid enough to expose boundary cases
https://en.wikipedia.org/wiki/Fuzzing
The University of Sydney Page 35
Fuzzing based on Inputs
Generation-based Fuzzer
Inputs are generated from scratch based on a provided model
Mutation-based Fuzzer
Inputs are generated by mutating provided seeds
Input-structure Fuzzer
Generate inputs for software that take structured input
Sequence of mouse or keyboard events
https://en.wikipedia.org/wiki/Fuzzing
The University of Sydney Page 36
Fuzzing based Program Structure
Black-box Fuzzer
Unaware of program structure randomly generating inputs
Can be run in parallel and scale
May discover obvious bugs
White-box Fuzzer
Use program analysis to systematically increase code coverage
Might use model-based testing to generate inputs and check outputs against
the program specifications
https://en.wikipedia.org/wiki/Fuzzing
The University of Sydney Page 37
System Testing
The University of Sydney Page 38
System Testing
Verifying the behavior of the entire system
Does the system meet the requirements?
Functional requirements
Non-functional properties (e.g., security, performance)
Quality of requirements impact ease of testing
Also considers external interfaces, operating environment, hardware
devices
Carried out by the development/testing team
The University of Sydney Page 39
Testing Levels System Testing
Unit Testing
Integration
Testing
System
Testing
Acceptance
Testing
Effective unit and integration testing should have identified many of the software
defects
System testing might uncover undetected defects (defect testing)
The University of Sydney Page 40
System Testing Functional Testing
Test system functionality (what the system suppose to do)
Test cases designed from the system requirements (use cases)
System as a Black-box
The University of Sydney Page 41
Functional Testing Example
Consider the following requirements of a healthcare system
If a patient is known to be allergic to any medication, then the prescription of
that medication shall result in a warning message being issued to the system user.
If a prescriber chooses to ignore an allergy warning, they shall provide a reason
why this had been ignored
How one can verify that these requirements have been satisfied?
The University of Sydney Page 42
Functional Testing Example
1. A patient record with no known allergies prescribed medication for allergies a warning
message is not issued by the system
2. A patient record with a known allergy prescribe the medication to that the patient is allergic
to a warning is issued by the system
3. A patient record in which allergies to two or more drugs prescribe these drugs separately
has a correct warning for each drug been issued?
4. Prescribe two drugs that the patient is allergic to two warnings are correctly issued?
5. Prescribe a drug that issues a warning and overrule that warning the system must require the
user to provide information explaining why the warning was overruled
The University of Sydney Page 43
Scenario based Testing
Approach for system testing where typical scenarios of system use are used for
testing
A story describes how the system might be used
Test several requirements including functional and non-functional
Scenarios should be realistic and real system users should reflect the user thinking
and system processes
Scenarios should include deliberate mistakes to verify the system behavior
Problems should be noted
Part of the system requirement analysis (use case scenarios)
The University of Sydney Page 44
Scenario based Testing Example
Consider the usage following scenario for a Mental Health Care-Patient Management
System. What functions/non-functional requirements will be tested?
The University of Sydney Page 45
System Testing Non-Functional Properties
Test how well the system behave regarding non-functional properties
Performance, security, usability, reliability, other
Defined as non-functional requirements from which test cases are designed
Quality attributes or ilities
Typically defined precisely and associated with quantifiable measures used
(e.g., response time, availability)
Testbeds and real production environments
The University of Sydney Page 47
Performance Testing
Verify how the system performs under intended workload
Typically performance measures
Speed: amount of time an application needs to perform a task
Once a user input search terms, search results should be displayed in less than
400ms
95% of transactions should be executed in less than 2 seonds
Throughput: the amount of work an application must perform in unit time
The application should handle 800 transactions per second
Scalability:
How does an application perform when number of simultaneous users increase
How does an application perform as the data it processes increases in size
The University of Sydney Page 49
Types of Performance Testing
Load Testing
Verify the system performance under expected workload (number of
users/transactions)
Identify performance bottlenecks (hardware and software)
Stress Testing
Verify software behavior at the maximum and beyond the designed load
Determine the behavioral limits and test robustness in mission-critical software
Spike testing
Verify the system performance under sudden workload increase/decrease
Determine how well the system will handle such sudden changes
https://en.wikipedia.org/wiki/Software_performance_testing#Load_testing
The University of Sydney Page 50
Types of Performance Testing
Soak (endurance) Testing
Verify how the system perform under a typical production load over extended
period of time
E.g., memory leaks could trigger after certain time duration of a typical
production load
Resources leak, performance degradation over time
Configuration Testing
Verify system performance under different configuration changes to systems
components
Server configuration, software configuration
The University of Sydney Page 51
Acceptance Testing
The University of Sydney Page 52
User Acceptance Testing
Unit Testing
Integration
Testing
System
Testing
Acceptance
Testing
Test from system users perspectives
Developers / Testers
Users
The University of Sydney Page 53
User (Acceptance) Testing
Users test the system by providing input and assess the system behavior
Developers/testing team try to replicated users environment and
interactions
Users determine if the system behave as expected in their work environment
in terms of functional and non-functionality aspects
Alpha, Beta and User testing
The University of Sydney Page 54
User Testing Alpha Testing
Users use the software at the developers environment where the software
used in a controlled setting
Users identify problems that are not obvious to the developers
Provide early validation of the system functionality
Agile methods involve users in testing developed functions in every
development iteration
The University of Sydney Page 55
User Testing Beta Testing
User use a release of the software in their environment to verify system
behavior
Group of lead users/customers
Also to discover problems between the software and features of the
environment
The University of Sydney Page 56
User Testing Acceptance Testing
Users test the system and decide whether it is ready for use
Previously agreed acceptance criteria
The University of Sydney Page 57
User Testing Acceptance Testing
Define acceptance criteria (early as part of the system contract
Plan acceptance testing
Resources, test plan and required coverage
Derive acceptance tests
Design tests that cover both functional and non-functional requirements
Run tests
Ideally in actual environment by end users
Negotiate test results
Based on acceptance criteria, identified issues to be addressed by developers
Accept/reject
Good enough for use or further development is needed
The University of Sydney Page 58
Software Testing Classification
Functional Testing
Unit testing
Integration testing
System testing
Regression testing
Interface testing
User Acceptance Testing (UAT) Alpha and Beta testing
Configuration, smoke, sanity, end-to-end testing
Non-Functional Testing
Performance testing
Load testing
Security testing
Stress testing
Reliability testing
Usability testing
The University of Sydney Page 59
Test/Code Coverage
The University of Sydney Page 60
Tools for Code Coverage
There are many tools/plug-ins for code coverage
in Java
Example: EclEmma*
EclEmma is a code coverage plug-in for Eclipse
It provides rich features for code coverage analysis
in Eclipse IDE
EclEmma is based on the JaCoCo code coverage
library
JaCoCo is a free code coverage library for
Java, which has been created by the EclEmma
team
https://www.eclemma.org/
The University of Sydney Page 61
EclEmma Counters
EclEmma supports different types of counters to be
summarized in code coverage overview
bytecode instructions, branches, lines, methods, types and
cyclomatic complexity
Should understand each counter and how it is measured
Counters are based on JaCoCon see JaCoCo
documentation for detailed counter definitions
https://www.eclemma.org/
http://www.jacoco.org/jacoco/trunk/doc/counters.html
The University of Sydney Page 62
EclEmma Coverage View
https://www.eclemma.org/
The Coverage view shows all analyzed Java elements within the common Java hierarchy.
Individual columns contain the numbers for the active session, always summarizing the child
elements of the respective Java element
The University of Sydney Page 63
EclEmma Coverage Source Code Annotations
https://www.eclemma.org/
Source lines color code:
green for fully covered lines,
yellow for partly covered lines (some
instructions or branches missed)
red for lines that have not been
executed at all
Diamonds color code
green for fully covered branches,
yellow for partly covered branches
red when no branches in the
particular line have been executed.
The University of Sydney Page 64
References
Ian Sommerville. 2016. Software Engineering (10th ed.) Global Edition. Pearson,
Essex England
Bernd Bruegge and Allen H. Dutoit. 2009. Object-Oriented Software Engineering
Using Uml, Patterns, and Java (3rd ed.). Pearson.
Martin Fowler, Test Coverage. https://martinfowler.com/bliki/TestCoverage.html
Software Performance Testing,
https://en.wikipedia.org/wiki/Software_performance_testing
Software Testing, https://en.wikipedia.org/wiki/Software_testing
Fuzz Testing (Fuzzing) https://en.wikipedia.org/wiki/Fuzzing
The University of Sydney Page 65
W5 Tutorial: Test Techniques 2 + W5
quiz
W5 Lecture: Review and Overview
of Design Patterns
Testing Assignment A2 release
Next Lecture/Tutorial
Reviews
There are no reviews yet.