[SOLVED] case study Java algorithm fuzzing Software Construction & Design 1

Software Construction & Design 1

The University of Sydney Page 1

Software Design and

Construction 2

SOFT3202 / COMP9202

Advanced Testing

Techniques (2)

School of Information Technologies

Dr. Basem Suleiman

The University of Sydney Page 2

Copyright Warning

COMMONWEALTH OF AUSTRALIA

Copyright Regulations 1969

WARNING

This material has been reproduced and communicated to
you by or on behalf of the University of Sydney
pursuant to Part VB of the Copyright Act 1968 (the
Act ).

The material in this communication may be subject
to copyright under the Act. Any further copying or
communication of this material by you may be the
subject of copyrightprotection under
the Act.

Do not remove this notice.

The University of Sydney Page 3

Agenda

Testing Techniques
Black-box, white-box and gray-box testing

Fuzzing

Code/Test Coverage

System Testing

User Acceptance Testing

The University of Sydney Page 4

Advanced Testing Types

The University of Sydney Page 5

Black-box Testing

No programming and software knowledge

Carried by software testers

May lead to poor coverage

Can be applied unit, integration, system and acceptance testing

Executable

software

code

Test

Case

Input

Test

Case

Output

The University of Sydney Page 6

Black Box Testing Example

Test planned without knowledge of the code

Based only on specification or design

E.g., given a function that computes sign (x+y)

f(X,Y)

x

y
sign(x+y)

The University of Sydney Page 7

White-box Testing

Verifies the internal structures of a source code (structural testing)

Test cases are designed based on code understanding and programming

skills (by software developers)

Unit, integration and system testing

Can help detecting many errors, but not unimplemented/missing

requirements

Test

Case

Input

Test

Case

Output

The University of Sydney Page 8

Grey-box Testing

Tests are designed based on the internal structures and algorithms
Test cases from high-level and detailed application documentation

To design more informed test cases (better test coverage)

Tests are executed from outside
Black-box testing

The University of Sydney Page 9

Black, White or Grey?

.

The University of Sydney Page 10

Black-box Testing Test Cases

Equivalence Class Value for

month

Value for

year

Months with 31 days, non-leap years 7 (July) 1901

Months with 31 days, leap years 7 (July) 1904

Months with 30 days, non-leap years 6 (June) 1901

Months with 30 days, leap year 6 (June) 1904

Months with 28 or 29 days, non-leap year 2 February 1901

Months with 28 or 29 days, leap year 2 February 1904

Equivalence Class Value for

month

Value

for year

Leap years divisible by 400 2 (February) 2000

Non-leap years divisible by

100

2 (February) 1900

Non-positive invalid month 0 1291

Positive invalid months 13 1315

The University of Sydney Page 11

Exercise White-box Testing

Based on the provided code, design test cases for

the method getNumDaysInMonth()

Hint: Consider various paths in the code

The University of Sydney Page 12

White-box Testing

Exercising all possible paths through the code at least once, most faults will

trigger failures

Paths identification requires understanding the source code and the data

structure

Diagrams (e.g., flow graph, activity diagram) may be utilized to visualize all

paths

Test cases that each edge in the activity diagram is traversed at least once
Condition select inputs both the true and false branches

The University of Sydney Page 13

White-box Testing Visualizing Paths

.

The University of Sydney Page 14

White-box Testing Test Cases

Test case Path

(year = 0, month = 1) {throw1}

(year = 1901, month = 1) {n=32 return}

(year = 1901, month = 2) (year = 1904, month = 2) {n=28 return}

(year = 1901, month = 4) {n=30 return}

(year = 1901, month = 0) {throw2}

The University of Sydney Page 15

White-box Testing Compare

Test case Path

(year = 0, month = 1) {throw1}

(year = 1901, month = 1) {n=32 return}

(year = 1901, month = 2) (year = 1904, month = 2) {n=28 return}

(year = 1901, month = 4) {n=30 return}

(year = 1901, month = 0) {throw2}

Equivalence Class Value for

month

Value for

year

Months with 31 days, non-leap years 7 (July) 1901

Months with 31 days, leap years 7 (July) 1904

Months with 30 days, non-leap years 6 (June) 1901

Months with 30 days, leap year 6 (June) 1904

Months with 28 or 29 days, non-leap year 2 February 1901

Months with 28 or 29 days, leap year 2 February 1904

Equivalence Class Value for

month

Value

for year

Leap years divisible by 400 2 (February) 2000

Non-leap years divisible by

100

2 (February) 1900

Non-positive invalid month 0 1291

Positive invalid months 13 1315

The University of Sydney Page 16

White-box Testing Compare

Both extensively test cases of February calculations

No test cases for years divisible by 100 path testing with isLeapYear()
Path testing exercise a path in the program (e.g., numDays=32)

Omission and missing requirements cannot be detected in path testing
Failure to handle the non-leap year 1900

The University of Sydney Page 17

Code (Test) Coverage

The University of Sydney Page 18

Code (Test) Coverage

A metric that measures the extent to which a source code has been

executed by a set of tests

White-box testing

Usually measured as percentage, e.g., 70% coverage

Different criteria to measure coverage

E.g., method, statement, loop

The University of Sydney Page 19

Coverage Criteria

Coverage Criteria Description

Method How many of the methods are called, during the tests

Call How many of the potential method calls are executed, during the tests

Statement How many statements are exercised, during the tests

Branch (decision) How many of the possible outcomes of the branches have occurred during

the tests

ConditionHas each separate condition within each branch been evaluated to both

true and false

Condition/decision

coverage

Requires both decision and condition coverage be satisfied

Loop Each loop executed zero times, once and more than once

Data flow coverage Each variable definition and its usage executed

The University of Sydney Page 20

Coverage Criteria Method

Examine the code. Given the following

test cases, calculate the method

coverage

month = 7, year = 1901

month 0, year = 2002

month = 2, year = 1901

month = 2, year = 1904

Discuss

The University of Sydney Page 21

Coverage Criteria Statement

Examine the code snippet.

Calculate the statement

coverage after executing each

of the following test cases:

year = 1901

year = 1904

Discuss

The University of Sydney Page 22

Coverage Criteria Branch

Examine the code snippet.

Identify test cases that will

ensure that each branch is

exercised at least once

Discuss

The University of Sydney Page 23

Coverage Criteria Condition

Examine the code snippet.

Calculate the condition

coverage after executing each

of the following test cases:

year = 2000

year = 2018

Discuss

The University of Sydney Page 25

Other Coverage Criteria

Path coverage

Has every possible route through a given part of the code been executed

Entry/exit coverage

Has every possible call and return of the function been executed

State coverage

Has every state in a FSM (finite-state-machine) been reached and explored

Other

The University of Sydney Page 26

Visualizing Paths Recall

.

The University of Sydney Page 27

Weather Station Case Study Revisit

The University of Sydney Page 28

State Coverage

Each state in a finite-state-machine

been exercised reached and explored

Testing states of WeatherStation using

state model

Identify sequences of state transitions to

be tested

Define event sequences to force these

transitions

Examples:
Shutdown Running Shutdown

Configuration Running Testing
Transmitting Running

Many others

The University of Sydney Page 29

Which Coverage Criteria?

Some coverage criteria are implied in other ones

Path coverage implies decision, statement and entry/exit coverage

Decision coverage implies statement

Full coverage might be infeasible

Module with n decisions

Loops can result in infinite number of paths

The University of Sydney Page 30

Coverage Target

What coverage should one aims for?

Software criticality determines coverage level

Extremely high coverage for safety-critical (dependable) software
Government/standardization organizations

E.g., European corporation for space standardization (ESS-E-ST-40C)

100% statement and decision coverage for 2 out of 4 criticality levels

The University of Sydney Page 31

Coverage Tests Quality

Coverage percentage might confusingly be linked to a quality target

Achieving certain coverage target not necessarily useful
e.g., software to be deployed when 80% coverage is achieved

High coverage numbers can be attained but probably by test smells

TDD can help but not sufficient to get good test

Understanding different types of coverage criteria can effectively help

in identifying effective test cases

https://martinfowler.com/bliki/TestCoverage.html

The University of Sydney Page 32

Fuzzy Testing (Fuzzing)

The University of Sydney Page 33

Fuzz Testing (Fuzzing)

Automated software testing techniques that verifies the softwarebehaviour

using invalid, unexpected or random data

Random testing or monkey testing

Infinite monkey theorem

a monkey hitting keys at random on a typewriter keyboard for an infinite

amount of time will almost surely type any given text, such as the complete

works of William Shakespeare

https://en.wikipedia.org/wiki/Infinite_monkey_theorem

The University of Sydney Page 34

Fuzz Testing

The Fuzzer would generate particular sequence of inputs that might

trigger a crash/fault

Typically input domain is structured

Effective fuzzing should generate inputs
Valid enough to create unexpected behaviours

invalid enough to expose boundary cases

https://en.wikipedia.org/wiki/Fuzzing

The University of Sydney Page 35

Fuzzing based on Inputs

Generation-based Fuzzer
Inputs are generated from scratch based on a provided model

Mutation-based Fuzzer
Inputs are generated by mutating provided seeds

Input-structure Fuzzer
Generate inputs for software that take structured input

Sequence of mouse or keyboard events

https://en.wikipedia.org/wiki/Fuzzing

The University of Sydney Page 36

Fuzzing based Program Structure

Black-box Fuzzer
Unaware of program structure randomly generating inputs

Can be run in parallel and scale

May discover obvious bugs

White-box Fuzzer
Use program analysis to systematically increase code coverage

Might use model-based testing to generate inputs and check outputs against

the program specifications

https://en.wikipedia.org/wiki/Fuzzing

The University of Sydney Page 37

System Testing

The University of Sydney Page 38

System Testing

Verifying the behavior of the entire system

Does the system meet the requirements?

Functional requirements

Non-functional properties (e.g., security, performance)

Quality of requirements impact ease of testing

Also considers external interfaces, operating environment, hardware
devices

Carried out by the development/testing team

The University of Sydney Page 39

Testing Levels System Testing

Unit Testing
Integration

Testing
System
Testing

Acceptance
Testing

Effective unit and integration testing should have identified many of the software

defects

System testing might uncover undetected defects (defect testing)

The University of Sydney Page 40

System Testing Functional Testing

Test system functionality (what the system suppose to do)

Test cases designed from the system requirements (use cases)

System as a Black-box

The University of Sydney Page 41

Functional Testing Example

Consider the following requirements of a healthcare system

If a patient is known to be allergic to any medication, then the prescription of
that medication shall result in a warning message being issued to the system user.

If a prescriber chooses to ignore an allergy warning, they shall provide a reason
why this had been ignored

How one can verify that these requirements have been satisfied?

The University of Sydney Page 42

Functional Testing Example

1. A patient record with no known allergies prescribed medication for allergies a warning
message is not issued by the system

2. A patient record with a known allergy prescribe the medication to that the patient is allergic
to a warning is issued by the system

3. A patient record in which allergies to two or more drugs prescribe these drugs separately
has a correct warning for each drug been issued?

4. Prescribe two drugs that the patient is allergic to two warnings are correctly issued?

5. Prescribe a drug that issues a warning and overrule that warning the system must require the
user to provide information explaining why the warning was overruled

The University of Sydney Page 43

Scenario based Testing

Approach for system testing where typical scenarios of system use are used for
testing

A story describes how the system might be used

Test several requirements including functional and non-functional

Scenarios should be realistic and real system users should reflect the user thinking
and system processes

Scenarios should include deliberate mistakes to verify the system behavior

Problems should be noted

Part of the system requirement analysis (use case scenarios)

The University of Sydney Page 44

Scenario based Testing Example

Consider the usage following scenario for a Mental Health Care-Patient Management
System. What functions/non-functional requirements will be tested?

The University of Sydney Page 45

System Testing Non-Functional Properties

Test how well the system behave regarding non-functional properties
Performance, security, usability, reliability, other

Defined as non-functional requirements from which test cases are designed

Quality attributes or ilities

Typically defined precisely and associated with quantifiable measures used
(e.g., response time, availability)

Testbeds and real production environments

The University of Sydney Page 47

Performance Testing

Verify how the system performs under intended workload

Typically performance measures

Speed: amount of time an application needs to perform a task

Once a user input search terms, search results should be displayed in less than
400ms

95% of transactions should be executed in less than 2 seonds

Throughput: the amount of work an application must perform in unit time

The application should handle 800 transactions per second

Scalability:

How does an application perform when number of simultaneous users increase

How does an application perform as the data it processes increases in size

The University of Sydney Page 49

Types of Performance Testing

Load Testing

Verify the system performance under expected workload (number of
users/transactions)

Identify performance bottlenecks (hardware and software)

Stress Testing

Verify software behavior at the maximum and beyond the designed load

Determine the behavioral limits and test robustness in mission-critical software

Spike testing
Verify the system performance under sudden workload increase/decrease

Determine how well the system will handle such sudden changes

https://en.wikipedia.org/wiki/Software_performance_testing#Load_testing

The University of Sydney Page 50

Types of Performance Testing

Soak (endurance) Testing

Verify how the system perform under a typical production load over extended
period of time

E.g., memory leaks could trigger after certain time duration of a typical
production load

Resources leak, performance degradation over time

Configuration Testing
Verify system performance under different configuration changes to systems

components

Server configuration, software configuration

The University of Sydney Page 51

Acceptance Testing

The University of Sydney Page 52

User Acceptance Testing

Unit Testing
Integration

Testing
System
Testing

Acceptance
Testing

Test from system users perspectives

Developers / Testers
Users

The University of Sydney Page 53

User (Acceptance) Testing

Users test the system by providing input and assess the system behavior

Developers/testing team try to replicated users environment and
interactions

Users determine if the system behave as expected in their work environment
in terms of functional and non-functionality aspects

Alpha, Beta and User testing

The University of Sydney Page 54

User Testing Alpha Testing

Users use the software at the developers environment where the software
used in a controlled setting

Users identify problems that are not obvious to the developers

Provide early validation of the system functionality

Agile methods involve users in testing developed functions in every
development iteration

The University of Sydney Page 55

User Testing Beta Testing

User use a release of the software in their environment to verify system
behavior

Group of lead users/customers

Also to discover problems between the software and features of the
environment

The University of Sydney Page 56

User Testing Acceptance Testing

Users test the system and decide whether it is ready for use

Previously agreed acceptance criteria

The University of Sydney Page 57

User Testing Acceptance Testing

Define acceptance criteria (early as part of the system contract

Plan acceptance testing

Resources, test plan and required coverage

Derive acceptance tests

Design tests that cover both functional and non-functional requirements

Run tests

Ideally in actual environment by end users

Negotiate test results
Based on acceptance criteria, identified issues to be addressed by developers

Accept/reject
Good enough for use or further development is needed

The University of Sydney Page 58

Software Testing Classification

Functional Testing

Unit testing

Integration testing

System testing

Regression testing

Interface testing

User Acceptance Testing (UAT) Alpha and Beta testing

Configuration, smoke, sanity, end-to-end testing

Non-Functional Testing

Performance testing

Load testing

Security testing

Stress testing

Reliability testing

Usability testing

The University of Sydney Page 59

Test/Code Coverage

The University of Sydney Page 60

Tools for Code Coverage

There are many tools/plug-ins for code coverage

in Java

Example: EclEmma*

EclEmma is a code coverage plug-in for Eclipse
It provides rich features for code coverage analysis

in Eclipse IDE

EclEmma is based on the JaCoCo code coverage

library

JaCoCo is a free code coverage library for

Java, which has been created by the EclEmma

team

https://www.eclemma.org/

The University of Sydney Page 61

EclEmma Counters

EclEmma supports different types of counters to be

summarized in code coverage overview

bytecode instructions, branches, lines, methods, types and

cyclomatic complexity

Should understand each counter and how it is measured

Counters are based on JaCoCon see JaCoCo

documentation for detailed counter definitions

https://www.eclemma.org/

http://www.jacoco.org/jacoco/trunk/doc/counters.html

The University of Sydney Page 62

EclEmma Coverage View

https://www.eclemma.org/

The Coverage view shows all analyzed Java elements within the common Java hierarchy.

Individual columns contain the numbers for the active session, always summarizing the child

elements of the respective Java element

The University of Sydney Page 63

EclEmma Coverage Source Code Annotations

https://www.eclemma.org/

Source lines color code:
green for fully covered lines,

yellow for partly covered lines (some

instructions or branches missed)

red for lines that have not been

executed at all

Diamonds color code
green for fully covered branches,

yellow for partly covered branches

red when no branches in the

particular line have been executed.

The University of Sydney Page 64

References

Ian Sommerville. 2016. Software Engineering (10th ed.) Global Edition. Pearson,
Essex England

Bernd Bruegge and Allen H. Dutoit. 2009. Object-Oriented Software Engineering
Using Uml, Patterns, and Java (3rd ed.). Pearson.

Martin Fowler, Test Coverage. https://martinfowler.com/bliki/TestCoverage.html

Software Performance Testing,
https://en.wikipedia.org/wiki/Software_performance_testing

Software Testing, https://en.wikipedia.org/wiki/Software_testing

Fuzz Testing (Fuzzing) https://en.wikipedia.org/wiki/Fuzzing

The University of Sydney Page 65

W5 Tutorial: Test Techniques 2 + W5

quiz

W5 Lecture: Review and Overview

of Design Patterns

Testing Assignment A2 release

Next Lecture/Tutorial