Software Construction & Design 1
The University of Sydney Page 1
Software Design and
Construction 2
SOFT3202 / COMP9202
Introduction
Software Testing
School of Information Technologies
Dr. Basem Suleiman
The University of Sydney Page 2
Copyright Warning
COMMONWEALTH OF AUSTRALIA
Copyright Regulations 1969
WARNING
This material has been reproduced and communicated to
you by or on behalf of the University of Sydney
pursuant to Part VB of the Copyright Act 1968 (the
Act ).
The material in this communication may be subject
to copyright under the Act. Any further copying or
communication of this material by you may be the
subject of copyrightprotection under
the Act.
Do not remove this notice.
The University of Sydney Page 3
Agenda
Software Engineering
Software Testing
Unit Testing
The University of Sydney Page 4
Testing in Software
Engineering
The University of Sydney Page 5
Societies, businesses and governments dependent on SW systems
Power, Telecommunication, Education, Government, Transport, Finance, Health
Work automation, communication, control of complex systems
Large software economies in developed countries
IT application development expenditure in the US more than $250bn/year1
Total value added GDP in the US2: $1.07 trillion
Emerging challenges
Security, robustness, human user-interface, and new computational platforms
1 Chaos Report, Standish group Report, 2014
2 softwareimpact.bsa.org
Software is Everywhere!
The University of Sydney Page 6
Software Engineering Body of Knowledge
Software Requirements
Software Design / Modelling
Software Construction
Software Testing
Software Maintenance
Software Configuration Management
Software Engineering Process
Software Engineering Tools and Methods
Software Quality
Software Engineering Body of Knowledge (SWEBOK) https://www.computer.org/web/swebok/
The University of Sydney Page 7
Need to build high quality software systems under resource constraints
Social
Satisfy user needs(e.g., functional, reliable, trustworthy)
Impact on peoples lives (e.g., software failure, data protection)
Economical
Reduce cost; open up new opportunities
Average cost of IT development ~$2.3m, ~$1.3m and ~$434k for large, medium
and small companies respectively3
Time to market
Deliver software on-time
3 Chaos Report, Standish group Report, 2014
Why Software Engineering?
The University of Sydney Page 8
Software Quality Assurance
Quality (products)
Fitness for use
Software quality
Satisfying end uses needs; correct behaviour, easy to use, does not crash, etc
Easy to the developers to debug and enhance
Quality Assurance (QA)
Processes and standards that lead to manufacturing high quality products
Software Quality Assurance
Ensuring software under development have high quality and creating processes and standards
in organization that lead to high quality software
Software quality is often determined through Testing
Juran and Gryna 1998
The University of Sydney Page 9
Software Testing
Why software testing?
The University of Sydney Page 10
Therac-25 Overdose*
What happened?
Therac-25 radiation therapy machine
Patients exposed to overdose of radiation (100 times more than intended) 3 lives!!
Why Did happen?
Particular nonstandard sequence of keystrokes was entered within 8 seconds
Operator override a warning message with error code (MALFUNCTION followed by
a number from 1 to 64) not explained in the user manual
Software checks safety replacing hardware interlocks in previous Therac versions
Absence of independent software code review
software and hardware integration has never been tested until assembled in the
hospital (Big Bang Testing)
*https://en.wikipedia.org/wiki/Therac-25#Problem_description
https://en.wikipedia.org/wiki/Therac-25#Problem_description
The University of Sydney Page 11
What happened?
European large rocket 10 years development, ~$7 billion
Unmanaged software exception resulted from a data conversion
from 64-bit floating point to a 16-bit signed integer
Backup processor failed straight after using the same software
Exploded 37 seconds after lift-off
Why did it happen?
Inadequate validation and verification, testing and reviews, design error, incorrect
analysis of changing requirements, .
5 http://iansommerville.com/software-engineering-book/files/2014/07/Bashar-Ariane5.pdf
Software Failure Ariane 5 Disaster5
http://iansommerville.com/software-engineering-book/files/2014/07/Bashar-Ariane5.pdf
The University of Sydney Page 12
Nissan Recall Airbag Defect*
What happened?
~ 3.53 million vehicles recall of various models 2013-2017
Front passenger airbag may not deploy in an accident
Why Did happen?
Software that activates airbags deployment improperly classify occupied
passenger seat as empty in case of accident
Software defect that could lead to improper airbag function (failure)
No warning that the airbag may not function properly
Software sensitivity calibration due to combination of factors (high engine
vibration and changing seat status)
http://www.reuters.com/article/us-autos-nissan-recall/nissan-to-recall-3-53-million-vehicles-air-bags-may-not-deploy-idUSKCN0XQ2A8
http://www.reuters.com/article/us-autos-nissan-recall/nissan-to-recall-3-53-million-vehicles-air-bags-may-not-deploy-idUSKCN0XQ2A8
The University of Sydney Page 15
https://en.wikipedia.org/wiki/List_of_failed_and_overbudget_custom_software_projects
Software Project Failures
Project Duration Cost Failure/Status
Pust Siebel Swedish Police
case management (Swedish
Police)
2011 2014 $53m (actual) Permanent failure scraped due to poor
functioning, inefficient in work environments
US Federal Government
Health Care Exchange
Web application
2013
ongoing
$93.7m (expected),
$1.5bn (actual)
Ongoing problems too slow, poor
performance, people get stuck in the
application process (frustrated users)
https://en.wikipedia.org/wiki/List_of_failed_and_overbudget_custom_software_projects
The University of Sydney Page 16
Why Software Testing?
Software development and maintenance costs
Big financial burden
Total costs of inadequate software testing on the US economy is $59.5bn
NIST study 2002*
One-third of the cost could be eliminated by improved software testing
Need to develop functional, robust and reliable software systems
Human/social factor society dependency on software in every aspect of their lives
Critical software systems medical devices, flight control, traffic control
Meet user needs and solve their problems
Small software errors could lead to disasters
* https://www.nist.gov/sites/default/files/documents/director/planning/report02-3.pdf
https://www.nist.gov/sites/default/files/documents/director/planning/report02-3.pdf
The University of Sydney Page 17
Software Testing Costs
Capers Jones, Applied software measurement (2nd ed.): assuring productivity and quality, (1997), McGraw-Hill
The University of Sydney Page 18
Software Testing
What is software testing?
The University of Sydney Page 19
Software Testing
Software process to
Demonstrate that software meets its requirements (validation testing)
Find incorrect or undesired behavior caused by defects/bugs (defect testing)
E.g., System crashes, incorrect computations, unnecessary interactions and data
corruptions
Part of software verification and validation (V&V) process
The University of Sydney Page 23
Testing (Levels)
Testing level Description
Unit / Functional Testing The process of verifying functionality of software components
(functional units, subprograms) independently from the whole system
Integration Testing The process of verifying interactions/communications among software
components. Incremental integration testing vs. Big Bang testing
System Testing The process of verifying the functionality and behaviour of the entire
software system including security, performance, reliability, and
external interfaces to other applications
Acceptance Testing The process of verifying desired acceptance criteria are met in the
system (functional and non-functional) from the user point of view
The University of Sydney Page 24
Software Verification and Validation
Software testing is part of software Verification and Validation (V&V)
The goal of V&V is to establish confidence that the software is fit for purpose
Software Validation
Are we building the right product?
Ensures that the software meets customer expectations
Software Verification
Are we building the product right?
Ensures that the software meets its stated functional and non-functional requirements
The University of Sydney Page 27
Unit Testing
The University of Sydney Page 28
Unit Testing
The process of verifying functionality of software components
independently
Unit methods, functions or object classes
Verify that each functional unit behaves as expected (defect testing)
Carried out by developers / SW testers
First level of testing
The University of Sydney Page 29
Why Unit Testing?
Change and maintain code at smaller scale
Discover defects early and fix it at cheaper costs
Ease debugging
Code reusability
Reduce integration testing
The University of Sydney Page 30
Unit Testing How?
1. Design test cases
2. Prepare test data
3. Run test cases using test data
4. Compare results to test cases
5. Prepare test reports
The University of Sydney Page 31
Software Testing Process
Ian Sommerville. 2016. Software Engineering (10th ed.). Addison-Wesley, USA.
The University of Sydney Page 32
Designing Test Cases
Effective test cases show:
The unit under test does what it spoused to do
Reveal defects in the unit, if there is any
Design two types of test cases
Test normal operation of the unit
Test abnormality (common problems/defects)
The University of Sydney Page 33
Choosing Test Cases Techniques
Partition testing (equivalence partitioning)
Identify groups of inputs that have common characteristics
From within each of these groups, choose tests
Use program specifications, documentation and/or experience
Guideline-based testing
Use testing guidelines based on previous experience of the kinds of errors
often made
The University of Sydney Page 34
Equivalence Partitioning
Different groups with common characteristics
E.g., positive numbers, negative numbers
Program behave in a comparable way for all members of a group
Choose test cases from each of the partitions
The University of Sydney Page 35
Test Case Selection
Understanding developers thinking
Developers tend to think of typical values of input
Developers sometimes overlook atypical values of input
Choose test cases that are:
On the boundaries of the partitions
Close to the midpoint of the partition
The University of Sydney Page 36
Test Cases Identifying Partitions
Consider the following brief program specification:
A program accepts 4 to 8 inputs which are five-digits integers greater than
10,000
Exercise:
Identify the input partitions and possible test input values
The University of Sydney Page 38
Choose Test Cases Testing Guidelines
Knowledge of type of test cases effective for finding errors
Example of test cases for testing sequences, arrays or lists:
Test single value
Test different sequences of different sizes in different tests
Test partition boundaries (first, middle and last elements)
The University of Sydney Page 39
General Testing Guidelines Examples
Choose inputs that force the system to generate all error messages;
Design inputs that cause input buffers to overflow
Repeat the same input or series of inputs numerous times
Force invalid outputs to be generated
Force computation results to be too large or too small
J. A. Wittaker (2009) Exploratory Software Testing, Adison-Wesley
The University of Sydney Page 40
Unit Testing
The University of Sydney Page 41
Unit Testing Terminology
Code under test
Unit test
a piece of code written by a developer that executes a specific functionality in the
code under test and asserts a certain behavior or state
Small unit of code e.g., a method or class
External dependencies are removed (mocks can be used)
Not suitable for complex user interface or component interaction
Test Fixture
The context for testing
Usually shared set of testing data
Methods for setup those data
E.g., a fixed string (test fixture), which is used as input for a method
The University of Sydney Page 42
Unit Testing Frameworks for Java
Junit
TestNG
Jtest (commercial)
Many others
https://en.wikipedia.org/wiki/List_of_unit_testing_frameworks#Java
The University of Sydney Page 43
Unit Testing Frameworks Junit
An open source framework for writing and running tests for Java
Test cases and fixtures, test suites, test runner
One of the unit testing frameworks collectively known as xUnit
Uses annotations to identify methods that specify a test
Can be integrated with Eclipse, and build automation tools (e.g., Ant,
Maven, Gradle)
https://github.com/junit-team/junit4
The University of Sydney Page 44
Junit Constructs
Junit test (Test class)
A method contained in a class which is only used for testing (called Test class)
Test suite
Contains several test classes which will be executed all in the specified order
Test Annotations
To define/denote test methods (e.g., @Test, @Before)
Such methods execute the code under test
Assertion methods (assert)
To check an expected result versus actual results
Variety of methods
Provide meaningful messages in assert statements
The University of Sydney Page 45
Junit Annotations
JUnit 4* Description
import org.junit.* Import statement for using the following annotations
@Test Identifies a method as a test method
@Before Executed before each test. To prepare the test environment (e.g., read input
data, initialize the class)
@After Executed after each test. To cleanup the test environment (e.g., delete temporary
data, restore defaults) and save memory
@BeforeClass Executed once, before the start of all tests. To perform time intensive activities,
e.g., to connect to a database. Need to be defined as static to work with Junit
@AfterClass Executed once, after all tests have been finished. To perform clean-up activities,
e.g., to disconnect from a database. Need to be defined as static to work with
Junit
*See Junit 5 annotations and compare them https://junit.org/junit5/docs/current/user-guide/#writing-tests-annotations
https://junit.org/junit5/docs/current/user-guide/#writing-tests-annotations
The University of Sydney Page 46
Junit Assert Class
Assert class provides static methods to test for certain conditions
Assertion method compares the actual value returned by a test to
the expected value
Specify the expected and actual results and the error message
Throw an AssertionException if the comparison fails
The University of Sydney Page 47
Junit Methods to Assert Test Results
Method Description
fail([message]) Let the method fail. E.g., to check that a certain part of the
code not reached or to have a failing test before the test code
implemented
assertTrue([message,] boolean condition)
assertFalse([message,] boolean condition)
Checks that the Boolean condition is true
Checks that the Boolean condition is false
assertEquals([message,] expected, actual) Tests that two values are the same. Note: for arrays the
reference is checked not the content of the arrays.
assertEquals([message,] expected, actual,
tolerance)
Test that float or double values match. The tolerance is the
number of decimals which must be the same
assertNull([message,] object)
assertNotNull([message,] object)
Checks that the object is null
Checks that the object is not null
*Also check assertions in Junit 5 https://junit.org/junit5/docs/current/user-guide/#writing-tests-assertions
https://junit.org/junit5/docs/current/user-guide/#writing-tests-assertions
The University of Sydney Page 48
Junit Test Example
MyClass multiply(int, int)
method
MyClassTests class for testing the method multiply(int, int)
The University of Sydney Page 49
Junit Executing Tests
Tests can be executed from the command line
runClass() of the org.junit.runner.JUnitCore class allows developers to run one or
several test classes
Information about tests can be retrieved using the org.junit.runner.Result object
Test automation
Build tools such as Maven or Gradle along with a Continuous Integration Server
(e.g., Jekins) can be configured to run tests automatically on a regular basis
Essential for regular daily tests (agile development)
The University of Sydney Page 50
Junit Executing Tests
To run tests from the command line
The University of Sydney Page 51
Junit Test Suites
Test suite contains several test classes which will be executed all in the
specified order
The University of Sydney Page 52
Junit Static Import
Static import is a feature that allows fields and methods in a class as public
static to be used without specifying the class in which the field s defined
The University of Sydney Page 53
Junit Eclipse Support
Create Junit tests via wizards or write them manually
Eclipse IDE also supports executing tests interactively
Run-as Junit Test will starts JUnit and execute all test methods in the selected
class
Extracting the failed tests and stack traces
Create test suites
The University of Sydney Page 54
Automated Test Frameworks
Key sign of good practice is that tests are kept as an asset of the process
and can be run automatically, and frequently
Also, reports should be easy to understand
Big Green or Red Signal
A framework is some software that allows test cases to be described in
standard form and run automatically
The University of Sydney Page 55
Tests Automation Junit with Gradle
To use Junit in your Gradle build, add a testCompile dependency to your build
file
Gradle adds the test task to the build graph and needs only appropriate Junit
JAR to be added to the classpath to fully activate the test execution
The University of Sydney Page 56
Junit with Gradle Parallel Tests
maximum simultaneous JVMs spawned
causes a test-running JVM to close and be replaced by a brand new
one after the specified number of tests have run under an instance
The University of Sydney Page 59
Next week .
Theory of Testing, Design of Test Cases and
more unit testing
Tutorial Unit Testing
The University of Sydney Page 60
References
Ian Sommerville. 2016. Software Engineering (10th ed.) Global Edition.
Pearson, Essex England
Junit 4, Project Documentation, https://junit.org/junit4/
Vogella GmbH, JUnt Testing with Junit Tutorial (Version 4.3,21.06.2016 )
http://www.vogella.com/tutorials/JUnit/article.html
Reviews
There are no reviews yet.