Software Construction & Design 1
The University of Sydney Page 1
Software Design and
Construction 2
SOFT3202 / COMP9202
Advanced Testing
Techniques (1)
School of Information Technologies
Dr. Basem Suleiman
The University of Sydney Page 2
Copyright Warning
COMMONWEALTH OF AUSTRALIA
Copyright Regulations 1969
WARNING
This material has been reproduced and communicated to
you by or on behalf of the University of Sydney
pursuant to Part VB of the Copyright Act 1968 (the
Act ).
The material in this communication may be subject
to copyright under the Act. Any further copying or
communication of this material by you may be the
subject of copyrightprotection under
the Act.
Do not remove this notice.
The University of Sydney Page 3
Agenda
Testing Types
Integration Testing, Regression Testing
Advanced Testing Techniques
Test doubles (Dummies, Fakes, Stubs, Spies, Mocks)
Contract Test
Testing Frameworks
Mockito
The University of Sydney Page 4
Advanced Testing Types
Integration testing, regression testing
The University of Sydney Page 5
Software Components/Sub-systems
The University of Sydney Page 6
Integration Testing
The process of verifying interactions/communications among software
components behave according to its specifications
Independently developed (and tested) units may not behave correctly when
they interact with each other
Activate corresponding components and run high-level tests
Incremental integration testing vs. Big Bang testing
.
The University of Sydney Page 9
Interaction Errors
Parameter interfaces
Methods in objects have a parameter interface
Procedural interfaces
Objects and reusable components
Message passing interfaces
One component encapsulates a service from another component by passing a
message to it
Shared memory interfaces
Interfaces in which block of memory is shared between components (e.g.,
embedded systems)
.
The University of Sydney Page 10
Your Testing Exposed Bugs
What would you do when your testing reveal bugs/errors?
You fixed the discovered bugs, what should happen next?
You extended one class with additional functionality (new feature), what
should happen next?
.
The University of Sydney Page 11
Regression Testing
Verifies that the software behaviour has not changed by incremental
changes to the software
Bug fixes, code extension, code enhancements
Modern software development processes are iterative/incremental
Changes may be introduced which may affect the validity of previous tests
Regression testing is to verify
Pre-tested functionality (and non-functional properties) still working as expected
No new bugs are introduced
.
The University of Sydney Page 12
Regression Testing Techniques
. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.460.5875&rep=rep1&type=pdf
Type Description
Retest All Re-run all the test cases in a test suit
Test Selection Re-run certain test cases based on the changes in the code
Test case
prioritization
Re-run test cases in order of its priority; high, medium, low. Priority
determined by how criticality and impact of test cases on the product
Hybrid Re-run selected test cases based on its priority
The University of Sydney Page 14
Test-Driven Development
The University of Sydney Page 15
Test-driven Development
A software development approach for developing the code incrementally
along with a set of tests for that increment
Write tests before code
All tests must pass before starting the next increment
Introduced in the XP agile development method
The University of Sydney Page 16
TDD Cycle
A rapid cycle of testing, coding, and refactoring
Kent Beck
Every few minutes, TDD provides proven code
that has been tested, designed, and coded
Red, Green, Refactor cycle
The University of Sydney Page 17
TDD Cycle Think
Think of a behavior you want your code to have(small
increment; few lines of code)
Think of atest (few lines of code) that will fail unless the
behavior is present
Pair programming helps
Driver and navigator
..
The University of Sydney Page 18
TDD Cycle Red (Run the Test)
Write the tests only enough code for the current
increment of behavior
Typically less than 5 lines of code
Code for the class behavior and its public interface
(encapsulation)
Tests use method and class names do not exist yet
Run your entire suite of tests and enjoy the test failure
Results in Red progress bar (testing tools)
..
https://www.jamesshore.com/Agile-Book/test_driven_development.html
The University of Sydney Page 19
TDD Cycle Green (Write Code)
Write the code; just enough to get the test to pass
Less than 5 lines
Its okay to hard code, youll refactor
Run your tests again, and enjoy the tests passing
Results in Green progress bar (testing tools)
..
The University of Sydney Page 20
TDD Cycle Refactor
Review the code and look for possible improvements
Ask your navigator if they made any notes
Series of very small refactorings
1-2 minutes each, no longer than 5 minutes
Run the tests after each refactoring
Should always be green (pass!)
Test failed and no obvious answer, get back to good code
Refactor many times, improve design
Refactoring isnt about changing behavior
..
The University of Sydney Page 21
TDD Cycle Repeat
Repeat to add new behavior, start the cycle over again
Tiny bit of well-tested, well-designed code will be incrementally
created
Typically, run through several cycles very quickly, then spend
more time on refactoring
Do not skip any step, especially refactoring, to speed up!
..
The University of Sydney Page 22
TDD Red, Green, Refactor
Use Red, Green, Refactor cycle to impalement TDD
Think of a code behavior, then choose a small increment and
then a test
Write the test for the current increment code and run the entire
suit of tests should fail (Red bar)
Write just enough code to get the test pass and run the tests
again should pass (Green bar)
Review the code and look for improvements small set of
refactoring and run the tests after each (Refactor)
..
https://www.jamesshore.com/Agile-Book/test_driven_development.html
The University of Sydney Page 23
TDD Example
Java class to parse HTTP query string (name-value pair)
E.g., http://example.com/page/to/page?title=Central+Park&where=US
Think
class to separate name/value pairs into a HashMap or class to put one
name/value pair into a HashpMap? Why?
Class QueryString wont return a HashMap, but a method (valueFor(name)) to
access the name-value pairs. Shall you proceed with writing the test?
Count() method instead to return total number of name-value pairs (more
suitable for one increment)
..
https://www.jamesshore.com/Agile-Book/test_driven_development.html
The University of Sydney Page 24
TDD Example Red Bar
..
public void testOneNameValuePair() {
QueryString qs = new QueryString(name=value);
assertEquals(1, qs.count());
}
public class QueryString {
public QueryString(String queryString) {}
public int count() { return 0; }
}
The University of Sydney Page 25
TDD Example Green Bar & Refractor
..
public int count() { return 1; }
Refactor
Change the QueryString name to HttpQuery() noted for next cycle
Another test to try
The University of Sydney Page 26
TDD Example Repeat
..
public void testNoNameValuePairs() {
QueryString qs = new QueryString();
assertEquals(0, qs.count());
}
Thinking
Remove the hard-coded line but not time yet to deal with multiple query string
Testing an empty string would require coding the count() properly
Emerging thoughts (noted for later cycles)
Test the case of a null argument to the QueryString constructor
Deal with the tests duplication tests that needed refactoring
The University of Sydney Page 27
TDD Example Green & Refactor
..
public class QueryString {
private String _query
public QueryString(string queryString) {
_query = queryString;
}
public int count() {
if (.equals(_query)) return 0;
else return 1;
}
}
Refactor (notes):
Rename QuerySting
testNull()
Refactor duplicate tests
The University of Sydney Page 28
TDD Example testNull()
..
public void testNull() {
try {
QueryString qs = new QueryString(null);
fail(Should throw exception);
}
catch (NullPointerException e) {
// expected
}
}
Test the case when the query string is null
Red Bar think of the behavior when the value is null
Through an exception(Null is illegal) simple design
public QueryString(String queryString) {
if (queryString == null) throw new
NullPointerException();
_query = queryString;
}
The University of Sydney Page 29
TDD Example valueFor()
..
public void testOneNameValuePair() {
QueryString qs = new
QueryString(name=value);
assertEquals(1, qs.count());
assertEquals(value,
qs.valueFor(name));
}
Implement valueFor() method to return the associated value give a name/value pair
Emerging thoughts: test for a name doesnt exist (noted)
public String valueFor(String name) {
String[] nameAndValue =
_query.split(=);
return nameAndValue[1];
}
The University of Sydney Page 30
TDD Example Repeat
..
Code passed the tests, but it was incomplete
Multiple name/value pairs
Repeat
The University of Sydney Page 31
TDD Benefits
Help developers to understand the requirements and write better
code
Simplify debugging
Easier to find and fix mistakes in small code chunks
Reduce cost of regression testing
Improved design and code quality
Research shows TDD substantially reduces the incidence of defects
Reuse tests as the software grow, and use it as documentation
The University of Sydney Page 32
Refactoring
Refactoring is the process of changing the design of your code without
changing its behavior Kent Beck
Change the how not the what
Refactoring is reversible!
Analyze the design of existing code and improve it
Code improvements can be identified with code smells
The University of Sydney Page 33
How to Refactor
Refactor constantly in a series of small transformations
Learn from in-depth catalog of refactoring
Refactor intuitively through learning the mindset behind refactoring
Learn how to refactor manually
Development frameworks/tools can help automating some refactoring
Use continuous integration practices and automation tools
Version control system, build and test automation, IDEs
The University of Sydney Page 34
Test Double
The University of Sydney Page 35
Movie Stunt Double
https://i.ytimg.com/vi/xm7kzxXHF38/maxresdefault.jpg
http://cdn.kickvick.com/wp-content/uploads/2014/07/celebrity-stunt-doubles.jpg
https://amp.businessinsider.com/images/525328ce6bb3f78e7afdcbb2-750-563.jpg
https://i.ytimg.com/vi/xm7kzxXHF38/maxresdefault.jpg
http://cdn.kickvick.com/wp-content/uploads/2014/07/celebrity-stunt-doubles.jpg
https://amp.businessinsider.com/images/525328ce6bb3f78e7afdcbb2-750-563.jpg
The University of Sydney Page 36
Test Double
A test double is an object that can stand in for a real object in a test, similar
to how a stunt double stands in for an actor in a movie Google Testing Blog
Includes stubs, mocks and fakes
Commonly referred to as mocks, but they have different uses!
Why test double?
Dependency on components that cannot be used
Reduce complexity, test indecently
https://testing.googleblog.com/2013/07/testing-on-toilet-know-your-test-doubles.html
https://testing.googleblog.com/2013/07/testing-on-toilet-know-your-test-doubles.html
The University of Sydney Page 38
Test Double Types
Type Description
Dummy Pass object(s) that never actually used (to fill parameter list)
Stub Test-specific object(s) that provide indirect inputs into SUT
Spy Capture indirect output calls made by the SUT to another component
for later verification
Fake Objects to provide simpler implementation of a heavy component
Mock Object(s) that verify indirect output of the tested code
The University of Sydney Page 39
Dummy Object
Dummy, dummy parameter/value
Pass object with no implementation (dummy) and never actually used
E.g., Fill in parameter lists
SUTs methods to be called often take objects stored in instance variables
Those objects, or some of its attributes, will never be used in the testing
Preparing the SUT into right state (conform to the signature of some methods
need to be called)
The University of Sydney Page 40
Dummy Object Example
The University of Sydney Page 41
(Test) Stub
A test-specific object that provides indirect inputs during tests
E.g., Object requires data from a database to answer a method call
Used to verify logic independently when it depends on inputs from other
components
Verify indirect inputs of the SUT
It does not deal with indirect outputs of the system
The University of Sydney Page 42
(Test) Stub Example
The University of Sydney Page 46
(Test) Spy
Capture output calls made by the SUT to another component for later
verification
Verify indirect outputs of the SUT
Get enough visibility of the outputs generated by the SUT (observation
point)
E.g., email service that records no. of messages sent
The University of Sydney Page 47
(Test) Spy
Capture output calls made by the SUT to another component for later
verification
Verify indirect outputs of the SUT
Get enough visibility of the outputs generated by the SUT (observation
point)
E.g., email service that records no. of messages sent
The University of Sydney Page 48
Fake (Object)
Objects to provide simplified implementation of a heavy (real) component
E.g., in-memory implementation of repository using simple collection to store
data
SUT depends on other components that are unavailable or make testing
complex or slow
Run tests faster
Should not be used when want to control inputs to SUT or outputs of SUT
The University of Sydney Page 49
Fake (Object) Example
The University of Sydney Page 51
Mock (Object)
Object(s) that verify indirect output of the SUT
E.g., function that calls email sending service, not to really send emails
but to verify that email sending service was called
Calling real implementation during testing is tedious, or the side effect is not
the testing goal
Unlike all doubles, mocks verify correctness against expectations
The University of Sydney Page 52
Mock (Object) Example
The University of Sydney Page 53
Mock (Object)
The University of Sydney Page 54
Test Doubles
Understand the differences carefully and use the one that serve the
verification type and purpose and how it should be run
Dont be fooled by the mocking frameworks terminology focus on the
verification purpose
Read Fowlers Mocks arent Stubs
Check xUnit Test Patterns for more advanced details
https://martinfowler.com/articles/mocksArentStubs.html
http://xunitpatterns.com/
The University of Sydney Page 55
Contract Test
The University of Sydney Page 56
Test Double External Services
Test double to interact with external/remote service
How accurate/reliable is a test double?
https://martinfowler.com/bliki/ContractTest.html
The University of Sydney Page 58
Contract Test
The process of running periodic tests against real components to check the
validity of test doubles results
How?
Run your own test against the double
Periodically run separate contract tests (real tests to call the real service)
Compare the results
Check the test double in case of results inconsistency/failures
Also, consider service contract changes
https://martinfowler.com/bliki/ContractTest.html
The University of Sydney Page 59
Integrated (Broad) Tests
Broad tests done with many modules active integrated testing
Read more for further discussion https://martinfowler.com/bliki/IntegrationTest.html
The University of Sydney Page 60
Collaborative (Narrow) Tests
Read more for further discussion https://martinfowler.com/bliki/IntegrationTest.html
Narrow tests of interaction with
individual test doubles Collaboration
Tests
supported by Contract Tests to ensure
the faithfulness of the double
Contract Tests
The University of Sydney Page 61
Integration Testing
Frameworks
Mockito
The University of Sydney Page 62
Mocking Frameworks
Mockito
JMock
EasyMock
Mountebank
Others
http://www.mbtest.org/
http://jmock.org/
http://easymock.org/
http://www.mbtest.org/
http://jmock.org/
http://easymock.org/
The University of Sydney Page 63
Mockito
An open source testing (test spy) framework for Java
It has a type called spy which is partial mock1
Verify interactions after executing tests (what you want)
Not expect-run-verify (look for irrelevant interactions)
Interaction among objects/components not unit testing
Allows to specify order of verification (not all interactions)
https://github.com/mockito/mockito/wiki/FAQ
https://github.com/mockito/mockito/wiki/FAQ
The University of Sydney Page 64
Mockito Constructs
Mockito Features Description
mock(), @Mock or
Mokito.mock()
Different ways to create a mock
Answer or MockSettings Interfaces to specify how a mock should behave (optional)
when() Specify the mock to return a value when a method is called
Spy() or @Spy Caution: creates a (partial mock) for a given object
@InjectMocks automatically inject mcoks/spies annotated with @Mock() or
@Spy()
verify() Check methods were called with given arguments
http://static.javadoc.io/org.mockito/mockito-core/2.24.0/org/mockito/Mockito.html
Note: call MockitoAnnotations.initMocks(testClass) (usually in a @Before method) to get the annotations to work.
Alternatively, use MockitoJUnit4Runner as a JUnit runner
http://static.javadoc.io/org.mockito/mockito-core/2.24.0/org/mockito/stubbing/Answer.html
http://static.javadoc.io/org.mockito/mockito-core/2.24.0/org/mockito/MockSettings.html
http://static.javadoc.io/org.mockito/mockito-core/2.24.0/org/mockito/Mockito.html
The University of Sydney Page 65
Mockito Example
The University of Sydney Page 66
Mockito Method Call
Use Mockito.when() and thenRturn() to specify a behavior when a method is
called
Example of methods supported in Mockito
Method Purpose
thenReturn(valueToBeReturned) Return a given value
thenThrow(Throwable tobeThrown) Throws given exception
Then(Answer answer) User created code to answer
The University of Sydney Page 67
Mockito When Example
http://static.javadoc.io/org.mockito/mockito-core/2.24.0/org/mockito/Mockito.html#when-T-
The University of Sydney Page 68
Mockito Verifying Behavior
Mockito.verify (T mockTobeVerified, verificationMode mode)
Verifies certain behavior happened at least once (default) e.g., a method is called once
Different verification modes are available
Verification Mode Description
Times(int wantedNoCalls) Called exactly n times, default = 1
atMost(in maxNoOfCalls) Called at most n times
atLeast(int minNoOfCalls) Called at least n times
never() Never called
Timeout (int milliseconds) Interacted in a specified time range
The University of Sydney Page 69
Mockito Verifying Behavior Example
Default mode is times (1) which can be omitted
Argument passed are compared suing equals() method
The University of Sydney Page 70
Mockito Verifying Order of Calls
InOrder (mocks) allows verifying mocks in order
verify(mock): verifies interactions happened once in order
verify(mock, VerificationMode mode): verifies interactions in order
http://static.javadoc.io/org.mockito/mockito-core/2.24.0/org/mockito/InOrder.html
http://static.javadoc.io/org.mockito/mockito-core/2.24.0/org/mockito/InOrder.html
The University of Sydney Page 71
Writing Good Tests
The University of Sydney Page 73
Writing Good Tests
Reliable
Free of bugs, defects or errors
Fast
Should not be counterproductive, will be run very frequently
Keep it compact and readable
Lots of refactoring
Follow recommended coding practices (e.g., naming conventions, documentation)
Cover wide range to show positive cases and errorenous code paths
https://github.com/mockito/mockito/wiki/How-to-write-good-tests
https://github.com/mockito/mockito/wiki/How-to-write-good-tests
The University of Sydney Page 74
Writing Good Tests
Do not mock everything
Its anti-pattern
Understand mocking frameworks capabilities
Mock syntax vs. actual purpose of mocking
Read Fowlers Mocks arent Stubs
Do not mock type you do not own
Third-party library or API owner change the signature and behavior of the API
Contract test ?
Do not mock value objects
Instantiating an object is too painful not a valid reason
Can be a sign that the code needs some serious refactoring or use builders for the value
objects (some tools such as Lombok can help)
https://github.com/mockito/mockito/wiki/How-to-write-good-tests
https://martinfowler.com/articles/mocksArentStubs.html
https://github.com/mockito/mockito/wiki/How-to-write-good-tests
The University of Sydney Page 75
W4 Lecture: Advanced Testing
Techniques 2
W4 Tutorial + quiz
Next Lecture/Tutorial
The University of Sydney Page 76
References
Ian Sommerville. 2016. Software Engineering (10th ed.) Global Edition. Pearson
James Shore and Shane Warden. 2007. The Art of Agile Development
Martin Fowler, various testing articles. https://martinfowler.com/
Gerard Meszaros, xUnit Test Patterns: Refactoring Test Code. Addison-Wesley
Martin Fowler, Mocks Arent Stubs,
[https://martinfowler.com/articles/mocksArentStubs.html]
Gaurav Duggal, Bharti Suri, Understanding Regression Testing Techniques.
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.460.5875&rep=rep1&t
ype=pdf
Bernd Bruegge and Allen, H. Dutoit. 2009. Object-Oriented Software Engineering
Using Uml, Patterns, and Java (3rd ed.). Pearson
Michal Lipski, Pragmatists: Test doubles: Fakes, Mocks and Stubs.
https://blog.pragmatists.com/test-doubles-fakes-mocks-and-stubs-1a7491dfa3da
Reviews
There are no reviews yet.