|
Perl 5 Wiki
Testing
Perl Testing ToolsCPAN has a good selection of modules to help with automated testing. In addition to testing applications written in Perl, it's also common to find Perl used as an automated testing tool for projects developed in other languages like C or C++. If you're new to Perl testing, start by reading Test::Tutorial. After that, playing around with Test::More should give you a practical feel for how easy it is to write tests in Perl. Testing StrategiesFor an introduction to Software Testing principles please see Wikipedia's article on software testing. Typical testingIn the vast majority of projects, formal testing is left to the end, if it is performed at all. Developers will probably test each component as they write it, but these tests are usually haphazard and thrown away. Test Driven DevelopmentIn test-driven development (TDD) test cases is as important (if not more important!) than writing code. TDD should be viewed more of a design process than a testing process. You use tests to drive the design of your program, rather than using tests to verify that your program does what its designed to do. Test-driven development can be summarised as a six step process:
Thus you write cases to cover your code's interface and the correct usage of each subroutine before you even start to think about the algorithms you may need to use. TDD code may take longer to write initially, as the author usually writes more lines of tests than lines of code, but it has a much shorter formal testing period as well as an excellent test suite which prevents code regression. For more information on TDD see the Wikipedia entry on TDD Unit testingIn unit testing, code is tested in small parts (units) to ensure that each of these work correctly. Interface testingInterface testing ensures that the code interfaces work as stated in the specification. These are useful in discovering cases where code authors have changed the argument order without consultation with other project members. Regression testingFixing bugs in one part of code often causes them to reappear in other parts of code - sometimes with the same symptoms. To prevent code from "regressing" (suffering from bugs that have already been fixed previously), most people recommend regression testing. This is just common sense. Instead of performing ''throw-away'' tests, regression testing suggests adding these tests to your test suite. Thus once you find a bug, you write a test which fails due to the presence of that bug. Once that test no longer fails, the bug has gone. These tests are collected into a test suite and run after every code change. Then if bugs re-emerge it's much easier to locate and remove them. Black box and White box TestingSimple black and white box testing can be carried out with any of Perl's testing tools. Although there is a conceptual distinction, this is rarely evident in the resulting test suites. Coverage testing can be carried out with Devel::Cover. Black box testingBlack box testing treats the code as if it were unreadable. Instead, much like interface testing, the tester writes tests based on the specification. For example if the specification states that a given field will take a date, then the tests will include valid dates, invalid dates, things that look like dates, things which don't. Black box tests are easier than white box tests, as they should not be affected by changes to the code internals. White box testingWhite box testing relies upon intimate knowledge of the code internals. This knowledge is then used to test all the code assumptions and boundaries. For example, if the code assumes that a particular field will only be 20 characters long, white box testing includes tests for 19, 20 and 21 characters. Prudent testing will also include tests for 0 and 1 characters for the same field. White box testings aims to ensure that the code behaves correctly when its assumptions are incorrect. Coverage testingCoverage testing is a special case of white box testing and measures how well the test suite exercises the code. There are a range of metrics for coverage testing:
Both 100% conditional coverage and path coverage imply 100% branch coverage. Path coverage is usually the hardest to achieve, and as such path coverage is usually limited to paths within subroutines or other smaller sections of the code. Achieving decent coverage testing is a worthy goal, as it often highlights areas of your code which are unreachable and tests areas of your code which are not usually exercised. Coverage ExampleAn example of using Devel::Cover on Class::DBI can be found at http://perl.net.au/Class-DBI-cover/ . TAP: Test Anything ProtocolThe Test Anything Protocol (TAP) is a standard format for displaying results of tests. An example TAP session may look like: 1..9 ok 1 - Beverage::Coffee loaded. ok 2 - Cup creation. ok 3 - Kettle location. ok 4 - Boiling water. ok 5 - Black coffee. not ok 6 - Milk location. # Milk not found in fridge. ok 7 - White coffee. # Skipped, no milk. ok 8 - Espresso # Skipped, espresso machine not available. not ok 9 - Non-dairy creamer # TODO, unimplemented. The TAP output can be easily parsed by Perl's testing frameworks, and numerous modules exist to generate TAP. The planTAP output usually starts with a plan line: 1..9 which specifies how many tests are to follow. The above example specifies 9 tests. This line is optional, but recommended. Should a test suite die part-way through, the plan allows the testing framework to recognise this situation, rather than assuming that all tests were completed. In the case of rapid test-suite development, it can be irritating to have to remember to update the number of expected tests. As such it is possible to specify that there is no plan. Typically this results in the test harness counting the number of tests run and creating a plan line as the last line of TAP output. The test lineAfter the plan come the test lines. Each of these can be broken down into several parts: ok 1 Description # Directive
DiagnosticsAny other lines are assumed to be diagnostics. In the case above, our diagnostic line: # Milk not found in fridge. tells us why the milk location test failed. We can also use diagnostic lines to segment test output, for example breaking up test results for database connection, template construction and data parsing. More information
For more information on the TAP visit the Test Anything Protocol Wiki or read the TAP documentation. Test::TAP::HTMLMatrix can be used to create a colourful html summary of TAP results. TAP-compatible tools for other programming languages
Testing examplePerl tests are often written using Test::More. For those who haven't written many tests before, Test::Simple may be an easier place to start.
# Specify our plan, how many tests we're writing
use Test::More tests => 8;
# or alternately, if we don't know how many:
# use Test::More qw(no_plan);
# Check that our module compiles and can be "use"d.
BEGIN { use_ok( 'PerlNet::TestMe' ); }
# Check our module can be required. Very similar test to that above.
require_ok( 'PerlNet::TestMe' );
# There are a number of ways to generate the "ok" tests. These are:
# ok: first argument is true, second argument is name of test.
# is: first argument equals (eq) second argument, third argument is name of test.
# isnt: first argument does not equal (ne) the second, third is name of test
# like: first argument matches regexp in second, third is name of test
# unlike: first argument does not match regexp, third is name of test
# cmp_ok: compares first and third argument with comparison in second. Forth is test name.
# Here are some examples
ok( (1+1) == 2, "Basic addition is working");
is ( 2 - 1, 1, "Basic subtraction is working");
isnt( 2 * 2, 5, "Basic multiplication doesn't fail");
like ("PerlNet is great", qr/PerlNet/i, "Finding PerlNet in a string");
unlike("PerlNet is great", qr/PythonNet/i, "Not finding PythonNet in a string");
cmp_ok($this, '==', $that, "Comparing $this and $that with integer ==");
Test::More provides a few other functions for testing including is_deeply which compares complex data structures for equality and can_ok, isa_ok which test object functionality. For more information, read the documentation. Mock objectsSometimes just testing your code can be really hard, because you need to setup so much before you have a system your code can use. Say you have written code to find and delete duplicate customers records. For historical reasons, some customers are in an LDAP directory, others are in a Oracle database, and still others are in an XML file in a server in another office. To test your code before you go deleting records off the live systems, you might try to setup duplicates of all of these, a complex, costly and difficult process. Or you could "just fake it". What if, in your code, where it tries to use these various systems, you could substitute in something that behaves just like an LDAP server, or Oracle DB, or remote server, but is in fact just a "mock-up" of a real system. Like those mock streets in Hollywood - from the front, they look just like a shop or garage, but behind they're just framing and sandbags. Mock objects are just like these mock streets - they have the same API as the real object they're mocking, but inside they do almost nothing - maybe they just return an "OK" status, something like that. In complex projects, many objects rely on other objects or layers of infrastructure. This can make testing these objects difficult when these collaborators do not exist, or it is impractical to instantiate them. Further, using mock objects allows you to isolate test failures from external influences. Take the example of an LDAP directory. There is a Perl module for connecting to LDAP servers, Net::LDAP. If you write code that connects to an LDAP server, your probably going to write lines of code like
package My::Deduplicator
# make use of LDAP server
use Net::LDAP;
sub connectLDAP {
my ($server) = @_;
return Net::LDAP->new($server) || "server $server is down";
}
1;
Now what if you wanted to test how your code behaves if the LDAP server is down ? In our example, it should return a message. You might be working in an environment where your not allowed to turn the LDAP server off, or you dont want to change your code to connect to a non-existent server. What you want is to mock the Net::LDAP module, so that the mock version behaves just like the real one does if the requested LDAP server is down. Looking at the Net::LDAP POD, we can see that if the connection cannot be established, Net::LDAP->new() returns a false value. It might be undef, 0, or '0'. It doesn't matter, as long as our mock version also returns false as well. If you carefully construct your test file to create a mock LDAP server first, then you can control exactly what happens when the call to Net::LDAP->new() is made.
# in t/ldap.t
use Test::MockObject;
use Test::More qw(no_plan); # tools for testing
# pretend the LDAP server is down...
my $mock = Test::MockObject->new();
# LDAP server is DOWN
$mock->fake_module(
"Net::LDAP",
new => sub { return; },
);
use_ok("My::Deduplicator"); # ends up using mocked version we just created
# test response when server is down
is(My::Deduplicator::connectLDAP("MockedServer"), "server MockedServer is down", 'expected failure message');
Test::MockObject allows you to set all kinds of actions for your mock objects - return the same value all the time, cycle though values, log the call etc. You could mock an LDAP server that failed every third connection request, or at a random interval, or dropped connections after the bind stage? What about testing how your code behaves when you connect and bind OK, but the LDAP server falls over on the second search query - how would you set that up in real life ? The common coding style for testing with mock objects is to:
Mock objects are very useful for situations where there is a lot of external complexity, where time is involved, where things happen in lots of different orders or there are complex chains of interaction e.g. a GUI spawns a process that connects to a server to FTP a file. Specialised mock objectsSome mock object forms are very common, such as database replacements. Some of these are available on CPAN:
Further information
Perl Testing Resources
Recent perl.com Articles on Testing
Other Testing Resources
CreditsThis page was originally developed at PerlNet. See the original page for the original text, history, and list of contributors. |
-
18630 views
-
34 revisions
