Perl has excellent testing support with frameworks such as Test::More, these tools present test results in a standard text format call TAP. As far as I can tell all, of the current CPAN testing frameworks are designed to test Perl Modules (.pm files). Because of this they are designed to be consumers of the services provided by the modules under test. (Update 10/Jan/07 -- Correction) However I don't write many modules -- most of what I do are either:
  1. Smaller programs, i.e. programs that are themselves the consumer of services.
  2. Testing larger system, such as ALM Suite. This requires testing the across multiple applications and the information flows between them, many of these applications are not even written in Perl. The problem is more complicated if you consider that when testing integrations between the suite compoenenets it is necessary to track the effect of tests across multiple applications and their data repository during a single test transaction.
At the "extreme" end of my requirements is the need to provision a complete virtual test environment to test the installation process of a product suite and run a full feature regression test, this is certainly possible as Sam Tregar has written. Whilst this can be done with current techniques, for example the use of Perl to control VMWare test environments, I need to get a better understanding of the best way to use this with TAP. I am also considering using user mode Linux rather than full virtulisation as well because it consumes less system resources. Here are some speculations on how I might approach this:
  1. Take my application code and re-factor it into a module and a thin application wrapper. The wrapper can then be replaced with the TAP testing program
  2. Use Expect.pm to implement the TAP testing program and run the application under test. Needs further analysis
  3. Use tools such as WWW::Mechanize to test a web application (thanks to Randal L Shwartz from Stonehenge for showing me how to do this at the OSDC tutorial this week)