|
||
"Committed to Excellence in Software Testing"
A New Test Approach to Software Testing Prof. David Powell MS Hon. 20 August 1999 BACKGROUND Testing information systems is a major concern for many organisations. It is often perceived as costly, time consuming, boring to do, and difficult to manage. In some organisations testing had become another word for contingency. Testers are sometimes blamed for having found faults which cause the application or system not to be deployed according to plan. But testing is unavoidable if even greater problems in production or in the live business environment are to be avoided. Automation of the test process with test tools has drawn a lot of attention in the industry as a possible means of relief. These tools are known as CAST tools: Computer Aided Software Testing. But many organisations and project managers have found that they can create more problems than they solve, and many have been consigned to shelfware. With so much negativity towards testing around in organisations, it’s surprising that anyone bothers to test at all !! Yet without a proper approach to testing things will get worse instead of better. After all if you automate chaos, all you’ll get is faster chaos. In 1994 a co-worker was called into the Amsterdam Stock Exchange to try to help a test project on a mission critical system (their electronic trading system) which was in trouble. From that project came a new concept for way to test, which removes the test design from the test tools, uses a powerful engine to interpret them, and uses the automated test tool to run the tests against the application. The Stock Exchange received its system on time and to the required quality standards, and today they use the tests that were produced as the foundation for their regression test suite which is run every night. A group of us (made up of test project managers, project directors and senior management consultants) worked on developing this structured approach (which CMG has now called TestFrame) in order to bring benefits to testing projects so that organisations can truly reap the advantages of automated testing. And when you think of structure, don’t think of inflexibility - this approach can be fitted into an organisation for both manual and automated testing. Hundreds of projects have now been completed, many in multinational companies across a whole range of industry and on mission critical systems - in one case where the system can only fail once in a thousand years !!! This structured approach to testing has become widely accepted as vastly improving the project planning, the time to market of the system, and the quality of what is delivered. In the following sections this article deals with a number of area about the TestFrame approach and posses some new testing ideas for project managers to consider. TEST APPROACH & PROCESS The use of automated test tools (known as "Record and Playback" or "Capture and Replay") added structure to the previous manual test approach, and gave project managers a number of advantages. These advantages were:
But there were also disadvantages to these test tools:
To overcome these problems we decided to remove the test design and build from the test tool and do this in spreadsheets. These spreadsheets are called Test Clusters. The Test Clusters contain the Test Conditions which are required to be tested, the physical test cases, an Action Word library (Action Words are described later), and any notes or special requirements that are needed to be able to run the tests. This overcame some of the major problems. Firstly, if the tests are to be run manually then there is structure and quality brought to the test design and build which has historically been lacking in manual tests. It also means that, if necessary, strategic test standards can be introduced first, and then the tests can be speeded up at a later date with an automated test tool. Secondly, test cases could be built by the business users and this offers huge advantages. Users most often make their tests more applicable to the "real world" environment. And with current salaries of computer staff a lot higher than that of users, this also offers the opportunity of lower project costs. Having users doing the test analysis can reduce any "resistance to change" from among their peers. Users who are involved as test analysts have the opportunity of directly influencing the software they are going to use (and not just the look and feel issues) and therefore buy in to the software. When rollout occurs, their peers are less likely to complain about having had it "forced" on them. There is revolutionary concept about having user involvement doing the test analysis and building the test cases. If users do this role does the business need to undertake a separate User Acceptance Test _ Finally, if for any reason test tools were changed, then the test cases can still be used by the new test tool, which cannot happen at the moment with the "record and playback" approach.
To allow these test cases to be read and processed by the test tools, "Action Words" are created by the Test Analyst and put into column A of the spreadsheet. Beginning with column B, the test data is inserted on the same line as the Action Word. An Action Word could be "enter client" or any similar description. The Action Words are usually specific for the application that is being tested. The field names are shown on the line above, but are not preceded by an Action Word. This way they are ignored by any test tool when the tests are run. The Action Word library serves a very important purpose. Firstly it stops the same Action Word being used for a different part of the application, and secondly it helps with the maintenance of the tests. The library stores all the Action Words on a separate worksheet within the Test Cluster. By anchoring the field names to the Test Cases where the Action Word is used, this allows any new fields in that screen to be added just once – which saves a lot of time when doing a batch run of thousands of records !!
The test automation is regarded as a separate activity, and is done by someone with some programming skills. To interpret and execute the commands in the Test Cluster a Navigation Script is written. This is usually written in the script language of a chosen test tool. Most test tools in the market have a language which is powerful enough to make implementation of a Navigation Script possible. The Navigation Script consists of two parts. Firstly there is the standard part which is referred to as the Engine. This contains standard functions, and has recently been produced as a ".dll" file with code which the test tool recognises. The second part must be specifically written for the application that is being tested and contains functions for each Action Word which tells the test tool where to put the test data associated with that Action Word. If a new field is added for a screen then this is added once in the Navigation Script in association with the relevant Action Word concerned. Should a field be deleted from a screen then that field is simply commented out of the script and all the associated tests will still run. TEST PLANNING There are various phases in the standard TestFrame project. These are:
But above everything, the objective of the project must be known by not only the project manager but by the team as well.
One project for a life company, was based around a single business requirement - to be able to launch new products through its direct salesforce, within 6 weeks and to be able to effect rate changes within 48 hours – including mailing time of the updated program CD. When it came to testing this application, the tests which were produced during development were going to form a regression test suite which would be run hundreds of time over the lifetime of that system.
In Rotterdam, their flood barrier was tested using TestFrame in order to achieve a different objective. This time the system was to have a failure rate of 1 in 1000 years !!! Time to market was of less importance than the quality of the system installed. The objective of the project needs to be defined.
So when should you start testing _ Frankly as soon as there is something which you can test. So many test projects start too late because a lot of people are still working with the mind-set that says we must wait to have the application before testing. On a recent project for an international medical insurance company, we tested the functional specifications and found on average 30 - 50 faults in these documents per system area, and there were 17 areas. Our total was over 600 faults. The project manager decided to fix these faults when writing the detailed design documents which included updated screen designs, and we tested these too. On average a further 10 faults were found in each of the these, giving a total of 870 faults found. Based on GILB, Software Inspection, 1993, the cost of fixing faults at different stages of a project lifecycle was shown. If it costs $1 to fix a fault at the requirements stage and $4.50 at the detailed design, this rises to $100 at the operational level. And this doesn’t bear any relation to the time delays, reputation damage, and lost revenue caused through finding these problems so late. In the medical insurance project, if it had cost the project $1365 to fix at the stages we found them, it would have cost the client $87000 after they had gone live !! I estimated that the true figures were at least six times higher than this. What is the best test tool for the project _ So often decisions are made on a project by project basis and not with the bigger picture in mind. Maybe it’s because project managers are rated on the success of their projects, that few seem to look at the wider picture or implication of their decision. What is best for future developments as well as the current project has to be taken into account. INVESTMENT
A project has recently begun in a 30% government owned organisation, which has had no formal testing approach over the 3 years since it’s formation. Most of the development work has been done in-house. This is not an uncommon situation. A global bank I met during the last week has no formal testing solution, preferring to leave it up to each of their 40 project managers to decide on the best approach based on that specific project. Building a regression suite of tests for an existing application takes time and money to achieve in the short term, but leads to big returns in the long term, as both these organisations now agree. Both are currently working on e-commerce sites for the internet, which makes having a reusable test suite even more important. With the internet, where sites can change daily, having a regression test suite is more a necessity than luxury - after all the price of failure is more immediate, and a competitors site is just a mouse click away !! When completed the part government owned organisation’s regression test environment will sit between the pre-production and the live environment as the final quality check before changes and new applications go live. Another bank which is using TestFrame, has reported saving 4000 man days from projects by having created an automated regression test suite for all their business applications. On average they release 2 updates per day !! In fact this organisation went as far as creating a "Model Bank" to simulate the live working environment and to allow them to gauge the impact on the business of new software and to train people. CONCLUSIONS There is so much more that could be said about managing test projects, solutions for improving test projects, training testers and users, or the different htmlects of testing applications. The main benefits of the approach outlined above are flexibility, manageability, cost effectiveness, quality and motivation. It is important to notice that the approach, although extremely useful, is no magic wand. Testing remains a complex and difficult activity that should never be under estimated, with or without approach outlined in this article. Strategic solutions require effort and time, but the rewards can be massive – certainly they are big enough to make it worth trying to achieve them. ============================================================================== AUTHOR PROFILE At the time of writing this article, David Powell was UK TestFrame Manager for CMG Plc, and draws on his practical experience of establishing strategic test solutions for international client organisations. David can be reached on dp@softtest.org ============================================================================== |
||