"Committed to Excellence in Software Testing"

Test Management - Solutions for Project Improvement

Prof. David Powell MS Hon.

Organisations operate in a changing business world where they are daily confronted by new developments, new requirements, greater expectations and bigger desires on the part of their clients. Information Technology Consulting has come to play a greater role in addressing these changes. No organisation or institution can now afford to ignore IT because the commercial and operational benefits are too great. Modern day technology offers many opportunities for astute businesses.

The quality of the information flow is also becoming far more important - critical would not be too strong a word to use. The growing integration between business processes and IT means that any interruption to the normal operation of the computer systems and the business side can have disastrous effects and expensive consequences. One major US organisation recently lost it’s internet booking system for over 3 days, meaning anyone wanting to book with them from overseas either had to call the States by telephone or contact their rivals website. My guess is most did not call !!

The importance of prevention rather than cure of such interruptions has now become both recognised and accepted. Software testing (or Software Quality Engineering as it is called in America) has developed into an essential specialist area within the IT industry.

In practice, however, the controllability of the testing process often proves limited. While the design, development and administration of systems is usually approached very systematically, the testing process has (and in many organisations it still is) frequently conducted on an ad hoc, unstructured basis or as an after-thought.

Primarily the process has to be appropriate to the organisation in which it is to be adopted - it must fit rather than conflict with the other practices and procedures. Secondly, testing can only offer quality control if it is approached structurally and if what is produced allows for easy maintenance throughout the lifetime of that process, system or application. And finally, the use of tools and accessories is essential to any modern testing process, so as to provide greater long-term returns.

This paper looks at the need for testing to be a framework in which the areas mentioned above are used. Due to testing being a rapidly developing professional area, the structure has to allow room for new methods, services and tools to be developed.

Testing is now an essential process for any organisation and provides certain guarantees regarding the correct functioning of systems. I hope that this paper provides you with food for thought when considering the solutions for improving the testing in your organisation.

4 main reasons for testing

There are - at least as far as this paper goes - 4 main areas or reasons why organisations undertake software and system testing.

Management of Risks

Risk Management has grown in importance within the IT industry, and I doubt I am the only person to be in software testing with a Masters degree in Risk Management.

Surprisingly many organisations still do not see that by managing the risks within both their development activities and within their testing work, they can achieve valuable long term benefits.

Management of Costs

"But the problem is that extensive testing to reduce risks costs too much"

That is a subjective attitude. It is not always apparent how much it would have cost had the faults not been found until after the system was in the live environment. How many orders would have been lost, how many helpdesk calls would have been logged, etc.

And costs are not always direct costs. A lost reputation might costs the organisation it’s very existence. If a factory or office closes, it is not just those workers who loose their jobs, but the impact on the local economy can be many times greater.

Management of Time

"Testing takes to long especially as the time to market for new products and services is becoming ever shorter."

The increasing globalisation of markets is having a direct effect of increasing competition. And history shows us that things will not get slower and are more likely to get faster.

Products have to be brought to the market as quickly as possible. Moreover, in practice IT development projects appear to become more time consuming as the delivery date approaches.

The paper addresses ways of removing the pressure points on the testing needed, to allow the business to keep ahead of the opposition.

Extensive manual testing aimed at achieving a quality to market solution is extremely time-consuming. Automated test tools whilst offering a number of significant advantages, also have associated issues which need to be borne in mind.

Management of Quality

The quality of the application or system delivered to the market is now of crucial importance. Whilst double entries in mailing lists or address files are a nuisance, double entries in an accounting package can cost organisations their futures.

Even more serious problems can arise when modern technology throws a 'spanner in the works' of the whole business operation. Bad quality will cost you customers and will damage your reputation. In short, systems failure can have far-reaching consequences for the core business of any company or institution.

Extensive manual testing aimed at achieving a quality to market solution is extremely time-consuming.

The Test Manager

Enter the test manager.

From experience some feel they should have first train as a juggler - after all, their experience shows that will be required to keep many balls in the air - without dropping them of course - and to juggle conflicting demands.

Their mission is to help the organisations bring their products and services to the market more rapidly and to the pre-determined level of quality.

Over the next few chapters we will look at each of these areas in more detail.

Management of Risks

The time when testing takes place is often left to chance. It's hardly surprising then that chance plays a great part in the reliability of the IT system !

To avoid falling into this trap, testing must become a serious project within an organisation. This starts with a thorough Risk Analysis, an adequate testing budget and sufficient resources to do the work.

This may seem costly and time consuming, but the consequences are worse still.

Risk Analysis

A Risk Analysis is an absolute necessity in order to give a test manager an advanced indication or warning of where problems might be found.

In his training course on Object Orientated testing, Lee Copeland spends the first half looking at the issues, challenges and problems which OO developers face before asking the question about where should we as testers focus our test effort.

These risks or potential problems may be within the system to be tested, but could also be within the organisation, the specifications, the test environments (or lack of them), the designed tests, etc.

A person (or on big projects this might be a team of people) should be appointed with the authority to determine whether there are risks and how they should be addressed.

These risks need not simply be "known" risks. They could be issues that are identified as being a risk should they happen.

For example, on one project the organisation concerned had a clear desk policy, which meant that anyone could sit anywhere, and the lack of assigned desks was identified as a potential risk. As the project grew there were more people than desks, so the lack of facilities materialised as a risk. Because this had been highlighted earlier to management, a solution was already being worked out, bringing time and cost management benefits.

Risk analysis is extremely important on projects where there are more than one supplier. I worked on a project which had 3 major IT consultancys building the system plus the client doing some of the development work. All were dependent on each other but because of the rivalry and competitiveness very little communication took place between the consultancys.

If this area is taken seriously, it will assist in relieving pressure on time management, cost management and quality management.

Cost Management

Business Focus

In really good organisations the test managers are involved from the earliest stages of project planning and costing. This becomes even more important in consultancy projects (especially fixed price contracts) where the clients do not appreciate delays or cost overruns.

Normally at board level, an agreement for capital investment is made and will not be changed. The business is always focused on the financial htmlects and good test managers should at the minimum have some understanding of the workings of the business in which they are working.

Fault Costs

The later faults are found the more money they cost to fix. In 1993 there was research done on this which produced the following information:

Adapted from: GILB, Software Inspection, 1993

The above chart shows that, if a fault found during the requirement stage cost £1 to fix, then it would cost 100 times that amount if the same fault was found after the system had gone live. In my experience it is likely that a fault found at the requirement stage will cost at least £50 to fix (including mantime and retesting) which means the same fault, if found after the system had gone live, would cost £5000 !!

So the earlier faults are detected then the more this will meet the business objectives of no increased costs and no delays. It also gives the test manager one other opportunity - to bring the project in under budget !!

This approach was tried on a recent project and slide 5 shows the effects that this had. There were 17 parts to the overall system, including interfaces to 3^rd party software such as PeopleSoft. The project was a RAD based development with each part of the system being built in isolation, and the inherited plan was to test it in the same way.

On average 30 faults were found when going through the functional specifications (which the client had signed off !!) giving a total of 510 faults.

Due to time constraints it was agreed that these would be fixed when the detailed design documents were produced. Once these were available the same exercise mentioned above was repeated – note we did not just re-test the fixes. A further 10 new faults were found on average across the system – a total of 170.

Based on the above model, if this had cost £540 to fix the functional specification faults and £765, the total would have been£1305. In reality it costs around 50 times this to fix ( an estimated £65,250).

But this is insignificant when compared to the amount had they been left until system testing as is the norm with that organisation.

Management of Time

Testing must be given the attention it deserves. It is simply no longer enough to approach the examination of information systems in a 'we'll do it if we have time' way. This is hardly an improvement on the old philosophy that testing is another word for contingency !!

Test Squeeze

The above model shows at a high level the normal project phases at the start of a new project.

But once things start, they have the alarming habit of slipping, all except the deployment date. So the model normally ends up looking like this:

Parallel Test Projects

Parallel testing allows for the test development to be done in parallel with the system development. It also allows the documentation produced during the system development to be tested as outlined in the previous section.

Obviously if the tests are to be executed manually then the Analysis phase (where the tests are designed and built would extend and the test execution might start earlier to allow longer to run the tests needed to meet the required quality standard.

Automated Test Tool

Testing tools can speed up and simplify the testing process. The actions normally carried out by a human tester (such as mouse movements and keystrokes) are taken over by a computer program. Such actions are included in a Test Script which functions as a control program for the test tool. The Scripts can be modified, as they have an underlying programming language.

The most significant advantage of the use of testing tools is that once the Test Scripts have been recorded they can be used time and time again without any intervention on the part of the tester. For example a huge number of tests can be carried out unsupervised at night or a batch run simulated.

The automation of routine and often boring testing activities has the additional advantage of making the testing more reliable because the human factor is completely eliminated. There is no possibility of not being able to reproduce an error.

Maintenance of any supporting programs is an important consideration. There is little or no benefit in using these programs if they take longer than manual testing.

The use of automated is a means to an end and not the end in themselves. Test tools add speed but they do not necessary add quality to the tests being run. After all if you automated chaos you simply get faster chaos.

Management of Quality

Earlier I gave the statistics of faults found on a project and the cost savings that this had on the project. Validation of the earlier development stages takes place against the former stage to make sure that what has been done is in correct.

Validation can be described as "Providing documented evidence with a high degree of assurance that the system will consistently produce a product or perform a process which meets the predetermined specifications and quality attributes".

Maintaining quality through out the design and build phase will reduce the number of faults found during System Testing, save time, and reduces costs.

But as Test Managers there should also be quality management over the tests which are produced. Good quality tests have a longer lifespan and most often can be used again and again by regression testing and then can be re-used when the system goes live for on-going maintenance.

Responsibility of the Test Manager

Test model

There are phases involved in setting up and implementing testing within an organisation. The model shown above does not rely on a predetermined and rigid plan. The arrows in the figure indicate that the areas need not be carried out in any set order and interact with each other.

An area can also be repeated more than once. In the case of the Strategy and Planning this should be a revisited if any changes are made to the system which will be tested or changes made to the development project.

The four phases form part of each new testing project. They can also be repeated as necessary depending on the test results.

Strategy & Planning

The following figure outlines a serious of questions which a test manager can ask at the start of the project to help them establish the basis for the Test Strategy.

The Strategy and Planning should not be a long drawn-out procedure. It is intended to:

• determine the feasibility of the project and carry out a Risk Analysis
• define a test organisation, procedures, tasks and responsibilities
• create a clear Test Strategy for the project which should be read by all team members
• set priorities for the testing
• establish which test techniques and test tools will be used during the project
• define the infrastructure needed so that the Test Analyst can record information and the Test Executor can begin to install and use their programs, develop the automated test scripts and - of course - test the system.

The Test Strategy will form the foundation for the testing team in carrying out their activities, and will consist of:

• organisational elements;
• the testing strategy;
• acceptance criteria;
• the various testing activities;
• any products to be supplied during the course of testing, and who is to supply them.

The following quality criteria must also be included:

• the extent to which the tests will examine each of the system functions
• the way in which the test components and the implementation of the tests take place
• the manner in which the results can be carefully and controllably established.

The main problem in drawing up a timetable for testing is its dependency on the development processes of the systems. As we have already seen, any delay in the development process will almost always lead to delays in the testing process.

In formulating the planning, the following should also be considered:

• test co-ordination during the project;
• equipment for the test environment;
• structure of the test;
• test automation of all or parts of the test;
• execution of the tests;
• assessment of the test results and how these are to be logged and reported;
• the implications of any retesting required should problems be found;
• control of the test products supplied.

What should the testing be _

The following diagram outlines 3 areas which need to be addressed:

Requirements driven:

• Ensuring the business needs have been correctly described
• Ensuring compliance to the requirements

Data driven:

• System functions correctly handle data, including special data cases - Have special cases been identified and tested, or are these cases overlooked

• Data has integrity - All data handled by the system is valid and correct from a business point of view, especially after many changes have been made to the system

• Data is safe (backup & recovery) - Data should be "owned" by the business unit; it is a company asset; is it protected as such

Error driven (or destructive):

• Anticipate human activities - Provide a user interface for software that appropriately captures and gracefully handles human inconsistencies in the system use and verify that software processes which interact with or are dependent upon human/manual processes are robust

• Errors are benign - When errors do occur, make sure data is not corrupted, business functionality is not compromised, etc.

• Worst-case - What are worst cases for system use, availability, number of transactions per day/hour, database growth rate, etc.

Test Design

At each level of test development, consideration must be given to what is going to be tested and how can we know if the test is complete _ Each test is divided into a logical structure, for example, by system area.

Next the Test Conditions are established for each of these system areas. These are then broken down into individual Test Cases which exercise the Test Conditions.

How these proceed depends on the test execution method. If the tests are to be run manually then each Test Case has Test Lines and Test Data added which will be input into the system. The very structure of this approach increases the quality of manual testing, and the tests can be produced in spreadsheets or a database.

If the tests are to be run using an automated test tool in it’s standard Record and Playback mode, then the test data is recorded straight into the test tool, and the test cases saved within the software.

Test Conditions

Test Conditions are an important part of the overall testing process. They are a description of what needs to be tested in order to prove that the system operates according to the functional or other specifications. The Test Conditions also indicate the depth of testing required.

For example, on an client record system, the Test Conditions identified from the specifications could be that:

• a clients record can be added to the system;
• a client record can be amended to show new information;
• a client record cannot be deleted.

The expected results are normally logged at this point so that they can be compared with the actual test results to see if a fault has occurred when the tests are run.

Test Cases

The Test Conditions are then translated into Test Cases.

For each Test Case, the computer functions are tested to ensure that the test results comply with the expected results.

For example, the tests for the first Test Condition shown above could be:

• a client record can be added with full details
• a client record can be added with partial details **
• a client record can be added with mandatory details only
• a client record cannot be added without mandatory details

** There may be a number of tests run to prove different ways of adding partial details, but it is not always necessary to test every perceivable permutation unless this is part of the system specification .

Test Lines

The definition of Test Lines is the final phase of structuring the test preparation and contain the data to be input into the system. They can also be details of any expected messages or functions that need to be performed, such a pressing the F9 key at a certain point.

Test Execution

Again the method of execution will have a project impact. If this is done manually then there will be need for more testers within the test team. If this is done with an automated test tool then this will need people who can use these tools and are able to amend the underlying Test Script if necessary.

The Test Report

A clear overview of the test results, of passes or any problems that have come to are essential in order to be able to deal with faults. Any carelessness can undermine the quality to market of the application.

With automated test tools they automatically produce a Test Report in which changes in the expected result are shown. These may be minor (such as a wrong screen message) but could be more serious (such as a system which cannot cope with a certain number of transactions)

With manual testing there needs to be mechanism put in place to log the test results and to log any faults found during the running of the test.

Results administration

If an error requiring attention is discovered in the system, it is important that the procedure for dealing with this is carefully monitored. The administration should describe and follow through all errors. Everyone involved in systems development or maintenance should be informed of the errors or problems in the relevant version.

Faults and errors can be divided into various categories, depending on the gravity of the problem. The most important category is one where the implementation of the system becomes impossible. The second category can include those problems which affect the proper functioning of major parts of the system. A third category fault may be for the more cosmetic problems, such as spelling mistakes in the help file, or an incorrect error message.

This division into categories is useful for any management report because it clearly indicates the nature of the problems. By regularly supplying such overviews, you will be able to closely monitor how the quality of the system is developing.

Repetition of some tests may be essential if major errors requiring attention are discovered in the system, and periodic full regression tests are also advisable, especially on RAD projects.

Once the tests show the required results, you can comfortably move onto the next phase.

Maintenance

Test maintenance can be roughly divided into four components:

• setting up and maintaining the Test Reports
• managing the results of the tests
• managing the different versions of the system being tested
• transfer of the test products to the production/live environment.

Short Term Focus vs. Long Term Vision

Re-usable testing products

Testing products are often subject to poor reuse. Many organisations find that they have to set up tests from scratch each time there is a minor modification to the system. Frequently this is due to negligence.

The requirement for a time management means that tests produced during the development phase of a system must be easily maintained and totally reusable.

On the other hand the quality management demands that the various stages of testing can produce an accurate risk assessment.

An easily maintainable test suite should be available from the very outset - one which can be used time and time again with just minor modifications when necessary.

The time savings will become particularly noticeable when updates involve only five to ten percent of the system. In testing the other 90 – 95% will be retested using existing test material.

Because the various testing resources are used time and again, the initial investment pays for itself after a few repetitions.

Strategic Implementation

In conclusion, the only way to address the four key areas mentioned above (time, quality, cost and risk management) is to have a strategic corporate solution to meet all the organisations testing needs.

One organisation I met had 40 different development project teams who each focused on one particular system which was used within that company. Each project manager defined how their development work was to be tested, giving a possible 40 different ways of testing. Once they were happy with the system it went live and the tests passed to the maintenance team.

How much easier, cheaper and faster it will be for the organisation when they adopt a strategic test approach, method and standards across all 40 projects.

A reusable test-suite reduces the costs and time to market for upgrades and new functionality, and I would suggest quality to market will be dramatically increased because the risks are reduced.

About Interim Technology

Software Quality Management

With today’s focus on achieving more with fewer resources, applying consistent quality to the software engineering process is critical. The costs of inadequate quality management include lost opportunities, excessive corrective procedures, reduced competitiveness, and reduced profits.

Interim Technology, The Consulting Group has developed a range of services and products designed to help Information Technology departments deliver defect-free software on time and within budget. Interim’s quality management procedures, based upon international standards, are compatible with any software development life cycle and have been proven in the field on hundreds of projects, on four continents, for more than twenty-five years.

Quality Procedures Development and Implementation
Interim has refined its extensive experience into practical, easy-to-implement adaptable strategies and procedures. Interim can help integrate these or similar approaches with organisational philosophies, methods, and goals.

Quality Management Handbook (QMH)
A guide to Interim’s comprehensive Quality System and its applicability to all htmlects of software development and maintenance. Based on international standards, the QMH is designed to be customised for in-house work and that provided by vendors.

Project Management Handbook (PMH)
The PMH contains steps and guidelines for effective management and control of software projects. The PMH is compatible with all recognised life cycle methodologies and project management tools, helping businesses get the most out of their software development process.

VALI/TEST Pro (SM)
Long recognised as the most comprehensive approach to the testing of custom developed or package-based software, VALI/TEST Pro covers all phases of the software life cycle. It is a Windows™-based hypertext tool, which is also available in book form.

Software Testing and Validation
Testing is vital in the creation of defect-free software. Effective testing not only identifies design and coding errors, but validates that the software truly meets business needs. Interim Technology’s validation approach focuses on requirements-based testing through the complete life cycle, including unit, integration, acceptance and post-implementation maintenance testing. VALI/TEST Pro, is the foundation of a range of testing services. This online methodology tool provides the most comprehensive approach to software testing and is the foundation of a range of testing services including: testing process assessment, testing, testing strategy and planning, user acceptance testing, testing process implementation, testing for the year 2000, and test project management.

Quality and Productivity Metrics
Interim’s staff of professionals will deliver a metrics strategy planning to help identify the software metrics most closely aligned to an organisation’s mission and goals. Metrics program implementation assists with the planning and implementation of a Metrics program.

Quality Services for Project Management
Interim helps organisations apply the fundamentals for success contained in our Project Management Handbook (PMH) and Quality Management Handbook (QMH). Doing so helps lay the foundation for project success and forms the basis of a number of Interim’s project management services that include: request for proposal (RFP) development and review, project office, contract management, and independent quality assurance.

Education and Training
Implementing effective quality management requires education and training at all levels, in addition to long-term management commitment. Interim Technology offers a full range of educational programs that can be customised to fit specific needs, as well as for individual mentoring and coaching. All are designed to instill and reinforce the understanding of quality principles. Interim’s programs address both management and implementation issues alike.

User Services
The goal of User Services is to forge a partnership between business and MIS management to ensure dependable systems that meet business requirements. Interim’s user services staff rely on a proven process that integrates user-related elements from several key disciplines to ensure maximum productivity. This speciality can provide support in the following areas: training, custom application training, requirements definition, acceptance testing, user project management, project office, package selection and implementation, and user documentation.

Quality Consulting
A proven project initiation and management process is essential in order to ensure dependable, high quality results with a minimum of inefficiency, risk and cost. Interim can assist by applying the fundamentals for success contained within its Project Management Handbook (PMH) and Quality Management Handbook (QMH).

TEST/CYCLEÒ
Interim Technology, The Consulting Group’s software testing speciality offers a comprehensive set of services and products, all based on the VALI/TEST Pro (SM) approach to testing, that can be tailored to an organisation’s testing objectives.

TEST/CYCLEÒ , an advanced PC- or LAN-based software automates management of the entire testing process. TEST/CYCLE complements and enhances the software development process, whether it includes a traditional life cycle, client/server, GUI, RAD, or other methods. It thoroughly supports all testing levels, including unit, integration, acceptance and regression testing.

==============================================================================

AUTHOR PROFILE

David Powell is a Managing Consultant with Interim Technology Consulting Group in the United Kingdom, and draws on his practical experience of establishing strategic test solutions for international client organisations.

David can be reached on davidpowell@interim.com

A downloadable Word 97 version (3.4Mb) of this paper is available here

==============================================================================