Sources of information You can gather data from people, documents, performance data, observation of events or any other empirical method such as experiments and benchmarking. Basic feedback gathering methods The ideal form of feedback gathering is to use a combination of the following methods depending on time and organisational factors. Observations, walkthroughs and site visits These are conducted to get first-hand information on performance and usability features of the system. The internal or external evaluators will observe all stakeholders using the technology and observe usability and performance indicators of the system. Walkthroughs are conducted where an evaluator walks through a certain feature to assess how the system performs that feature with respect to usability and performance indicators. Interviews Here are some tips for conducting interviews: Choose stakeholders who would have greater or unique involvement with the new system. Communicate the purpose of the interview to the interviewee. Ask brief questions relevant to performance and usability of the system. Don’t interrupt. Be a good listener. Take notes. Focus groups These are group interview situations where discussions can take place about the usability and the performance of the hardware and software. Here are some tips for conducting focus groups: Reward the attendees by providing refreshments as this could be a good motivator. Start and finish on time. Be prepared to hear positive and negative comments. Be prepared with prompting questions to start the discussions about usability and performance of the system. Let the participants communicate and listen carefully. Avoid being defensive. Listen to all comments. Engage a note-taker. Surveys and questionnaires These are used to gather quantifiable data about the system from a large number of people. You should make allowances for the low response rate and the slow response time. Here are some tips for conducting surveys: State the objective of the survey as evaluating the performance and usability of the new technology. Keep the survey to a manageable length. Use both open-ended and closed questions.
Analysing and processing feedback All data regardless of how and from where it was collected must be summarised against the performance and usability indicators so they are more manageable. In almost all case, both quantitative and qualitative data will be collected and used. The performance measurements such as error rates will be quantitative and will be easy to interpret. Performance measurements such as reliability could be a combination of qualitative and quantitative data. Usability indicators could be qualitative or quantitative. Indicators Poor Satisfactory Good Excellent Support Help desk services Support mechanisms are or not adequate. Support mechanisms exist. But fees associated with help desk calls are high and response times are slow. Support mechanisms exist. Fees for help desk calls are reasonable but response times are slow. Cheat sheets and how-to guides are available for some features. Excellent support mechanisms. Reasonable fees and acceptable response times. Relevant cheat sheets and how-to guides are accessible through the web-based support system. Technical needs assessment Needs assessment conducted for implementation Needs assessment was not conducted. Some groups of stakeholders were surveyed to identify computer hardware requirements. Needs for all stakeholders identified however not all needs are accounted for in the final implementation. An elaborate and comprehensive needs assessment was conducted and it all stakeholders were well informed about the improvements that would be introduced. Indicators Poor Satisfactory Good Excellent Training Training prior to and during implementation Training was not provided. Training was provided in large groups. More customise training for small groups. A comprehensive training plan was derived and all users were trained in groups for common skills and on a one-to-one basis for user specific tasks. Customisation User customisation All features are set and customisation is not an option Desktop attributes such as fonts and colours could be customised to suit the user’s needs Power users can customise certain features. Ordinary uses cannot change any features Power users can customise most features where as ordinary users can change a limited number of features.
Integration Compatibility The new software is not compatible with any old packages that performed similar tasks. New software is partially compatible with old technologies however the administrator has stopped making any reference to data in the previous system. New software is backward compatible with the old software but does need some intervention with data conversions. New software is totally backward compatible with the old software. Indicators Poor Satisfactory Good Excellent Performance Value for Money The technology has not produced any cost advantages that was anticipated. In fact, the new technology costs more money to the organisation. The new technology has not enforced any additional expenses (running costs) compared to the old technologies used previously. The technology has minimized costs The technology is proving to be producing a profit. Speed (Throughput) Output of new technology is slower compared to the previous technologies. New Technology is comparable with old other technologies. New Technology is lot more efficient than all previous technologies. New Technology is producing more than three times faster than the previous system. Quality The quality of output is not acceptable. Error rate is more than 5%. The quality of output is comparable to older technologies and is acceptable. Less than 5% defects. Produces good quality output. Less than 2% defects. The quality of output is rated as very high. Less than 0.5% defects. Summary We began with a discussion of the need for and process of evaluation when implementing new technology. Then we moved on to planning an evaluation, key indicators of usability and performance, and environmental considerations when purchasing new equipment. Then we explored using feedback, its value and how to gather, analyse and process it. We finished with an example of a scoring system to identify how the new system is performing with regards to usability and performance.