HTML Template for submissions to the Workshop on Evaluation of Virtual Environments.
This paper outlines a taxonomy and model, leading to a draft methodology, for handling such a wider concept of usability. It is both theoretically and empirically based.
Failure has many causes. There are many anecdotes of failure resulting from too much emphasis on technical development at the expense of other factors, and Kivijärvi and Zmud (1993) have demonstrated this more rigorously. But the picture is not always so simple. For instance, the London Ambulance System disaster was apparently due to poor technical design, but this itself can be traced back to a hidden agenda by the senior management to attempt to reduce the power of trades unions.
Information system failure is not the exception, it seems to be the rule; Hirscheim (1997) has estimated the failure rate to be between 70% and 90%. Even if an information system is very easy to use, if these other, wider, factors are not taken into account, factors like indirectly caused harm, inter-personal effects, organizational effects, power games, etc., then its overall usability will be in doubt. This leads us to a wider concept of usability.
Is there any way in which this failure rate can be reduced for virtual environments, and wider usability increased? A certain amount of technocentrism is valid in the early days of a new technology, while the technology itself is being developed. In virtual environments, one of the key areas of current technical development is the user interface: multimodal input, ease of use, novel interaction devices, etc. But if this blinds developers to other important factors then frequent failure in virtual environments is likely.
But it is not yet clear what factors contribute to differentiating failure from success, nor how those factors are related. If we are to reduce the incidence of failure in virtual environments, to design for success, to evaluate wider usability, then we need a framework for understanding this wider concept of usability.
If virtual environments is to achieve a higher rate of success than other technologies, then we must find a way of understanding usefulness of the technology, not just ease of use. Such understanding will then aid design, evaluation, and redesign of specific virtual environments, as well as prediction of potential benefits during user requirements analysis.
There are several significant differences between ease of use and usefulness, as shown in Table 1. The main root of the differences is that ease of use tends to focus on the relationship between the virtual environment and the user, treating each as equal partners in some interaction, while usefulness treats the virtual environment as a tool in a multi-human context. The result of such difference in focus is that ease of use research is interested in the dialogue between human and computer while usefulness research is interested in benefit.
If we are to research usefulness, therefore, we must centre on the issue of 'benefit'.
| Ease Of Use | Usefulness |
| Concerns relationship between individual user and virtual environment | Concerns virtual environment as part of a human working context |
| Virtual environment as interactive (intelligent?) agent | Virtual environment as tool |
| Can be considered apart from context | Context is integral part |
| Technical Research Topics: Hardware interaction devices, algorithms, structures, low level dialogue, etc. | Technical Research Topics: How to match technical features to human tasks. |
| Human Research Topics: Psychology, esp. cognitive. | Human Research Topics: Human functioning in context. |
| Overall Research Topic: User-Computer Dialogue | Overall Research Topic: Benefit |
TABLE 1. Differences between Ease of Use and Usefulness
We need a model of benefits, and this means a model of usage of technical artifacts.
Hart (1984) laid philosophic foundations for understanding usage of a technical artifact, proposing there are three distinct levels at which to consider it, and he gave the example of using a phone:
Hart distinguishes between tasks and roles by saying that tasks are something one can put in a diary (compare "3.0 pm. Ring home" with "3.0 pm. Fulfil my role as husband").
The EPSRC-funded KBSIU project (Knowledge Based Systems in Use) independently discovered these three levels in usage of a KBS used widely among members of a professional group. The ELSIE was in use among quantity surveyors in the late 1980s and early 1990s to aid them in enacting a 'lead consultant' role, with the tasks of early budget setting, development appraisal and advice on time and procurement. The project interviewed users in a number of ways, ascertaining their attitudes to the technology, which features they valued in ELSIE, what they used ELSIE for, and how their way of working had changed. One interesting finding was that use of ELSIE split the task of early budget setting into two separate but related tasks, of giving a quick estimate and then meeting the client for detailed refinement which led to a re-examination of their real requirements. This sometimes changed their respective roles from that of expert and novice to two equal partners working towards a shared goal.
There are many anecdotes of such radical changes in both task and role brought about by the impact of information technology, but few controlled studies of this kind. The question is: how can such results be generalised, especially to translate the findings from KBS to virtual environments?
First we differentiated five degrees of involvement with the technology:
and classified respondents into these five categories. However it was sometimes difficult to distinguish future tense from present tense.
With the last degree, and to some extent the fourth, we analysed their text to find benefits either stated or implied. These were then classified as far as possible into the three levels of role benefits, task benefits and feature benefits. Some examples are (with thanks to Tim Watts, Univ. Manchester, who provided some of the case studies):
[a feature], that it was impossible to get cleaning equipment under the seats [a task benefit]."
[a task], making it brighter and softer, 'installing' a variety of fixtures, before finally settling on a level and style that was acceptable [implied task benefit]."
[task], the Engineering [department] avoided the use of a wooden mock-up, the design going directly to prototype [change in task profile]."
[a feature dis-benefit]
[feature benefit]
[a role statement]
[role benefit] that could result from concurrent engineering with the customer [task]."
As can be seen, this model seems to provide a useful framework for analysing the usage of virtual environments technology. It took the researcher a little while (a few weeks) to internalize the differences in meaning between feature, task and role, and to be able to decipher what is sometimes very ambiguous text. But, having done this, it proved a good tool for analysis. Not only could useful distinctions be made, but it also helped to highlight the 'causality' between them, whereby features enable or support tasks and tasks fulfil roles. The research is on-going but we hope it will yield a clearly understood and practical framework for analysing usage retrospectively. Then we hope to employ it prospectively in system design.