HTML Template for submissions to the Workshop on Evaluation of Virtual Environments.

Three Levels of Usability in Virtual Environments

Andrew Basden

The Centre for Virtual Environments, University of Salford
A.Basden @
salford . ac. uk


Usability is wider than mere ease of use. It must take into account a wide variety of social, inter-personal, cultural factors if it is to adequately reflect the real usefulness of a technology like virtual environments.

This paper outlines a taxonomy and model, leading to a draft methodology, for handling such a wider concept of usability. It is both theoretically and empirically based.


Benefits, Usage, Impact of Virtual Environments, Tasks, Roles, Working Context.

Position Statement

Summary below.

A Wider Concept of Usability

Increasingly it is being recognised that there is more to usability than mere ease of use and good user interface. Many, even of the most usable, information systems are considered failures. Some failures are dramatic, such as 'the day the phones stopped' (Lee, 1992); others, less dramatic, are no less a failure even if they are still being used, such as the French Railways Booking System (Mitev, 1996). Failure, and success, has many aspects, not all of them technical, and in most cases not of extreme type. Often failure takes the form of hindrance to people's ways of working when there should be help, and unexpected harm to others involved with the system.

Failure has many causes. There are many anecdotes of failure resulting from too much emphasis on technical development at the expense of other factors, and Kivijärvi and Zmud (1993) have demonstrated this more rigorously. But the picture is not always so simple. For instance, the London Ambulance System disaster was apparently due to poor technical design, but this itself can be traced back to a hidden agenda by the senior management to attempt to reduce the power of trades unions.

Information system failure is not the exception, it seems to be the rule; Hirscheim (1997) has estimated the failure rate to be between 70% and 90%. Even if an information system is very easy to use, if these other, wider, factors are not taken into account, factors like indirectly caused harm, inter-personal effects, organizational effects, power games, etc., then its overall usability will be in doubt. This leads us to a wider concept of usability.

Is there any way in which this failure rate can be reduced for virtual environments, and wider usability increased? A certain amount of technocentrism is valid in the early days of a new technology, while the technology itself is being developed. In virtual environments, one of the key areas of current technical development is the user interface: multimodal input, ease of use, novel interaction devices, etc. But if this blinds developers to other important factors then frequent failure in virtual environments is likely.

But it is not yet clear what factors contribute to differentiating failure from success, nor how those factors are related. If we are to reduce the incidence of failure in virtual environments, to design for success, to evaluate wider usability, then we need a framework for understanding this wider concept of usability.

Ease of Use - or Usefulness

"I've got this wonderful technology on my desk," my manager once said to me, "It is easy to use. I've been on training courses so I know how to use it. But what do I use it for?" Though emanating from the days in the 1980s before PCs were everywhere, that statement shows there is a clear difference between ease of use (narrow usability) and usefulness (wider usability). Many virtual environments offer high degrees of ease of use, with highly proximal, immersive user interfaces (Basden, Brown, Tetlow and Hibberd, 1996). But are they useful?

If virtual environments is to achieve a higher rate of success than other technologies, then we must find a way of understanding usefulness of the technology, not just ease of use. Such understanding will then aid design, evaluation, and redesign of specific virtual environments, as well as prediction of potential benefits during user requirements analysis.

There are several significant differences between ease of use and usefulness, as shown in Table 1. The main root of the differences is that ease of use tends to focus on the relationship between the virtual environment and the user, treating each as equal partners in some interaction, while usefulness treats the virtual environment as a tool in a multi-human context. The result of such difference in focus is that ease of use research is interested in the dialogue between human and computer while usefulness research is interested in benefit.

If we are to research usefulness, therefore, we must centre on the issue of 'benefit'.

Ease Of Use Usefulness
Concerns relationship between individual user and virtual environment Concerns virtual environment as part of a human working context
Virtual environment as interactive (intelligent?) agent Virtual environment as tool
Can be considered apart from context Context is integral part
Technical Research Topics: Hardware interaction devices, algorithms, structures, low level dialogue, etc. Technical Research Topics: How to match technical features to human tasks.
Human Research Topics: Psychology, esp. cognitive. Human Research Topics: Human functioning in context.
Overall Research Topic: User-Computer Dialogue Overall Research Topic: Benefit

TABLE 1. Differences between Ease of Use and Usefulness


How do we research 'benefit'? The realm of finance offers us 'cost-benefit analysis'. However, this means trying to reduce all benefits to financial terms, and the severe limitations of this approach are well known. We need an approach that copes with wider variety of types of benefit, and also with a variety of different stakeholders. The complexities we face include:

We need a model of benefits, and this means a model of usage of technical artifacts.

Towards a Model of Usage and Benefits

Using a technical artifact cannot be understood purely in technical terms, nor in purely psychological terms (as is assumed by many cognitive scientists who study usability), nor even in purely sociological terms (e.g. of group functioning). Using a technical artifact is essentially a multi-aspectual activity.

Hart (1984) laid philosophic foundations for understanding usage of a technical artifact, proposing there are three distinct levels at which to consider it, and he gave the example of using a phone:

Hart distinguishes between tasks and roles by saying that tasks are something one can put in a diary (compare "3.0 pm. Ring home" with "3.0 pm. Fulfil my role as husband").

The EPSRC-funded KBSIU project (Knowledge Based Systems in Use) independently discovered these three levels in usage of a KBS used widely among members of a professional group. The ELSIE was in use among quantity surveyors in the late 1980s and early 1990s to aid them in enacting a 'lead consultant' role, with the tasks of early budget setting, development appraisal and advice on time and procurement. The project interviewed users in a number of ways, ascertaining their attitudes to the technology, which features they valued in ELSIE, what they used ELSIE for, and how their way of working had changed. One interesting finding was that use of ELSIE split the task of early budget setting into two separate but related tasks, of giving a quick estimate and then meeting the client for detailed refinement which led to a re-examination of their real requirements. This sometimes changed their respective roles from that of expert and novice to two equal partners working towards a shared goal.

There are many anecdotes of such radical changes in both task and role brought about by the impact of information technology, but few controlled studies of this kind. The question is: how can such results be generalised, especially to translate the findings from KBS to virtual environments?

Evaluating the Wider Usability of Virtual Environments

We are currently undertaking an in-depth study of a number of users of virtual environment and virtual reality technologies, employing the model of usage outlined above. We have perused detailed reports of projects in which use or intended use of virtual environments have been described.

First we differentiated five degrees of involvement with the technology:

  1. General awareness of the technology
  2. Specific awareness of how the technology might be relevant to their own situation
  3. Intention to become involved with the technology, with plans made
  4. Commitment of resources to using the technology, including the obtaiing of technology and training in its use, and initial trials
  5. Established use of the technology

and classified respondents into these five categories. However it was sometimes difficult to distinguish future tense from present tense.

With the last degree, and to some extent the fourth, we analysed their text to find benefits either stated or implied. These were then classified as far as possible into the three levels of role benefits, task benefits and feature benefits. Some examples are (with thanks to Tim Watts, Univ. Manchester, who provided some of the case studies):

As can be seen, this model seems to provide a useful framework for analysing the usage of virtual environments technology. It took the researcher a little while (a few weeks) to internalize the differences in meaning between feature, task and role, and to be able to decipher what is sometimes very ambiguous text. But, having done this, it proved a good tool for analysis. Not only could useful distinctions be made, but it also helped to highlight the 'causality' between them, whereby features enable or support tasks and tasks fulfil roles. The research is on-going but we hope it will yield a clearly understood and practical framework for analysing usage retrospectively. Then we hope to employ it prospectively in system design.



Basden A, Brown A J, Tetlow S D A, Hibberd P R, (1996), "Design of a user interface for a knowledge refinement tool", Int. J. Human Computer Studies, v.45, pp.157-183.
Carroll J M, Campbell R L, (1989), "Artifacts as Psychological Theories: The Case of Human-Computer Interaction", Behaviour and Information Technology, v.8, pp.247-256.
Hart H, (1984), Understanding Our World: An Integral Ontology, University Press of America.
Hirscheim figures: exact reference being sought. ====
Kivijarvi H, Zmud R W, (1993), "DSS implementation activities, problem domain characteristics and DSS success", Eur. J. Inf. Systs., v.2, no.3, pp.159-168.
Lee L, (1992), The Day the Phones Stopped, Primus, D.I. Fine, USA.
London Ambulance System critique: reference being sought. ====
Mitev N N, (1996), "More than a failure? The computerized reservation systems at French Railways", Information Technology and People, v.9, No.4, pp.8-19.

Compiled 28 August 1998 by, and copyright (c) of, Andrew Basden. See: Updated 4 February 2002 removal of mailto.