Evaluating digital services: a visitors and residents approach
This guide contains advice on evaluating the services you offer to your users. The focus is primarily on digital/online services but set within the broader context of more traditional services, exploring the relationship between the two.
Much of the information presented here is derived from a longitudinal research project, which explored how individuals at various educational stages (emerging, establishing, embedding, and experiencing) engage with the digital environment for learning.
Last year high school/secondary school and first year undergraduate college/university students
Upper division undergraduate college/university students
The ‘emerging stage’ bridges the transition between school and university. It was designed to do this as it was felt that in terms of digital engagement or digital literacies the ‘gap’ between school and university is not as stark as the education sector might imagine.
We outline broad findings from the research and describe the methods used to gain a deeper understanding of what motivates users to engage with specific aspects of the information environment in a given context.
The contents of this guide are not presented as research findings in the traditional sense, but represent themes that have emerged over the course of the research which can usefully inform our understanding of how learners are navigating the overlapping worlds of the web and formal education.
In particular, this guide is designed to facilitate critical thinking about web-based tools and services, and more effective digital engagement on the terms that our users have come to expect.
What do we mean by visitors and residents?
Well, it’s perhaps best described as a continuum, with two modes of online engagement at either end.
When in visitor mode, individuals have a defined goal or task and select an appropriate online tool to meet their needs. There is very little in terms of social visibility or trace when online in visitor mode.
When in resident mode the individual is going online to connect to, or to be with, other people. This mode is about social presence. We’ll discuss visitors and residents in more detail throughout the guide.
Paying attention to which services people adopt can lead to more effective engagement between individuals and institutions, and the resources/services therein. It’s not just about identifying how individuals engage with technology and how they acquire their information but also why they make the choices that they do.
We need to understand the larger, more complex contexts that surround individual engagement with digital resources, spaces, and tools before we can think constructively about how to (if, indeed in some cases, we should) provide institutionally-based digital resources and services. This is also important when considering how institutional systems interface, or overlap, with the environment and culture of the wider web.
Evaluation tools such as surveys and compiled statistics are ubiquitous. For example, within a library setting, traditional quantitative data that report outputs (the number of books circulated, the number of reference questions answered or the attendance at library) and inputs (space, budget, collections, equipment and staff) create a narrow picture of ‘performance’.
Such reporting is useful for describing things that are easily measured, but some important information is not measurable as direct statistics and requires a different form of investigation and assessment.
Insights into the ‘how’ and ‘why’ of user engagement with technology and resources often can be most effectively gained by employing qualitative research methods.
We want to help practitioners generate their own qualitative inquiries, to inform their own policy needs and to frame the organisation (corporate, department, faculty) and its services within the institutional agendas surrounding student experience and engagement. For a detailed review of the qualitative/quantitative methods used in the research informs this guide please see: Evaluating digital services – research methods.
We have also developed a practice focused mapping tool which uses the visitors and residents continuum to explore online engagement.
This guide is a collaborative project between Jisc, the University of Oxford, and OCLC Research, and in partnership with the University of North Carolina, Charlotte.
For more than a decade Marc Prensky’s (2001) notions of digital natives and digital immigrants has had a powerful influence on how educational institutions perceive students and technology:
"Prensky’s distinction between people who are entirely at ease within a digital environment and those who manage to learn to exist but who (in his view) will never be fully competent, has gained enormous currency and, until recently, widespread acceptance. Similarly, his linked assertion that the differentiation also signals the need for an educational revolution, requiring a new approach which accommodates the up-and-coming Natives, has not only been largely believed but has provoked a sense of panic among ‘Immigrant educators’ who now perceive themselves wrong–footed and unable to step up to the plate." White and Le Cornu, 2011
In recent years educational researchers have come to treat the natives and immigrants idea with suspicion. Nevertheless it has become embedded in many areas, and still forms the basis for much strategic thinking and implicitly underpins the decision making process in universities where the digital is concerned.
This guide draws on the findings and methods of the Jisc/OCLC funded digital visitors and residents (V&R) project which is underpinned by an alternative to Prensky’s typing of technology users. Visitors and residents is a simple way of describing a wide range, or continuum of, modes of online engagement. It has proved to be a useful way to come to an understanding of individuals’ motivations when they use the web in differing contexts.
We are not proposing that one mode of engagement is better than the other, simply that different modes are employed depending on the individual’s motivation and context at the time.
When in Visitor mode, individuals decide on the task they wish to undertake. For example, discovering a particular piece of information online, completing the task and then going offline or moving on to another task.
"We propose that visitors understand the web as akin to an untidy garden tool shed. They have defined a goal or task and go into the shed to select an appropriate tool which they use to attain their goal. Task over, the tool is returned to the shed. It may not have been perfect for the task, but they are happy to make do so long as some progress is made." White and Le Cornu (2011)
In visitor mode individuals do not leave any social trace online. Much online activity is undertaken in this mode as illustrated by our research participants:
"Well, I’ll Google and see what comes up, and if I need to know just the basis and just to get my head around it I’ll use Wikipedia." UK, emerging, male, age 18
When in resident mode the individual is going online to connect to, or to be with, other people. This mode is about social presence.
"Residents, on the other hand, see the Web as a place, perhaps like a park or a building in which there are clusters of friends and colleagues whom they can approach and with whom they can share information about their life and work. A proportion of their lives is actually lived out online where the distinction between online and offline is increasingly blurred." White and Le Cornu (2011)
Resident behaviour has a certain degree of social visibility: for example, posting to the wall in Facebook, tweeting, blogging, or posting comments on blogs. This type of online behaviour leaves a persistent social trace which could be within a closed group such as a cohort of students in a Virtual Learning Environment (VLE)/Learning Management System (LMS) or on the open web.
In information-seeking, Resident behaviour is more relevant in cases where individuals are going online to seek out other people for information. This might be by asking a direct question online or by asking for advice on trusted sources.
"The Facebook group is extremely active also, if not just for complaining about an assignment or trying to find a particular reading, but also sharing current news articles with each other." UK, embedding, female, age 22
In this case the provenance of information shared is partly related to the person who shared it, and also in how trusted they are in assessing the validity of a source. They are vouching for the course and are bringing a social dimension to the information-seeking process. The other significant factor in Resident behaviour is the production of non-traditional sources such as blog posts, which are in turn used by learners.
For an example of how the continuum can be employed to assess user engagement see the visitors and residents mapping section of this guide.
The Visitors and Residents (V&R) project explored what motivates different types of engagement with the digital environment for learning. The investigation focused on the sources learners turn to in order to gather information, and which ‘spaces’ (on and offline) they choose to interact in as part of the learning process. It used the V&R framework to map learner’s modes of engagement in both personal and institutional contexts.
The project assessed whether individual approaches shift according to the learners’ educational stage or whether they develop practices/literacies in early stages that remain largely unchanged as they progress through their educational career. Learners from both the UK and the US participated in the project, enabling the researchers to explore potential cultural differences between the two countries.
The pilot phase focused on the emerging educational stage which spans last year high school/secondary school and first year undergraduate college/university students. Phase 2 included interviews with participants in the three later educational stages – establishing (upper division undergraduate college/university students); embedding (graduates); and experienced (faculty).
A detailed code book has been compiled, enabling a first draft analysis of the data. Emerging findings are encouraging and indicate (amongst other things):
A variety of ways in which learners and scholars engage with the digital information environment, some of which correspond to the concepts of visitors and residents
Certain sources which are widely used by students against the advice of teachers and lecturers (eg Wikipedia); therefore, ‘covertly,’ used suggesting the presence of a type of ‘learning black market’
Potential differences in approaches between US and UK learners and scholars
Discovering and defining a number of ‘learner owned’ digital literacies, which are not highly visible to educational institutions
No significant shift in modes of engagement when transitioning from school to university
It is anticipated that the project will evolve into a three-year longitudinal study which will investigate the four educational-stages from school level to scholar. The planned phase 3 will centre on a large targeted survey which is designed to test findings developed from interview data in the previous phases
Our website has more information about this project and its outputs. This guide explores some of the key themes emerging from the V&R project to date.
Stakeholder snapshots – visitor mode
This section provides a snapshot of different stakeholders in relation to a visitor mode.
Visitor modes tend to be well-represented in academic libraries generally, as a tool-based approach is traditional for databases, monographs, and instruction, among other things. Patrons go to the resource, use it, and put it back. Libraries put a great deal of institutional energy into configuring Visitor-mode tools that patrons use to search for resources paid for by the library. This is very much in keeping with the needs of traditional academia which tends to be geared around the ‘lone scholar’ principle.
Even when students have personal learning practices that are resident in character (such as participating in Facebook groups for their courses), Visitor modes remain the predominant way that institutions see their students operating.
For example, students submitting work to the virtual learning environment (VLE)/Learning Management System (LMS), engaging with other systems to complete assignments, or viewing instructional videos or other materials. While there is often Resident-mode functionality embedded in VLE/LMS platforms, it is likely that students will prefer to keep their resident practices in online spaces not owned by the institution.
Data from the Visitors and Residents (V&R) project indicates that students see email as the main method of communication within a formal institutional setting. Individuals who have to interact in official ways with institutions tend to have email as a regular part of their digital practices, regardless of age. Such institutions do not have to be schools, but can also be churches, scouting organisations, and so on.
Ultimately, most educational institutions predominantly engage students in Visitor modes, especially with regard to formal assessment, which is usually focused on individual ability rather than collaboration skills. Students who do well in terms of marks are usually adept with Visitor mode practices, such as seeking out and evaluating credible information online. In fact, much of the intellectual effort expended in Visitor mode for students is assessing to what extent non-traditional online sources can be cited or spoken about in formal educational contexts.
Many such successful students might also communicate about their course of study with peers in Resident online spaces, but may not categorise this as ‘learning’ when asked.
For those who are teaching, visitor mode often means gathering and curating information and instructional materials. VLEs/LMSs such as Moodle or Blackboard are treated as document repositories and portals to operational websites such as the library. In visitor mode, there is little expectation of being engaged in an online discourse around these resources. The resources provided in this way are clearly extremely useful and highly valued by students without involving any personal online presence on their part.
Visitor mode is the most traditional way of searching for resources. Online databases, library catalogues are tools and information sources that do not require online presence, or interaction with other people. Expert information seekers can actually spend more time looking for the right resources than novices, because they have the knowledge that makes them aware of how much further they need to go to find the ideal resource.
Speed is not an effective measure of efficient or effective searches in the case of expert researchers.
"Yes, for many journals it’s directly to – I don’t go through [academic library name] as much anymore. Because I have the journals I look at bookmarked. And many of them are through certain publishers like American Society for Microbiology, ASM, they publish a dozen journals. So I just go to ASM site. And there’s a couple of other journals that have a – they’re published by a group that publishes multiple journals.
So if I look for a specific article I still use [academic library name], usually Science Citation Index, one of their electronic resources, medline. I mean SCI are some others. CSA and some others. But usually it’s a couple of different search engines that are used for looking for an article or just what’s new on this topic." USA, experiencing, male, age 54
Use of email is an important tool for communication that leaves no visible trace on the open web. It is very common for both faculty and graduate student researchers to use email eg Visitor modes of collaborative research and/or writing frequently involve emailing Word documents back and forth, with track changes turned on.
This section provides a snapshot of different stakeholders in relation to a resident mode.
Resident modes are increasingly emerging in library spaces, but as there is less history of such modes it remains a challenge to think of how/whether to facilitate Resident engagement on an institutional level. It used to be that the library was one of the only sources of information so users built their workflows around the library; resources were scarce and users’ attention was abundant. Now that available resources are abundant and users’ attention is scarce, libraries need to build their services around the users’ workflows (Dempsey, 2008).
Across the educational stages, it is clear that individuals do contact other people, not just to conduct searches with Visitor-mode tools such as search engines, when they need help or specific information. Libraries would do well, therefore to engage in resident strategies to initiate and develop relationships with their patrons. Institutional Twitter and Facebook accounts can make the library a visible presence, even for students who do not frequent the physical building.
Engaging with users and potential users of library systems and services requires presence and availability in digital spaces in which they dwell allowing institutions to become an involved and interactive presence within the social media environment.
In early educational stages the focus is on the Visitor mode practice of seeking convenient and/or credible sources of information. Residency in a learning context tends to be in personal online spaces eg asking peers for advice on sources in Facebook.
As students move into later stages they might begin to see the value of developing an online presence around a course of study, eg starting or running a course focused Facebook group or beginning to engage in subject related discussions online rather than simply ‘lurking’ (becoming vocal in Twitter for example).
Some students in later educational stages might see the value of undertaking their professional practice in open or visible online spaces. This is especially pertinent for certain disciplines where the production of work and its critique by a given audience is integral; for example in arts, music, or literature.
It is also relevant to any discipline at the point where individuals feel it is important for their point of view to become part of the discourse around a given subject. In this way Resident practices can be an important part of students developing their ‘voice’ within their chosen field.
Teaching staff may choose to engage students online in a more resident manner by facilitating discussions in an institutional Virtual Learning Environment (VLE)/Learning Management System (LMS). They might also post course materials in a platform which allows for discussion and comment.
More directly, tutors might require their students to undertake assignments in visible online spaces such a blogs or to create short YouTube videos as part of assessed work. In this way both the teaching and the learning process become Resident in nature and students are challenged to develop their thinking and express their thoughts as part of an open discourse eg Wikipedia’s education programme.
An increasing number of researchers (especially those early in their careers) consider Resident practices to be one way to increase the impact of their work. This is usually by developing a professional persona online and building a network through platforms such as Twitter or expressing expert opinion via a blog.
Traditional forms of disseminating research such as journal papers can be given greater impact when discussed in blog posts or highlighted in Tweets. Researchers may also be highly Resident in subject related communities online.
The visitors and residents (V and R) continuum illustrates the range of possible modes of engagement individuals now have available to them via the web. It can be a valuable tool for exploring how users are engaging with the services you provide. One pragmatic application of the continuum is the V and R mapping process.
This is an activity which can be used to gain a picture of individuals’ overall engagement landscape. It can also be used to gain a picture of the overall ‘digital presence’ of a group or department by bringing together and overlaying multiple maps from staff.
What does the mapping process show?
The mapping process indicates not only what services individuals are using, but more importantly how and why they are using them. This becomes particularly significant when there is engagement with technology in a Resident mode as this will be an experiential as well as a functional form of engagement. The way in which the technology is ‘used’ at the Resident end of the continuum cannot be predicted by describing its face-value functionality.
The V and R process works well in a face to face focus group interview format as the discussion that arises from the mapping often garners the most valuable information about individuals’ attitudes and expectations towards a set of services.
The process itself is relatively simple and has been used successfully with academics, senior managers and students from a variety of disciplines and academic levels.
This video outlines the process in a form that is predominantly designed to be used in a learning and teaching context with fairly experienced learners. You may want to adjust the mapping activity, based on the characteristics of your focus group.
Following feedback from mapping sessions at EDUCAUSE, American Library Association (ALA), and the Higher Education Academy challenges of residency project we have discovered a number of elements of the activity which can be adjusted to make it more effective in certain contexts (Higher Education Academy, 2013).
The vertical axis
The exact labelling of the vertical axis can be changed to make it more relevant for specific groups. For example, early stage students have very little concept of the ‘institution’ and are happier to use terms such as ‘my course’ or ‘university’. We have also found in some cases that a few members of the academic staff cannot distinguish between the personal and the institutional, which indicates the vocational nature of many educational posts.
One possible approach is to discuss with the group the most appropriate terms for the vertical axis as the term they decide on could be a useful indication of how they perceive the institution/faculty/department.
The ‘personal’ and the ‘engagement’ maps
In the full video it’s suggested that participants create a personal map and then create a map which indicates how they engage with the digital services for their learning. In theory, given the inclusion of the vertical axis, both of these maps should be very similar. We have found that early stage (emerging and establishing) students don’t make a ‘personal’ and ‘engagement’ distinction; therefore, a single map with a holistic approach works well.
You can also create a map for a particular department/service eg for the library. Department/service staff should start by creating personal maps (and then discuss what they feel is important/relevant about them). Then the group can create a department/service V and R map eg a map depicting how the library’s services engage users to meet their technology and information needs. This will provide a model of the department/service’s digital identity and may reveal areas where relatively simple interventions can have an impact on engagement.
Discussing your own map
The mapping process might feel slightly intrusive for some. Most people never have to reveal how they engage with technology and other sources in such a comprehensive manner. A good way to break the ice and to explain the process is to draw your own map as part of the session. This tends to start the discussion rolling and provides an environment where participants feel comfortable asking for clarification.
Example visitor and resident maps
Below are a number of maps we have collected in various sessions. They are included here to highlight some of the themes we have regularly encountered. There is, of course, no standard way to create a map so don’t worry if the maps you receive from users look different than these.
Firstly, for clarity, the following are digital copies of the maps David draws in the video.
Note that David uses Facebook only as a kind of address book; therefore, it is in the visitor end of the continuum.
This is a map created by a senior manager at a session delivered at EDUCAUSE 2012. The highlights indicate how this individual has used two services with equivalent functionality with different modes of engagement.
This is a good example of why it’s not possible to predict the mode of use from the apparent functionality of the tool ie some tools are appropriated very differently by different users.
This is a map from a UK-based mapping session, created by a university educational consultant. You can see that this individual is most resident in well established technologies rather than social media.
We would argue that this is not highly resident in the sense that it’s not openly visible in the manner of a blog comment or a Facebook wall post. It is, however, where this individual is most ‘present’ within technology.
This map was created at the American Library Association (ALA) 2013 annual conference by a library professional. This individual is resident in the library catalogue, presumably spending significant amounts of time guiding users in their searches. They also have chosen to map the physical library in the centre of the map which would indicate that it is integral to all aspects of his/her life.
This is not a typical mapping position for the physical library as most individuals position it at the bottom left. Blogging also is central to this map. This mapping, with the positioning of the physical library and the central position of blogging, indicates that this individual’s personal and professional lives are intertwined.
This map was created as part of the Higher Education Academy’s 'challenges of residency' project by a communication and media student in the UK. Things to note here are the position of the physical library and the two instances of YouTube, one when logged in and one when simply browsing.
This map was created as part of the Higher Education Academy’s challenges of residency project by a master’s level biomedical science student. This map has relatively little in the top left and bottom right quadrants, which is fairly common, and might be an effect of the individual’s academic level and discipline.
We also see a more sophisticated mapping of the information services made available to the individual by the university/library.
People trust people
The desire that individuals have to know that they are using the ‘right’ kind of information or resources leads them to ask for help within their personal networks. The composition of those networks shifts over time and the visitors and residents (V&R) interview data reflects that last year of high school/secondary school students and first year college/university students consult parents, siblings and friends about academic work.
The upper division undergraduate college/university students indicate they consult their room-mates, classmates, and siblings who have taken similar classes before and the graduate students consult graduate school peers and professors (but reach out far less than any other educational stage). Faculty most often consult their peers.
Emerging (Last year of high school/secondary school)
Establishing (First year college/university)
Embedding (graduate students)
The point is that individuals seek what they need within the relationships that surround them. As they move through the educational stages, their networks are increasingly populated with people who have relevant subject expertise, so that by the time individuals are faculty members, when they say they called a ‘friend’ about an article, it is nearly certain that the friend is also an expert in the field.
Relationships continue to be a major component in how individuals get their information and whom they choose for collaboration and one of the reasons for engaging with technology (Connaway and Radford, 2011).
The diagram indicates the mix of sources participants of the V&R project mentioned when discussing their information seeking approaches in both study/professional and personal contexts.
This tree diagram shows the relative frequency with which key themes were mentioned by participants in the V&R study across all four of the Educational Stages. The larger and more green the square, the greater the number of coded occurrences for that theme. Notice the importance of friends and colleagues, peers, and even parents before experts, professionals, or librarians. We have excluded ‘online search engine’ from this diagram as this is not in itself a source. Google search and other search engines were frequently mentioned and as expected were the starting point for the majority of information seeking online.
In the V&R interviews, the tendency to collaborate is most evident in the participants in their last year of school/first year of a degree, it decreases sharply for upper division undergraduates, and then increases again for faculty. This may reflect not only the need for collaboration in their work among secondary school students and first year undergraduates, but also the emphasis on individual/isolating work for upper division undergraduate students as well as graduate students. The high level of faculty collaboration is an interesting contrast to the sort of training the graduate students apparently are receiving in their respective fields.
Thought should be given to identifying ways to help graduate students connect with their peers in other institutions since these individuals may be their professional colleagues and collaborators as they gain specialised knowledge in their field of study. Social media, and Resident forms of online practice, can be a tool for decreasing the isolation of the graduate experience, and better preparing them to be senior scholars and knowledgeable professionals in their respective disciplines
The importance of embedding services and resources in trusted relationships with our users cannot be overestimated. We recommend that institutions consider digital and face to face community building as a cornerstone of their enterprise-wide policies. Social media tools can be used to build such relationships, but not without careful thought. Many individuals carefully compartmentalise their social media presence into ‘personal’ and ‘professional/institutional/academic’, and institutions risk alienating the very people with whom they wish to connect if they overstep boundaries.
It’s also important to account for the fact that you represent the authority of the institution and are likely to be regarded with suspicion in online social spaces. This can be problematic if you are represented by an institutional profile rather than a personal one.
If you do decide to represent your service using a personal Social Media account then you need to consider the possibility that the personal and professional aspects of that individual’s life might become blurred online. This is especially likely if they are already Resident in a particular platform before they take on the role of representing the library via their profile. Concerns about the collapse of institutional and personal lives into one messy category were sounded repeatedly at the Visitors and Residents expert session during the 2013 American Library Association (ALA) annual meetings in Chicago.
The tension between the need to be visible, and to make connections, with the desire to keep private lives private, and to have safe places where individuals can be more than just their professional self, is a concern for professionals and users alike, as the following quotes illustrate.
"I’ve kept [Facebook and Twitter] separate from the beginning. I was an early adopter of both but very clear [about keeping them separate]… I realise that I’m not a really private person, but in terms of my online identity, I’ve become really careful. I used to use Twitter, lots of IM, SecondLife, travel sites…I’ve chosen not to use any of those anymore because I don’t need the world to know exactly where I am and who I’m with." Library professional – V&R workshop ALA conference 2013, Chicago
Individuals who are senior in their field are far more likely to have personal networks that overlap heavily with their professional networks. Undergraduates and junior scholars tend to have much more discrete circles and far less collapse in the personal-institutional continuum of their Visitors and Residents map.
Professionals and students alike have different personae that they adopt in the contexts (online and offline) that they inhabit. Providing for people to be comfortable with the personae they interact with and can inhabit in the institutional contexts in which they find themselves is crucial groundwork for the construction of effective relationships.
"I work with grad and high school students and there seems to be a division of what they use for institutional use. We’ve seen this migration to Instagram and Vine. There’s been a disconnect because at some point…what they think they post online that they thought was not visible to the universe is. My kids are absolutely leaving Facebook and have charted the [migration] from MySpace to Facebook, but it’s interesting that more adults are becoming resident (many nod). They’ll have more than one profile or a decoy cell phone." Library professional – V&R workshop ALA conference 2013, Chicago
It is very difficult, if not impossible for institutions to predict at any given point what tool, persona, or agenda individuals will have when engaging with institutional staff. Institutions would benefit from providing a variety of possible ways for individuals to connect, and by acknowledging that a range of modes of engagement is expected by individuals, not only in their personal lives but also in their institutional/professional/academic lives.
Institutions should therefore commit to a diverse presence in both digital and physical spaces.
Convenient doesn't always mean simple
Too often, institutions characterise individuals’ reluctance to use their resources as a result of people wanting what is ‘convenient’. The problem with throwing up one’s hands over the desire for convenience, and the fact that people turn to convenient resources rather than quality or official or library resources, is that ‘convenience’ means different things to different people and can change radically depending on context.
The analysis of visitor and resident modes of engagement can be a way of parsing just what is meant by ‘convenient’, and is a way for institutions to have a better grasp of how to position their services and resources effectively.
A significant portion of our work to date has focused on libraries and so we’ll use that as an example. V and R data agree with other research projects that indicate that some individuals are frustrated with libraries and do not physically visit them because of limited hours, long travel distances, and the time needed to do their research in the library (Connaway 2013b; Connaway, Lanclos, and Hood, 2013a; Connaway, Lanclos, White, Le Cornu, and Hood, 2013; Pullinger, 1999; White and Connaway, 2011-2012).
Speed and efficiency
Search engines (in particular, Google) are preferred because they are more convenient and faster – they are always available (De Rosa et al, 2005), and also because they are perceived to be reliable, as well as fast.
"I find Google a lot easier than going to the library website because I don’t know, it’s just like sometimes so many journals come up and when you look at the first ten and they just don’t make any sense I, kind of, give up." USA, emerging female age 19
"I will say that I’ll probably go to Google before I will search [the library database system] primarily because it’s easy and I don’t have to log in." UK, embedding female age 22
Speed and efficiency are relatively conventional ways of defining ‘convenience’, and they are certainly factors in the decisions people make about where to go and what to use, when seeking information. Data from the V and R project reveals that individuals in all educational stages cite the relevance of convenience/ease of use to their decision-making, trumping all other reasons for selecting and using a source (White and Connaway, 2011-2012; Connaway, White, and Lanclos, 2011).
This tree diagram shows the relative frequency of key themes that were mentioned by participants in the V and R study across all four of the educational stages. The larger and greener the square the greater the number of coded occurrences for that theme. What is of note here is how convenience is higher in the mind of our participants than themes such as relevance and reliability.
Aspects of convenience include choosing the source, familiarity with the source, perceived ease of use, time available, and physical proximity. If one is working in Visitor mode, primarily with operational tools online, ‘convenience’ is not interpersonal. In Visitor mode even if it is likely to result in a lower quality information than asking an expert, friend or family member, it is seen as less onerous or socially risky to not have to interact with a person.
"It’s convenience. It’s the immediacy of it. And also the fact that Google doesn’t judge you." UK, experiencing male, age 52
If you’re in resident mode, operating widely and socially within digital spaces, ‘convenience’ might mean not just online on the ‘free’ web, but also embedded in trusted personal networks, valuing contact with people as a part of the process. See also people trust people.
"I use Facebook for organising my life basically, with friends and stuff. So, it’s extremely helpful, and then, umm, like I also use that in education to talk to my friends about an equation, the things I don’t understand and it works quite well." UK, emerging male, age 18
It was important to emerging students that the digital sources they found were authoritative, specifically that they were identified by trustworthy human sources: people who, in their personal experience, had proven that they were knowledgeable, either generally, or specifically about the topic at hand (Connaway, White, Lanclos, and Le Cornu, 2012). They often were unaware that many of these sources actually came from the library (Connaway, Lanclos, and Hood, 2013b), characterising them as simply ‘online’ resources.
It’s worth bearing in mind that those trustworthy human sources were not always experts in the field, but could simply be family members who had taken a class before, or gone to college ever, or who were just known to be ‘on top of things’. Getting a source quickly was not enough, the source had to also be perceived as a quality one, or time would have been wasted.
Familiarity appeared to also be a major component of individuals’ perceptions both of authority and convenience. If an individual had done something/used something before, they were more likely to perceive that thing as being ‘convenient’, even if it objectively took more steps and/or time than other solutions.
While the methods some individuals used to seek and assess information were inelegant, they had been developed at an early age, and had become well worn and comfortable, if not terribly efficient or sophisticated. They continued using these methods in part because they tended to yield effective results.
"What I’ll do is I’ll just Google it. And it’s like ‘Oh, I need this. Hold on.’ And I was like ‘Specifically it involves these three words – bam, bam, bam.’ And then whatever comes up. And if I’m not satisfied with that I’ll try another set of words until I get it." USA, embedding female, age 30
"I Google things occasionally when I’m stuck. When like I’m sitting in the canteen and doing work and stuff, if I get stuck on something, I don’t have the right textbook, I might Google it because it’s easy to get out of my pocket which is actually really useful." UK, emerging female, age 17
These students also weighed the cost of their time against the importance of the assignment on their potential grade, in declaring a course of action ‘convenient’. That is, it was not just about the time and effort, but also about some sort of academic return on investment. Such a perspective was also reported in several other studies (Connaway, Radford, Dickey, Williams, and Confer, 2008; Connaway, 2013b; Connaway and Radford, 2011; Connaway, 2008; Connaway and Radford, 2007; Connaway, Lanclos, and Hood, 2013a; Connaway, Lanclos, White, Le Cornu and Hood, 2013).
Students are acutely aware of what sort of academic product they are working towards, and how they look for and evaluate sources for their five page essay will be different from a 20 page senior honours thesis (see also 'academia isn’t always learning'). Such differences in approach, such flexibility on the part of individuals in deciding what is convenient or easy to use within the wide array of higher education micro contexts is of great concern at a time when pedagogical conversations are much concerned not just with what students use in their academic work, but how they use sources in constructing and supporting arguments (cf. for example B. Fister’s blog post on library agendas and 2013 Library Orientation Exchange (LOEX) talk about information literacy).
So while convenience does appear as the most significant factor in choosing information sources, what is convenient is still dictated by the context and situation at the time of the information need and can be either physical or digital (Connaway, 2013a; Connaway, 2013b; Connaway, Lanclos, White, Le Cornu, and Hood, 2013; Connaway, White, Lanclos, and Le Cornu, 2012). The everyday practices of the web also mean that people do not have to learn complicated systems to find satisfying information.
Authoritative, valuable information
People of all academic stages had difficulty finding answers to the question, “Can you think of a time when you struggled to find information you needed in a personal (non-academic) situation?”, because it was genuinely hard for them to remember a time when they could not easily find what they needed. They were far more likely to have examples to share of struggles in searching for what they needed in academic contexts. That is, it was relatively easy for people to find useful information but more challenging to find this information in an authoritative form that would be acceptable in the context of formal education, ie a source they would be prepared to cite.
The point is that convenience is not necessarily the same as the most ‘simple’ or ‘quick’ solution but is about the perceived value of the effort required to find relevant and authoritative sources in a given context. Convenience is also closely linked to the preferred mode-of-engagement (visitor or resident) at a particular moment in time. Someone operating in visitor mode may have a very personal motivation, one linked just to the need they have for a source to use in an assignment, and for that source to be an acceptable one given the guidelines for effective academic work.
The motives that arise when operating in resident mode, however, can be about the potential that a given piece of information has to travel through their digital social network. Someone operating within resident mode might be thinking far less about the content (or quality, or authority) of what they are passing on than in the potential visibility that content might yield within their digital social network, whether it is Twitter, Facebook, or LinkedIn.
In the context of an institutional service convenience does not necessarily mean making everything look like Google. It does mean making the value of the resources you offer clear, and make the logic behind using more complex resources (like databases) evident. People have to trust that the more complex resources will actually yield a return on their investment (of time, effort, etc), such that it was worth learning how to use a completely new interface and resource. This is not to say that the clarity and ease with which people can use your resources is a trivial matter. For services to be perceived as valuable, they need to be easy to access, and not require training to use (Connaway, Dickey, and Radford 2011).
"It’s number one. I actually learnt that like it’s placed number one because it has the most links to other websites. It’s more credible." USA, emerging female, age 19
"So if I’m looking for something and I know there’s somebody at work who can answer that a lot better, instead of contacting them in whatever way – you know, calling them or emailing them or whatever – just thrashing a few things into a Google search. I do tend to rely on that." UK, experiencing male, age 52
Academia isn't (always) learning
Clearly the web provides access to a vast range of information resources ranging from asking a friend on Facebook or getting information from academic journals. Many non-traditional sources, such as Wikipedia or YouTube videos, can be used effectively for learning while not being highly credible in a formal academic sense. This presents learners with a challenge as they often have to decide on a balance between ease of use and academic credibility.
"Okay it’s just a text, it’s fine. I think it’s because maybe people of older generations are used to using books more. And so like my parents will always go, ‘Well look it up in a book, go to the library.’ And I’ll go, ‘Well there’s the internet just there.’ " UK, emerging female age 19
Those participants in the Emerging educational stage tended to consider physical books as the most academically credible source but often used what they considered less authoritative information from the web. This usually was because searching online was the most convenient method of discovering what they needed and usually provided more focused answers than alternatives.
"Well pretty much every research paper and things I’ve done have said ‘Use internet sources if you desire, but at least a certain number of hard copy sources’. And, again, I generally would use the hard copy source just to find a speck of information and then go back to the internet because it’s just … it’s so easy. Whereas getting a book this thick and trying to find a quotable sentence that really relates to the context could take two hours." USA, emerging male, age 18
Participants in the later educational stages (embedding and experienced) had more sophisticated critical evaluation techniques and a deeper understanding of their chosen subject so were prepared to vouch for less traditional sources. However, they still were tentative about citing non-traditional resources in formal work even when they knew the information to be high quality.
All of the students we interviewed had constructed a precise, if conservative, model of what they thought their institution would accept as legitimate practices in their academic work and were adept at only presenting these during class time and in formal assessments. For tutors, faculty, and other academic staff, the notion of what constitutes legitimate practice was far less clear and appeared to be negotiated on an individual basis. See the learning black market.
"She was very direct about certain stuff and wanted me to go to the library. Of course like the library’s like a second home for me, so that’s fine. But the research I needed wasn’t showing up. So I was like okay, I’m going to the internet. And I had to find quotes from books, so I just like was able to go on Google, Google book search, and find the quote I needed. And I didn’t write down it was from the internet." USA, emerging female, age 19
This tension between using sources which are ‘efficient’ for learning and those which are highly credible in an academic context is useful to consider from users’ perspectives when assessing their engagement with your services. In the context of the visitors and residents continuum, it is interesting to consider how, broadly speaking, Visitor mode practices are generally considered by the emerging stage participants to be more legitimate in a formal academic context than resident forms of practice (White and Connaway, 2011-2012).
The learning black market
Many tutors, faculty, and other members of academic staff are sceptical about the validity of non-traditional online resources, especially those where provenance is difficult or impossible to establish. It would appear that instead of teaching critical evaluation skills and helping learners to situate non-traditional resources within larger information-seeking frameworks, tutors commonly advise that platforms such as Wikipedia should be totally avoided, particularly true within the United States (US).
"For that one, no, but for the other two projects they were – they both said we had to have three sources, only one of them could be from a website, and it had to be a reliable website, which is the code word for ‘You can’t use Wikipedia’." USA, emerging female, age 17
This creates a tension between academia and learninG as our participants rarely, if ever, find that a source such as Wikipedia provides them with false or poor quality information.
Participant: Yes just because generally anybody can write anything on Wikipedia so you don’t know if it’s true or not true. You could change – I mean we could go on there and change it to something just to do it. So they just say it’s an unreliable source.
Interviewer: In the stuff that you’ve looked up on Wikipedia have you ever been sort of caught out in thinking something was true that wasn’t?
Participant: I actually haven’t. Not yet.
USA, embedding female, 23
"I just don’t – I really don’t understand why Wikipedia is so taboo because – I mean, I do understand that anyone can add information on there but then again anyone can make a website, anyone can make a journal, it doesn’t make it like an educational source." USA, emerging female, age 19
The convenience of these types of resources means that their use is almost inevitable, especially by students who are encountering a new topic area or are refreshing their understanding. This kind of use is mirrored in the behaviour of faculty members, who also use Wikipedia and Google to browse, and get themselves up to speed on unfamiliar topics. But this use is hidden by learners from their instructors, because they are well aware that non-traditional resources, of which Wikipedia is the exemplar, are not viewed as legitimate by their educational institution.
Not only will learners be wary of citing sources of this type, they also will not discuss their use in formal contexts, such as the lecture theatre or class room.
This furtive thinking and behaviour around open-web resources such as Wikipedia masks the level of use of non-traditional resources and also masks the methods learners use to increase their understanding of subjects, creating what we have called the learning black market. The point at which learning takes place is often not being discussed because either explicitly or implicitly learners are being told by their educational intuitions or perceive that the educational institutions view that their information-seeking practices are not legitimate.
In many cases this has led our participants to cite ‘acceptable’ sources which they haven’t properly engaged with. Resources that are cited but not read become an academic façade for the learning that often has taken place by using Wikipedia, YouTube and blog posts.
"…he wants me to try and go find another source or go look up a book about it that was written by somebody that was approved, you know is credible. But a lot of times people will – they’ll look up their source online… – figure out the information they need and then they’ll just quote another source or they’ll cite another source." USA, emerging male, age 19
A common example of this practice, as discussed by our participants, was to cite the references from a Wikipedia article but not the article itself.
"Also you know I have put Wikipedia as a source, but at the bottom there is like a ton of sources. You can use any one of those sites or books that they use for their actual article. So I can just go through – like go on Wikipedia and look up like – and just take five other sources and put it in my paper. It is like ‘Yes I used these sources’." USA, establishing male, age 19
Of course this style of citing unread, unreviewed sources pre-dates the web. However, we would argue that the prevalence of non-traditional sources, which provide focused ‘answers’ has amplified this practice and that the ‘banning’ approach of some academic staff is simply widening the gap between the students’ actual learning practices and the academic requirements.
"Yes, I do find I quite often go against what the teachers ask me to do, because I have my preferences. So if they ask me to look at a website, or to look at a book, I tend to – for example, if they give me a website to look at and I know that I’ve already found one that I like and I know my way around, I’ll use that one instead." UK, emerging female, age 17
The concept of the learning black market is significant for libraries and those populating the institutional virtual learning environment/learning management system as many students, especially those in the earlier educational stages (last year of high school/secondary school and first-year college/university undergraduate students), may be engaging with institutional services to simply find legitimate, citable sources rather than to deepen their understanding of a subject.
In fact, it was not uncommon for our participants to characterise the library as only the location of highly legitimate physical sources, most notably, books. Often this was because they had been required to cite a set number of non-web based resources by their tutor as a method of ensuring that at least some traditional sources were used.
The prevalence of the learning black market tends to fade as learners reach the later educational stages (third/fourth year college/university undergraduate and graduate students) because their increasing expertise in their chosen discipline provides them with the subject knowledge and expertise to determine valid information.
"I usually check. I try to test the information, if it’s – my judgment, or also my knowledge in the subject or – or I will read more to see if the information is right or not. I don’t trust it, like from the first second." USA, embedding female, age 45
The faculty we interviewed for the visitors and residents project certainly used it when they felt a need. We would argue that the use of Wikipedia is more uniform than this research would indicate because later-stage learners are simply more confident in admitting that they use it.
The learning black market also is evident when students collaborate or share sources online using services such as Facebook. The fear of being accused of plagiarism or collusion means that learners often are wary of discussing their learning practices in Social Media. This is especially important to consider if you are planning to engage your users via these kinds of platforms. It is also important to account for the fact that you represent the authority of the institution and are likely to be regarded with suspicion in online social spaces.
The 'Crowdsourcing: the wiki way of working' infoKit explains an approach to organising work that is in some ways the opposite of traditional planning and management, instead it focuses on the wisdom of the crowd.
Think less - find more
Our participants’ expectations of using technology for information seeking tended to be influenced more by the web than by the services provided by libraries. The phrase, 'think less – find more,' is a satirical comment on the concept that Google is ‘selling’ with its approach to search. Web-based search engines have moved from traditional Boolean functionality to natural language queries and Google is now attempting to answer users’ questions before they are asked by analysing data from a plethora of sources to model life-patterns.
This is less the case in a learning or research context, however, there is an indication that some learners implicitly view the role of technology as the means by which they receive a ‘correct’ answer. This was evidenced in the responses to a ‘magic-wand’ question which participants often responded to by describing the ‘perfect’ search engine. This is an imagined single search box which only ever returns the ideal answer:
"Perfect thing, I think it would be that all the useful, accurate, reliable information would like glow a different colour or something so I could tell without wasting my time going through all of them, thinking ‘Oh no, that’s not useful, that’s not useful,’ actually like all the information that I need and could actually use would be somehow highlighted…" UK, emerging female, age 17
What is traditionally thought of as ‘research’ is actually seen by some as an inconvenient time-consuming task that has to be done because the technology doesn’t work well enough yet. Barbara Fister discussed this at length in her decoding academy talk at LOEX 2013, noting the results of the ongoing work of Howard and Jamieson’s Citation Project, which reveal the student tendency to ‘patchwrite’, to string quotes – generally found in online resources – together rather than come up with an argument, or analysis (Fister, 2013; The Citation Project).
"No need to search out and read any sources at all. Perhaps more dispiritingly, the video shows how a student with dreams and an urge to create something meaningful is finally able to do that – once he has completed that tiresome paper." Fister, 2013
This phenomenon has implications for both services and pedagogical process and highlights one of the fundamental principles at stake for education in the context of the web:
"The problem with Wikipedia is it’s too easy. You can go to Wikipedia, you can get an answer, you don’t actually learn anything, you just get an answer." Tutor – via USA, emerging male, age 28
The implication for many services is that if what you provide differs in its functionality from web-based paradigms it is essential that the benefit of your service’s approach is clearly articulated. Perceived value is now often contextualised relative to the web.
Assessing non traditional sources
"I’ve used them but besides that when I’m going through random websites and I’m trying to figure out which one can actually be used, it’s, kind of, hard because there really isn’t anything, there isn’t like a stamp that, you know, someone puts like, ‘This one’s real.’" USA, emerging female, age 19
Students in the emerging educational stage are operating in the largest information environment. That is to say that emerging stage students are most likely to simply Google and use the first few links returned as their main source of information.
As students transition through the educational stages, their study-related information environment becomes smaller in tandem with specialising in a given discipline. They become familiar with the authoritative, subject-specific online sources and information and increasingly use formal sources, such as peer-reviewed journals.
Most of our participants in the earlier (emerging and establishing) educational stages were preoccupied with the validity of the information they discovered, as were those in the later (embedding and experiencing) educational stages if they were tackling a new subject area.
What many students describe themselves doing is not the critical evaluation of the content of the sources they need to use. Rather, they are describing the evaluation of the provenance of the sources, looking in particular for signifiers of credibility. They are not, particularly in Emerging educational stages, capable of evaluating the content critically, because they do not know enough about any given academic field.
Information literacy instruction commonly replicates this process of evaluation via provenance by recommending that students go straight to library databases. The assumption is that these databases are populated with quality, credible sources, and therefore students do not have to exercise a great deal of critical thinking once they acquire those sources.
Faculty members and graduate students are far better equipped to evaluate sources not located in officially-sanctioned locations such as library databases and university press publications. Their expertise allows them to critique the content, and no longer rely on the ‘surface’ qualities of sources as a proxy for credibility.
The question becomes how do instructors give student opportunities to practice the evaluation of content in the absence of expert content knowledge. Our research indicates that individuals exercise a great deal of critical evaluation of sources in non-academic contexts (as in the case of looking for good restaurants, or shopping for new cars, for instance). It may be of value to explore these personal evaluation practices with students, encouraging them to apply them in academic contexts.
This, over time, helps to shift the focus from finding ‘credible’ sources to critically evaluating the substance of the sources itself and helps to counter the ‘think less – find more’ problem. Another way to potentially develop students’ thinking in this area is the integration of more resident forms of online practice into the curriculum. Using the web to convene relatively open discussions about the credibility of certain forms of knowledge both highlights the issues and encourages students to become part of a healthy discourse about the value of the subject.
Such web-based practices can also facilitate students developing their own learner or professional personas online.
Many of our participants regularly used a combination of the following methods to assess the validity of non-traditional online resources.
1. Comparing the ‘top three’ links returned by Google: If they say roughly the same thing, then the information is often treated as valid.
"Well I don’t like pick the first one I see. I try to evaluate two or three and see if there’s some common things between them. Like if two of them say the same thing then that must be right." USA, emerging male, age 17
"I’ll try and check a couple of different sites to see if they are all the same and that’s also how you check credibility; if they are all saying the same thing you know one sites not messing with you." USA, emerging male, age 19
2. Checking the provenance of the website via the URL. Trusted provenance includes established institutions such as universities, established charities or ‘good quality’ media organisations such as the BBC. Interestingly in the US students are often advised not to trust .com sites as they 'will be trying to sell you something.'
So, usually I’ll do .gov, .org, .net. A lot of people, I know, will just put something on the internet and it won’t necessarily be true; like Wikipedia. I’ll try and limit the searches to ones I know will be real, and not just someone’s opinion. USA, emereging female, age 17
"So I usually just stick to journals. Sometimes I’ll, like I said, just Google it for some general information. Maybe like current events and things like that but unless it’s .org or .edu or something like that is another way that we kind of signal that those are okay. But usually, in political science as an undergrad, I was told usually don’t use .com because it’s kind of like can be for an organisation or a certain group of people like bias or steered in one way." USA, embedding female, age 23
"…just looking at the websites themselves, I mean he said use anything that ends at edu or gov or org, stuff like that. You know, try to avoid the .com’s because they’re usually just people who make websites. I mean, it is not like their answers are wrong but they’re just not acceptable I guess in a college essay." USA, emerging female, age 19
3. The ‘look’ of the site: If it has a clean design (looks contemporary) and is not coated in adverts, then the information is usually seen as more valid.
"Usually websites that have like one colour backgrounds, where it’s a lot of just like basic text, there’s not a lot of graphics. They don’t really look nice, they look, kind of, like somebody just threw a website together and put it up online. It’s like I don’t trust those because it’s like, you know, they didn’t put a whole lot of time into this, I wonder how credible they are, so I don’t trust them usually." USA, emerging male, age 19