Early Canadiana Online (ECO) is a digital collection (www.canadiana.org) providing more than 1,300,000 pages of Canada's printed heritage and growing by 250,000 pages each year. It features works published from the time of the first European settlers to the early 20th century. MARC records, document metadata and searchable full-text facilitate access to digitized documents. The documents in ECO are facsimiles that convey the look and feel of the original print publications. Educational resources that complement ECO documents include lesson plans and thematic student activities to encourage classroom use. ECO includes French and English texts. The user interface, all information and educational resources on the Web site are in both official languages. Approximately 60% of the collection, partially funded by grants and donations, is open access; the remainder is restricted to subscribers and registered members.
ECO is produced by the Canadian Institute of Historical Microreproduction (CIHM), a not-for-profit organization for preserving and providing access to early Canadian printed works, originally on microfiche and now online. Founded in 1978, CIHM has produced microfiche and catalogue records for nearly 100,000 Canadiana titles published between 1558 and 1920. With assistance from the Andrew W. Mellon Foundation, ECO began in 1996 as a pilot project to convert several thousand titles in specific subject sets from microfiche to digital format. The project had multiple objectives, two of which were "to investigate storage and retrieval from the user's viewpoint" and "to assess user satisfaction with different access options (Turko, 1999)." With the success of the conversion project, Canada in the Making, a collection of Canadian government publications, became phase two of ECO.
Clifford Lynch (2002) reflected last year on the distinction between digital collections and interpretation, teaching or presentation of these materials. Digital collections are the raw materials of cultural heritage; they are source materials with fairly simple access tools that persist generation after generation. Libraries, archives and museums are the major creators of digital collections. Learning and interpretation materials generally have a shorter life span and will include continued reinterpretation. The creators of presentation and interpretation materials from these collections will likely be numerous and diverse. ECO includes both raw materials and interpretative material. CIHM is using multiple methods of user consultation and assessment to build knowledge about the two types of resources.
During the ECO pilot phase, researchers at the University of Toronto Faculty of Information Studies conducted two user surveys, the first in spring/summer 1999 (Duff and Cherry, 2000) when the database was incomplete and the second in spring/summer 2000 (Cherry and Duff, 2002).
The purpose of the first study was to compare use and user satisfaction with early Canadiana material in three formats: original paper, microfiche and online. The study also sought to identify ways to improve the online version. These objectives reflect concerns of the pilot phase at a time of transition. The study employed a user survey, an analysis of server logs and focus group sessions. Surveys were distributed, in French or English, to users of paper, microfiche and online early Canadiana documents. Respondents in the three survey groups used the items differently. Online users were more likely to be using the item for personal interest/hobby or curiosity while microfiche or paper users were more likely to be involved in student assignments or research projects. Respondents to the online survey were more likely to be infrequent users of early Canadiana, less likely to be part of the university community, and more likely to be involved in genealogical research.
The great majority of all users were satisfied overall with the format of the item they used. Forty-six (of 265 total) respondents who had used all three formats answered questions related to format preference. Overall, they liked the online format best and the microfiche least and said that the online format would be most useful in their work. Interestingly, although 41.3% liked the paper format most, only 17.4% said it would be most useful in their work. Almost three-quarters (73.9%) of those who had used all three formats identified the online as the most useful format. It was the most useful because of the ability to search quickly, to make copies and to access material where and when desired. These early and positive user responses gave CIHM confidence that the expectation expressed in the proposal to the Mellon Foundation would be fulfilled. "It may be a little bold to say, but it begins to look as though material on the Web will be much more widely used and that materials accessible only in conventional form will be much less used (Digitization, 1996)."
The same researchers conducted a second user survey, using a Web-based questionnaire, a year later. The objectives were to see if there were any changes in the nature of use or type of user; to investigate satisfaction levels and gather suggestions for further improvements; and to explore a new area--the use of ECO in teaching and research. There were no questions about use of paper and microfiche formats. Overall, findings from the second user survey paralleled those from the first survey. Respondents rated accessibility, searchability and scanned images very highly. Printing remained an area of complaint. Sixty-two (of 159) respondents replied to the questions about teaching and research. There were differences in their use of ECO, probably as a result of the types of tasks they carry out and the culture of teaching and research.
A planned third user study was deferred when operational requirements delayed introduction of a new software release. It was decided that in the absence of additional features, a survey would confirm existing information, rather than provide information about the effect of changes.
Fourteen people, who represented CIHM's traditional audience, participated in four focus groups about midway through the ECO pilot phase. They included five scholars, two graduate students, one undergraduate student, a journalist, two business people and three people who worked in museums or libraries with interests in history, genealogy, Canadian studies, anthropology, English literature and linguistics. Many comments on the benefits of the ECO site corroborated what was found in the survey comments. Focus groups were also an opportunity to explore suggested improvements more fully. The most requested improvements were: the ability to print an entire document, fast response time, the ability to download an entire document. Problems of terminology in the interface were discussed in focus groups.
A Dalhousie/University of Toronto faculty member was commissioned to assess the search and browse functions in ECO from a user perspective and to recommend improvements (Toms, 2001). There were recommendations to improve the flow, terminology, and aesthetics of the search form; to change the display and ordering of search results; to provide options for record display and to modify the page display. Possible additional features in each of these areas were described. She noted that support for browsing demands a different strategy and techniques than searching. Because browsing relies on an ability to recognize the needed information, users need more visual clues. Unlike many full text search systems, ECO has the advantage of full catalogue records which could be used to enrich browse searches.
The observations and recommendations of an experienced observer closely attuned to user behaviour and expectation were very insightful and many of her recommendations will be implemented in the next software release. Current plans are to conduct usability tests during development of the new interface (Toms, 2002). Unlike surveys and focus groups which primarily gather user opinions and perceptions, usability tests capture how people actually use ECO. Simulation studies of what people do rather than what they say they do are an important component for improving the system. The performance (efficiency, effectiveness and satisfaction) of 40 participants from the ECO user community will be assessed using typical tasks. Data will be collected using questionnaires, video screen capture, transactions logs and tape-recorded interviews.
Server logs for 8,226 users sessions during a one month period of the pilot project were analyzed. Results corroborated a few aspects of the survey, but did not provide substantial additional information. Web logs are currently being used to generate aggregate totals and individual totals for document requests, search requests, browse requests, page image requests and multi-page PDF feature, but no further analysis of server logs has been carried out.
A recent study on business models for digital cultural content concluded that "any new digital products that are to be developed should meet actual user needs in a user-friendly format. Moreover, there should be a complementary strategy to attract and grow an audience for the digital product, and raise the "visibility" of the product among potential users (Wall, 2003)." The report noted that whereas few organizations had conducted good analyses of the potential demand for new and expanded digital products among potential users, "CIHM is an exception in this respect. It has an ongoing practice of relying on an advisory committee (with representatives from the scholarly research and library communities) in order to prioritize requests and develop plans for future projects. In this way, CIHM has established a relatively sound basis for ensuring that there is clear demand for the digital products it develops and that, for those that are developed, they are satisfactorily meeting the needs of their target audience. CIHM's approach is likely related to the fact that it relies on a subscription-based model to fund its operations (Wall Communications, 2002)." It may also be related to the governance model of a not-for-profit corporation; CIHM policy is established by an elected board of research librarians and scholars. There have been advisory committees of librarians and scholars for both English and French Canadian literature and there are currently advisory committees for the government document project and the planned periodicals project as well as a technical advisory committee.
Subscriptions are an essential component of CIHM's business model and in the coming year, the staff will work with a market consultant to "grow" the audience both in Canada and internationally. Marketing is an opportunity to gather information from users and potential users and a more diverse audience may influence the shape of ECO in the future. Participation at a recent genealogy fair was an opportunity to communicate with a growing user group.
The "contact us" link on the ECO site facilitates communication from users. It has been an excellent opportunity to document the varied-and sometimes unexpected-uses of ECO and the educational resources. For example, an English editor wrote that he had never before come across PDF facsimiles that are searchable. ECO texts enabled him to solve an enigma about a reference to "frozen sandstone" in a Darwin letter. User consternation about inclusion of Maria Monk, a notorious 19th century text, was an opportunity for education about the roles of ECO.
The first study (Bron, 2002), conducted by an educational consultant, evaluated the usability and usefulness of the material on two educational Web sites Exploration, the Fur Trade and Hudson's Bay Company and Canada in the Making. It examined design, language levels and preferred teaching strategies. Online exhibitions and lesson plans were tested in five urban classrooms (grades 5, 8 and 12). The research tried to simulate conditions in which the sites were meant to be used and activities were chosen by the classroom teacher in consultation with the researcher and were tied to the Ontario Curriculum. The second study a year later was conducted by the same consultant (Bron, 2003). It evaluated the effectiveness and relevance of the revised Canada in the Making resources.
In the first study, all simulations were held in computer labs because there were too few computers in classrooms. In each simulation the researcher and teachers noted Web download time, task completion success, and strengths and weaknesses in the activities and online materials. On completion of the activities, students and teachers completed a survey questionnaire. Student work was reviewed for quality and completion. A few other teachers evaluated the sites using a modified questionnaire. The questionnaire sought assessment of the site and the activities regarding aesthetics, content, interest, readability and technical aspects.
Teachers were impressed with the Exploration, the Fur Trade and Hudson's Bay Company site. Students were also positive, although they rated the site lower than teachers. Their work met or exceeded expectations and teachers rated their engagement as much higher than in normal classroom activities. Results indicated that the Canada in the Making site will be used differently by different levels. Grade 8 teachers considered the digitized primary sources as above the level of the students and most useful as a resource for teachers. Grade 12 teachers were interested in the access to primary source documents which would enable them to engage students in higher level thinking skills such as the interpretation of documents and extrapolation of relevance of historical documents.
The consultant made a number of recommendations regarding the development of educational, or interpretive, resources.
The revised Canada in the Making resources contain narrative texts integrating primary source materials from ECO into explanatory narratives based on themes in Canadian history. The Canadian Constitution is the major theme. The evaluation methods were: post-test after an assignment, survey of students and teachers, and interviews with teachers. The evaluation sites were similar to those in the first study. Results showed improvement in the rating of access to primary sources, design and layout, readability, and navigation. Teachers rated student engagement as high.
When all students attempted to download the same page from the database, many received error messages. It is not certain whether this reflect the capacity of the school computers, the Internet connection of the ECO server. Although the site content has links to all provincial curricula, there was not a strong curriculum connection at the schools and classes where the testing was conducted and thus interest did not increase significantly. Nevertheless, the overall rating rose from 63% in the first study to 72% in the second. To increase the relevance of the site for the grades 11 and 12, the consultant recommended expanding material on Aboriginal history. Colour, images and multimedia were again requested by students and teachers.
Currently, statistics for requests to educational resources are collected but not analyzed.
Judy Luther (2002) has observed that organizations undertake program evaluation for two basic reasons: To Know and To Show. We need to know to manage internally and to show to manage externally. These interrelated requirements motivate ECO's user assessment and evaluation activities.
To Know: The initial user survey of ECO was one of the first studies of a large digital collection and an early opportunity to assess user acceptance of digitized texts of early Canadiana. Similarly, the case studies of the educational resources provided early responses to innovative learning material. From these studies, and the focus groups and interviews that accompanied them, CIHM learned a great deal about what users value. This knowledge informs ongoing product refinement and program planning. Advisory committees and user consultation help CIHM determine the content that has the highest priority among scholars and the features which they expect. For example, scholars emphasize the importance of authenticity and comprehensiveness. It is this knowledge that makes CIHM willing to place 108 interlibrary loan requests to six different libraries for 85 issues of a single government journal. Digitization of a single collection would not create this resource. ECO, like JSTOR, is creating or recreating an authoritative collection that didn't exist before.
The Andrew W. Mellon Foundation has a continuing interesting in helping scholarly organizations understand and shape digital collections. Their support for the pilot phase of ECO included funding for a number of the studies described above. The Foundation provides tangible and intangible support for building knowledge through evaluation and sharing the learning with the community.
Information from user studies has directly affected decisions about design and policy. As a result of recommendations from the first two ECO user surveys the following functional changes have been made:
A recommendation based on experience during the case studies of educational resources influenced the decision to offer ECO without charge to all elementary and secondary schools.
To Show: As with most grant-funded projects, there is an accountability imperative. Understandably and rightly, funders seek evidence that their grants are making a difference for users and not simply keeping an organization busy. Thus, formal user evaluation and reporting are a mandated activity. The expectation of evaluation helps insure that goals and objectives are well conceived and clearly articulated before work begins. CIHM has not conducted evaluation merely for compliance, but has usually been able to build upon the results in its continuing activities. Although subscribers do not generally expect the kind of formal evaluation as granting agencies, they too seek evidence that license fees are money well spent. For subscribers, evaluation results are an important supplement to usage statistics.
CIHM has had very good experience with contracting evaluation and assessment to faculty researchers or consultants. They bring evaluation and assessment knowledge, skills, and experience as well as contacts that could not be maintained within the Institute's small staff. Evaluation is a specialized skill that is appropriately outsourced. Established researchers may increase the credibility of the findings. The cost of two user surveys was $14,000 (Canadian) including prize draw and translation. They demonstrated that Web-based surveys are an efficient and effective way to gather data on users and their uses of digital collections. The cost of two classroom case studies was about $9,000 (Canadian). Heritage Canada requested permission to use them as a template for other evaluation projects. The expenditure for consultants has been substantially less than 1% of the project cost.
In spite of the demonstrated benefits and feasibility of user input and evaluation, there are factors that mitigate against allocating greater resources to these activities. They include paucity of management time, the "future" bias of public organizations, and the political orientation of national programs.
In a small organization with unstable funding, every dollar is carefully budgeted. Nevertheless, the financial resources for modest evaluation activities have been less difficult to achieve than the managerial time to oversee studies and utilize results. Even when consultants conduct the evaluation, it is necessary to prepare specifications, negotiate agreements, analyze and interpret reports. Judging when and how to incorporate information from assessments into decision-making is another responsibility. Synchronizing the time-frames of evaluation protocols with operational requirements has been a continuing challenge for CIHM. A user-centered approach requires constant leadership and management attention. These are always scarce resources, and particularly so at a small organization in transition.
A researcher at the Kennedy School of Government contrasts public and private organizations less on the basis of a bottom line orientation than on the basis of focusing on past performance or future possibilities. Profitability is, after all, a measure of what happened in the past. Public enterprises, however, tend to value policy analysis about the future more than program evaluation about the past. He counsels that "the lesson from the private sector seems to be that it is extremely valuable to develop accurate information about performance in the past rather than concentrate all one's efforts on guessing about the future (Moore, 1995) ." The atmosphere of impatience that pervades many government programs and some libraries is hard to counter and mitigates against building good information about use and users. Only the bravest administrator can embrace Abbey Smith's concept of a "successful failure:" to document what went right and wrong and report the results so that they can be leveraged by others rather than repeated (2003).
A successful user-centered approach to digital libraries will depend upon data-based evaluation. There are, of course, other ways to assess programs and services. A standard text on utilization-focused evaluation identifies a philanthropic orientation and a constituency-based political orientation as alternative methods (Patton). Another text on program evaluation cautions that programs are usually paid for by parties other than the client of the program. A program's well-being may be more closely dependent on keeping program funders happy than on keeping users satisfied (Posavac and Carey, 1997). The board and management of CIHM cannot ignore any form of assessment; they all demand attention. In recent years, the critical decisions regarding the future of the organization have not depended on the use of and satisfaction with ECO by users. Survival decisions have been made primarily on other grounds and outside the organization. Responsible administrators of digital library programs are gathering data on use and user, celebrating success and seeking to learn from failures. At the same time, they must recognize limits of data-driven decision-making and attend to other types of assessment, education and advocacy.
In addition to the individuals cited in the footnotes who have been involved with ECO, I would like to acknowledge the contributions of Pam Bjornson, former CIHM Executive Director; Magdalene Albert, current CIHM Executive Director; and William Wueppelmann, CIHM Electronic Systems Specialist..
Bron, I.G. (2003) Canada in the making phase II: evaluation report."
Bron, I.G. (2002) "Assessment and evaluation of CIHM educational resources. " [Prepared for] The Canadian Institute of Historical Microreproductions.
Cherry, J.M. and Duff, W.M. (2002) "Studying digital library users over time: a follow-up survey of Early Canadiana online". Information Research 7(2), http://informationr.net/ir/7-2/paper123.html (30 May 2003)
"Digitization and Access " (1996) Proposal to the Andrew W. Mellon Foundation [from] Canadian Institute for Historical Microreproductions (CIHM), National Library of Canada, Université Laval, University of Toronto. p. 2. Quoting Malcom Getz.Duff, W.M and Cherry, J.M. (2000) "Use of historical documents in a digital world: comparisons with original materials and microfiche." Information Research 6(1), http://informationr.net/ir/6-1/paper86.html(30 May 2003)
Luther, J. (2002) "Getting statistics we an use." In: Proceedings of the 4th Northumbria international conference on performance measurement in libraries and information services "meaningful measures for emerging realities, pp. 321-330. Washington, D.C.: Association for Research Libraries. p. 326.
Lynch, C. (2002) "Digital collections, digital libraries and the digitization of cultural heritage information." First Monday 7(5) http://firstmonday.org/issues/issue7_5/index.html. (30 May 2003)
Moore, M.H. (1995) Creating public value; strategic management in government. Cambridge, MA: Harvard University Press. p. 35.
Patton, M.Q. (1997) Utilization-focused evaluation: the new century text. 3d ed. Thousand Oaks, CA.: Sage. p. 25.
Posavac, E.J. and Carey, R.G. (1997) Program evaluation: methods and case studies. 5th ed. Upper Saddle River, N.J.: Prentice-Hall. p. 3.
http://www.ianbron.com/portfolio/execsum.pdf and http://www.ianbron.com/portfolio/report.pdfSmith, A. (2003) "Issues in sustainability: creating value for online users. " First Monday 8(5) http://firstmonday.org/issues/issue8_5/smith/index.html (30 May 2003) p. 2.
Toms, E.G. (2001) "Early Canadiana online recommendations for searching and browsing; a report to the Canadian Institute for Historical Microreproduction."
Toms, E.G. (2002) "Proposed ECO Assessment."
Turko, K. (1999) "Final report of the Early Canadiana online/Notre memoire en ligne project to the Andrew W. Mellon Foundation."
Wall, G. (2003) "Business model issues in the development of digital cultural content." First Monday, 8(5), http://firstmonday.org/issues/issue8_5/wall/index.html. (30 May 2003) p. 5.
Wall Communications, Inc. (2002) "A Study of business models sustaining the development of digital cultural content." Prepared for Department of Canadian Heritage. p. 32.