Evaluation of the New Jersey Digital Highway

The aim of this research is to study the usefulness of the New Jersey Digital Highway (NJDH, www.njdigitalhighway.org) and its portal structure. The NJDH intends to provide an immersive and user-centered portal for New Jersey history and culture. The research recruited 145 participants and used a Web-based questionnaire that contained three sections: for everyone, for educators, and for curators. The feedback on the usefulness of the NJDH was positive and the portal structure was favorable. The research uncovered several reasons why some collections did not want to or could not participate. The findings also suggested priorities for further development. This study is one of the few on the evaluation of cultural heritage digital library.


Judy Jeng
The aim of this research is to study the usefulness of the New Jersey Digital Highway (NJDH, www.njdigitalhigh way.org) and its portal structure.The NJDH intends to provide an immersive and user-centered portal for New Jersey history and culture.The research recruited 145 participants and used a Web-based questionnaire that contained three sections: for everyone, for educators, and for curators.The feedback on the usefulness of the NJDH was positive and the portal structure was favorable.The research uncovered several reasons why some collections did not want to or could not participate.The findings also suggested priorities for further development.This study is one of the few on the evaluation of cultural heritage digital library.
T he New Jersey Digital Highway (NJDH, www .njdigitalhighway.org) is a digital library for New Jersey history and culture, including collections of New Jersey libraries, museums, archives, and his torical societies.The NJDH, funded in part by the 2003 National Leadership Grant of the Institute for Museum and Library Services, is a joint project by New Jersey State Library, the New Jersey Division of Archives and Records Management at Rutgers University Libraries, the New Jersey Historical Society, and the American Labor Museum.As part of the project, the NJDH identifies 686 cultural heritage institutions (public libraries, archives, historical societies, and museums).As of November 2007, there are more than ten thousand objects (pictures, records, and oral histories) in the repository.More are being added daily.The NJDH, at this writing, is still very much a work in process.The principal investigator of this project continues to extend opportunities to more com munities to link their sites and scan their images. 1he NJDH provides portals for four different groups of people: everyone, educators, students, and librarians and curators.Its mission is to develop an immersive, user-centered information portal and to support the New Jersey learner through a collaboration among cul tural heritage institutions that supports preservation of the past, new access strategies for the future, and active engagement with resources at the local and the global level for shared access and local ownership.
The NJDH uses FEDORA (Flexible Extensible Digital Object Repository Architecture) as a platform to mount participating institutions' digital objects and metadata.FEDORA is developed jointly by Cornell University and the University of Virginia and is currently supported through an Andrew W. Mellon Foundation grant that is customizable and allows local institutions to have true control over what they digitize and post. 2 FEDORA is built on XML with core standards that support flexibility and interoperability such as METS (Metadata Encoding and Transmition Standard, www.loc.gov/standards/mets) and OAI-PMH (Open Archives Initiative Protocol for Metadata Harvesting, www.openarchives.org)func tions.FEDORA is chosen for the NJDH because it can effectively accommodate and manage a broad array of information sources with the flexibility to integrate with other information repositories.
The NJDH uses a metadata structure based on MODS (Metadata Open Description Schema, www.loc.gov/standards/mods), METS, NISO, and PREMIS (Preservation Metadata, www.loc.gov/standards/premis)metadata standards to support preservation of digital objects, to ensure scalability for projects and interoperability with other systems through OAI-PMH.This hybrid approach enables NJDH collection managers and metadata creators to provide information through multiple presentation standards in a schema easily understood within distinc tive cultural heritage organization communities.MODS is used for descriptive metadata, provides and retains stan dard bibliographic cataloging principles, and is therefore easily mapped to MARC.The NJDH therefore includes a mapping utility that allows the export of records from the NJDH to online catalogs for any organization that wants to make its digital objects accessible within its integrated library system.Additionally, there are four other types of metadata in NJDH: source metadata describes provenance, condition, and conservation of analog source materials such as photographs, books, maps, audio, and video; technical metadata describes born digital images and pro vides information about the digital master files that will be maintained for long-term preservation and access; rights metadata identifies the rights holder(s) for each informa tion source, identifies the permissions for use including any restrictions, and documents the copyright status of each work; digital provenance metadata provides a digital "audit trail" of any changes to the metadata. 3he use of the NJDH has steadily grown and has some three thousand unique visitors a month averaging eight to ten thousand visits per month. 4ior cultural heritage digital library n evaluations Literature review indicates that few researchers have investigated the usability or the evaluation of cultural heritage digital libraries.
The MINERVA (Ministerial Network for Valorising Activities) project proposed a number of criteria and principles specifically for usability evaluations of cul tural Web applications, including visibility, affordance, natural mapping, constraints, conceptual models, feed back, safety, flexibility, the scope and aim of the site, meaningful organization of the website's functions, quality of content (for example, consistency, complete ness, conciseness, accuracy, objectivity), design of func tional layout, consistent use of graphics and multimedia components, as well as provision for navigation tools and search mechanisms. 5In addition, Vaki, Dallas, and Dalla proposed sixteen usability guidelines for cultural applications. 6aroufallou, Siatri, and Balatsoukas reported their research on the user interface of the VeriaGrid applica tion. 7The VeriaGrid system (www.theveriagrid.org) is a platform based on digital cartography that supports a vector map of the city of Veria organized by layers and linked to multimedia objects such as text, images, photos, and video clips.The researchers were interested in learnability, errors, and satisfaction.

Usefulness as the primary n evaluation criterion for the NJDH
The NJDH aims to serve heterogeneous communities and information needs.Like other digital cultural ser vices, it is not easy to address usability issues.Lynch has said that digital libraries of cultural heritage don't really have natural communities around them and that digital materials find their own unexpected user communities. 8aroufallou, Siatri, and Balatsoukas said that "different types of users, such as students and scholars or tourists and travelers look at these services from different angles (for example, scholarly or recreational needs).Thus, the provision of accessible and user-friendly systems is important for the wider use and acceptance of these services." 9he aim of this evaluation was to assess usefulness of the NJDH from the perspectives of general users, educa tors, and cultural heritage professionals.Usefulness is one of the criteria of usability with a focus on "Did it really help me?" and "Was it worth the effort?"Usefulness dif fers from usableness in that usableness refers to functions such as "Can I turn it on?"or "Can I invoke that func tion?" Usefulness can also mean "serving an intended purpose."In the Technology Acceptance Model (TAM) developed by Davis and his colleagues, perceived useful ness refers to the extent to which an information system will enhance a user 's performance. 10n addition to usefulness and usableness, Jeng has gathered a comprehensive collection of usability criteria such as effectiveness, efficiency, satisfaction, learnability, ease of use, memorability, mistake recovery, and interface effectiveness. 11Usability is a multidimensional construct and has a theoretical root in human-computer interaction.
Although usefulness may be an important evalua tion criterion, Thomas and Jeng report that usefulness is an often overlooked criterion of usability. 12Literature review indicates that usefulness has been used as either the primary or one of the criteria in the following evaluations of digital libraries: eLibraryHub, the Digital Work Environment, GROW (Geotechnical, Rock, and Water Engineering, www.grow.arizona.edu),McMaster University Library's Gateway, the Miguel de Cervantes Virtual Library, Minnesota's Foundations Project, and the Moving Image Collections. 13This paper reports the evaluation of the NJDH.
n Research method A Web-based online survey was conducted in September-December 2006.The questionnaire was designed, col lected, and analyzed using Web-based software called SurveyMonkey.Convenience sampling method was used in this study.Subjects were recruited by posting a link on the NJDH website, by posting announcements on a number of electronic discussion lists for educators and cultural heritage professionals, and by word-of-mouth invitations.
The participants were asked to complete a two-part questionnaire.The first part gathered demographic data such as gender, age, ethnic background, educational background, the county they live in, and how they learned about the NJDH.The second part contained three sections: one for everyone, one for educators, and one for cultural heritage professionals.
The section for everyone contained twenty-six ques tions, including seven-point Likert scales and open-ended questions with a focus on the digital library's usefulness, navigation, design, terminology, and user lostness.In addition to this general section, educators were asked to complete another fifteen questions pertaining specifically to the educators' portal; the cultural heritage profession als had another thirteen questions regarding the librar ians and curators' portal.
A total of 145 individuals participated in the survey, of which 32 were educators (22%) and 28 (20%) were cultural heritage professionals.The participants were mostly white (127 respondents or 89%), mostly female (118 respondents or 81%), and most had a master 's or doctoral degree (114 respondents or 79%).In terms of age distribution, more than half of the participants were over 50 (79 respondents or 55%) (see table 1).Nearly all (136 respondents or 94%) were residents of New Jersey.
As to the survey participants who identified themselves as cultural heri tage professionals, 61% (17 respondents) worked at libraries, 11% (3 respondents) worked at museums, 11% (3 respondents) worked at historical societies, and 4% (1 respondent) worked with archives.In terms of their roles at those organizations, 61% (17 respondents) said they were fac ulty or staff, 18% (5 respondents) were administrators, one was a consultant, one n Findings

How do users find out about the nJDH and will they come back?
The survey found that more than half of the respon dents (58 participants or 40%) learned about the NJDH from their colleagues or friends, 19 participants (13%) learned through attending conferences, 16 participants (11%) were linked from other websites (see figure 1).
The NJDH digital library intends to build rich and "one stop shop" digital collections of New Jersey history and culture.Cultural heritage digital library plays a par ticularly important role for students of the humanities because the digital library is the humanist's laboratory, its resources are the scholar 's primary data. 14It is important to enhance users' awareness of this digital library among New Jerseyans and even promote this cultural heritage digital library to users at global level.
In terms of the purposes of visiting the NJDH, the study found 72 respondents (76%) were just browsing and 23 respondents (24%) were looking for specific infor mation such as a specific county information, history, and family genealogy (see figure 2).
Seventy-two respondents (74%) replied that they will come back to use the NJDH again (see figure 3).Those who said "No" gave reasons such as their doubts on whether the information in the NJDH is reliable and authoritative, the depth and breadth of content in this digital library, and the inconsistency of fonts and font sizes.
n Navigation Navigation has been reported in literature as a common problem in a digital library.Users could accidentally leave the digital library, following the links to other Web-based resources, and were unaware that they were no longer using the digital library.Brinck, Gergle, and Wood report that disorientation is among the biggest frustrations for Web users. 15aluation oF tHE nEw JErsEY DiGital HiGHwaY | JEnG 19 No answer 3  11   This survey found the overall response regarding the navigation of the NJDH was very positive (response average 2.54 on a 7-point Likert scale, 1 being easy to navigate and 7 being difficult to navigate).Twentythree participants (25%) marked 1 on the Likert scale, 28 participants (30%) marked 2, and 26 participants (28%) marked 3.These brought the total of the top three points to 83%.
The overall response regarding user lostness was also not a problem (response average 2.42 on a 7-point Likert scale, 1 being not lost at all and 7 being very lost).Only two participants expressed they were very lost and one expressed lost.The reasons that could lead to user lostness include the lack of mate rial in the collections so far, the need for explanation of how relevance is ranked, the home page being text heavy and cluttered, the photos not being legible, the lack of author information in documents, no indica tion of a trail of how one got there, lengthy URLs, the need for better chosen direct links instead of layered links, and patrons' unfamiliarity with icons and their functions.

n Layout
The rating for the layout of the NJDH was very posi tive (response average 2.54 on a 7-point Likert scale, 1 being good and 7 being bad).However, the site may improve its appearance in the following areas: there is currently too much text per page (the font is too small and the use of typography, informational hierarchy, and white space must be improved); more important information needs to go at the top of pages; and more colors need to be used.

n Terminology
The degree to which users interact with a digital library depends on how well users understand the terminology displayed on the system interface.Literature review has indicated that the inappropri ate use of jargon has been a common problem in digital library design.Hartson, Shivakumar, and Pérez-Quinones report from their usability inspec tion of the Networked Computer Science Technical Reference Library (www.ncstrol.org)that problems with wording accounted for 36% of the digital library's usability problems. 16System designers often assume too much about the extent of user knowledge.The precise use of words in a user interface is one of the utmost important design consid erations for usability.n Usefulness Usefulness was the funda mental research focus of this study.This research investigated whether the NJDH was useful to the general public, educators, and students.The responses were overwhelmingly positive: 73% of the respondents gave 1-3 ratings on the 7-point Likert scale (1 being useful and 7 being not useful)-30% (29 respondents) marked 1, 33% (32 respondents) marked 2, and 12% (12 respondents) marked 3. The average response was 2.63.This was a very positive response.
When it comes to the specific section for educators to evaluate the educator 's portal, the rating was also posi tive (response average 3.04).Those educators felt that the most useful information was the "how to" information for teaching with digital resources, research genealogy, developing an oral history, and so on.Twelve respon dents (44%) indicated they would encourage their stu dents to use the NJDH site for term papers or homework assignments.Thirteen respondents (50%) indicated they would make their own lesson plans using the resources and information from the NJDH.
Regarding the student's portal, those educators who responded to the survey indicated that, from their per spectives, the most useful information for students was the general information about New Jersey, including a direc tory of cultural heritage organizations, places to visit, etc.
As for the librarians and curators' portal, those cul tural heritage professionals identified the Librarians and Curators' Resource Center as the most useful resource in the NJDH, followed by the Digital Highway Collections Roadmap and associated guidelines, calendar, the searching capabilities of New Jersey Cultural Heritage Organizations, and New Jersey information.Sixteen respondents (67%) said they would recommend this digi tal library to their patrons, two respondents (8%) won't, and six respondents (25%) were not sure.It is obvious that the NJDH administrators need to work harder in this area to enhance usefulness for cultural heritage profes sionals and their patrons.
The survey asked all respondents to suggest what themes should be enriched in the NJDH collections.The suggestions were, in this order: New Jersey history, New Jersey state and county documents, New Jersey culture, genealogy, everyday life in New Jersey, New Jersey indus try, more immigration resources, education in New Jersey, New Jersey in wartime, and transportation.
Regarding the librarians and curators' portal, the respondents suggested the contents of this particular por tal should be enhanced in the following priority order: (1) more links to other websites with history resources and activities, (2) access to mentors experienced in digitizing and metadata who can provide one-to-one assistance, (3) a discussion list or blog where users can ask questions or share ideas with others, (4) information about training sessions around New Jersey on digitization and metadata, (5) more resources on digital preservation and metadata, (6) educational activities that users can share with their patrons, (7) a tool for users to create their own interac tive activities using the NJDH resources, and ( 8) more information about helping patrons to use the NJDH more effectively.
n Portal structure The NJDH provides four portals for different target users: everyone, educators, students, and librarians and curators.Each portal provides different interface and packages dif ferent information for a different type of user.The survey found 80% of the subjects understood the purpose of the four portals (by marking 1 or 2 on the 7-point Likert scale) and only 4 participants (4%) found this type of portal struc ture confusing.The survey further found 65% of partici pants felt this kind of portal structure helpful to them.Why not contributing to the NJDH n collections?
The respondents indicated that the barriers for them to contribute collections or resources to the NJDH were, in this order: (1) lack of staff or time, (2) lack of funding, (3) lack of knowledge, and (4) copyright concerns.

n Statistical analyses
The study found demographic factors, such as age, gen der, ethnic background, and educational level, do not have significant effects on a number of areas: (1) how the participants ranked usefulness of the digital library, (2) usefulness evalu ation of the four-portal structure, (3) understanding of terminology, (4) ease of navigation, and ( 5) lostness.
The study found the correlation between navigation and lostness was statistically significant: r (66) = .83,p < .001.When a user felt the system easy to navigate, the user felt less lost.
The study also found usefulness of the digital library has a statistically significant effect on a user 's return decision.A one-way analysis of vari ance was conducted.The analysis of variance was significant, F (2, 59) = 20.42,p < .001.The strength of rela tionship between usefulness rank ing and the decision of whether to revisit the digital library, as assessed by n 2 , was strong, with the useful ness factor accounting for 41% of the variance of the return decision.Because the overall F test was significant, follow-up tests were conducted to evaluate pairwise differences among the means.Using the Turkey test, the pairwise comparisons Yes vs.No and Yes vs.Not Sure were significant.The pairwise com parison No vs.Not Sure was not significant.
n Conclusions Usability evaluation is a user-centered evaluation to learn from users' needs, expectations, and satisfaction.This research studied usefulness, navigation, user lost ness, terminology, and layout.The overall response was positive, and the finding was that the NJDH was useful in providing New Jersey history and culture information.Designers of the NJDH learned from the study the priori ties of adding various New Jersey themes to the collec tions and how to make the site easier to use.As a result of the study, lifelong learners are identified as an important target audience.
This research provided insights on why people came to use this particular digital library, their pleasure of using it, how to improve ease-of-use, navigation, website appearance, and the use of terminology and labeling.The front page of the website was redesigned to address the overuse of text on each page.The study also helped to discover what components of the site were more useful and why.Furthermore, it investigated why some muse ums or collections in New Jersey have not participated in this digital library development project.As a result of the study, more emphasis has been placed on building tools to increase independent collection contribution by muse ums and archives.The observations of this study may help the development of other academic digital librar ies because the barriers found in the study are common obstacles.After eighteen months of the study, the NJDH Governance Planning Committee still uses the evalua tion report to address more complex and fundamental changes and the reorganization of the digital library.
The study confirmed that users of this digital library appreciated the idea of providing different portals for different users.
The study did not find demographic factors (age, gender, ethnic background, and educational level) play statistically significant roles in the usefulness rankings of the digital library or portal structure, terminology, ease of use, or user lostness.
The study found there was a strong correlation between ease of navigation and user lostness.Users don't have feelings of lostness when a system is easy to navigate.
The study also found users will come back to revisit a digital library when they find the site is useful.

Figure 1 .
Figure 1.Where did you hear about NJDH?

Figure 2 .
Figure 2. Purpose of the most recent visit

Table 1 .
Demographic Data (N = 145) was a librarian, and one was a volunteer (see table3).