Applying Hierarchical Task Analysis Method to Discovery Layer Evaluation
Libraries are implementing discovery layers to offer better user experiences. While usability tests have been helpful in evaluating the success or failure of implementing discovery layers in the library context, the focus has remained on its relative interface benefits over the traditional federated search. The informal site- and context specific usability tests have offered little to test the rigor of the discovery layers against the user goals, motivations and workflow they have been designed to support. This study proposes hierarchical task analysis (HTA) as an important complementary evaluation method to usability testing of discovery layers. Relevant literature is reviewed for the discovery layers and the HTA method. As no previous application of HTA to the evaluation of discovery layers was found, this paper presents the application of HTA as an expert based and workflow centered (e.g. retrieving a relevant book or a journal article) method to evaluating discovery layers. Purdue University’s Primo by Ex Libris was used to map eleven use cases as HTA charts. Nielsen’s Goal Composition theory was used as an analytical framework to evaluate the goal carts from two perspectives: a) users’ physical interactions (i.e. clicks), and b) user’s cognitive steps (i.e. decision points for what to do next). A brief comparison of HTA and usability test findings is offered as a way of conclusion.
Alling, Emily, and Rachael Naismith. “Protocol Analysis of a Federated Search Tool: Designing for Users.” Internet Reference Services Quarterly 12, 1/2 (2007): 195-201.
Annett, John, and Keith D. Duncan. "Task Analysis and Training Design." Occupational Psychology 41, (1967): 211-221.
Annett, John, and Neville Anthony Stanton, eds. Task Analysis. CRC Press, 2000.
Battleson, Brenda, Austin Booth, and Jane Weintrop. "Usability Testing of an Academic Library Website: a Case Study." The Journal of Academic Librarianship 27, 3 (2001): 188-198.
Bhatnagar, G., S. Dennis, G. Duque, S. Henry, M. MacEachern, S. Teasley, K. Varnum. University of Michigan Article Discovery Working Group Final Report. 2010.
Card, Stuart K., Thomas P. Moran, and Allen Newell, eds. The Psychology of Human Computer Interaction. Routledge, 1983.
Caswell, Jerry V., and John D. Wynstra. "Improving the Search Experience: Federated Search and the Library Gateway." Library Hi Tech 28, 3 (2010): 391-401.
Chojecki, P. (2006) “How to Increase Website Usability with Link Annotations”. Fraunhofer-Institute for Telecommunications - Heinrich-Hertz-Institute, Berlin,Germany.
Crystal, Abe, and Beth Ellington. "Task Analysis and Human-Computer Interaction: Approaches, Techniques, and Levels of Analysis." In AMCIS, (2004): 391.
Embrey, David. “Task Analysis Techniques.” Human Reliability Ltd. 2000. Accessed December 15, 2013. http://www.humanreliability.com/articles/Task%20Analysis%20Techniques.pdf
Fagan, Jody Condit, Meris A. Mandernach, Carl S. Nelson, Jonathan R. Paulo, and Grover Saunders. "Usability test results for a discovery tool in an academic library." Information technology and libraries 31, no. 1 (2012): 83-112.
Felipe, Sarah K., Anne E. Adams, Wendy A. Rogers, and Arthur D. Fisk. "Training novices on Hierarchical Task Analysis." In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 54, 23, (2010): 2005-2009. SAGE Publications.
Ford, Lyle. “Better than Google Scholar?” Presented at Internet Librarian 2010, Monterey, California. 2010. Accessed December 13, 2013. http://conferences.infotoday.com/documents/111/A105_Ford.pdf
Gorrell, Michael. “The 21st Century Searcher: How the Growth of Search Engines Affected the Redesign of EBSCOhost.”Against the Grain 20, 3 (2008): 22-26.
Green, Rebecca S. Cognitive Task Analyses for Life Science Automation Training Program Design. ProQuest, 2008.
Harris, S. “Discovery Services Sift Through Expert Resources.” Research Information, April/May, (2011): 18–21.
Hollan, James, Edwin Hutchins, and David Kirsh. "Distributed Cognition: Toward a New Foundation for Human-Computer Interaction Research." ACM Transactions on Computer-Human Interaction (TOCHI) 7, 2 (2000): 174-196.
John, Bonnie E., and David E. Kieras. "Using GOMS for User Interface Design and Evaluation: Which Technique?." ACM Transactions on Computer-Human Interaction (TOCHI) 3, 4 (1996): 287-319.
Johns-Smith, Susan. "Evaluation and Implementation of a Discovery Tool." Kansas Library Association College and University Libraries Section Proceedings 2 (2012): 17-23.
Kieras, David E., and David E. Meyer. "An Overview of the EPIC Architecture for Cognition and Performance with Application to Human-Computer Interaction." Human-Computer Interaction 12, 4 (1997): 391-438.
Khoo, Michael, and Catherine Hall. "What Would ‘Google’Do? Users’ Mental Models of a Digital Library Search Engine." In Theory and Practice of Digital Libraries, pp. 1-12. Springer Berlin Heidelberg, 2012.
Luther, Judy, and Maureen C. Kelly. "The Next Generation of Discovery." Library Journal 136, 5 (2011): 66-71.
Militello, Laura G., and Robert JB Hutton. "Applied Cognitive Task Analysis (ACTA): A Practitioner's Toolkit for Understanding Cognitive Task Demands." Ergonomics 41, 11 (1998): 1618-1641.
Nielsen, Jackob. “Goal Composition: Extending Task Analysis to Predict Things People May Want to Do”. Articles. Nielsen Norman Group. January 1, (1994). Available via: [January 2014]. http://www.nngroup.com/articles/goal-composition/
Nielsen, Jackob. “Why You Only Need to Test with 5 Users”. Articles. Nielsen Norman Group. March 19, (2000). Available via: [January 2014]. http://www.nngroup.com/articles/goal-composition/
Nielsen, Jakob. "Finding usability problems through heuristic evaluation." In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 1992, pp. 373-380.
Payne, Stephen J., and Thomas R. G. Green. "The Structure of Command Languages: an Experiment on Task-Action Grammar." International Journal of Man-Machine Studies 30, 2 (1989): 213-234.
Rauterberg, Matthias, and Morten Fjeld. "Task Analysis in Human-Computer Interaction: Supporting Action Regulation Theory by Simulation." Zeitschrift Fur Arbeitswissenschaft 52, (1998): 152-161.
Reason, James. "Combating Omission Rrrors Through Task Analysis and Good Reminders." Quality and Safety in Health Care 11, 1 (2002): 40-44.
Stanton, Neville A. "Hierarchical task analysis: Developments, Applications, and Extensions." Applied ergonomics 37, 1 (2006): 55-79.
Stone, Graham. "Searching life, the universe and everything? The implementation of Summon at the University of Huddersfield." Liber Quarterly20, 1 (2010): 25-51.
Thomsett-Scott, Beth, and Patricia E. Reese. "Academic Libraries and Discovery Tools: A Survey of the Literature." College & Undergraduate Libraries 19, 2-4 (2012): 123-143.
Tosaka, Yuji, and Cathy Weng. "Reexamining content-enriched access: Its effect on usage and discovery." College & Research Libraries 72, 5 (2011): 412-427.
Wisniewski, Jeff. “Web Scale Discovery: The Future’s So Bright, I Gotta Wear Shades.” Online 34, 4 (2010): 55-57.
Williams, Sarah C., and Anita K. Foster. "Promise fulfilled? An EBSCO Discovery Service Usability Study." Journal of Web Librarianship 5, 3 (2011): 179-198.
Yang, Sharon Q., and Kurt Wagner. "Evaluating and Comparing Discovery Tools: How Close are We Towards Next Generation Catalog?." Library Hi Tech 28, 4 (2010): 690-709.
Case Study References:
Find an Article:
Case 1. Yang, T. C., and Kevin D. Heaney. "Network-Assisted Underwater Acoustic Communications." In Proceedings of the Seventh ACM International Conference on Underwater Networks and Systems, p. 37. ACM, 2012.
Case 2. Jorgensen, William L., Jayaraman Chandrasekhar, Jeffry D. Madura, Roger W. Impey, and Michael L. Klein. "Comparison of Simple Potential Functions for Simulating Liquid Water." The Journal of Chemical Physics 79 (1983): 926.
Case 3. “Design Annual.” Graphis Inc., 2008
Case 4. Walb, M. C., J. E. Moore, A. Attia, K. T. Wheeler, M. S. Miller, and M. T. Munley. "A Technique for Murine Irradiation in a Controlled Gas Environment." Biomedical Sciences Instrumentation 48 (2012): 470.
Find a Book (physical):
Case 5. Few, Stephen. Show Me the Numbers: Designing Tables and Graphs to Enlighten. Vol. 1, no. 1. Oakland, CA: Analytics Press, 2004.
Case 6. Metcalf, Christine. The Love of Cats. Crescent Books, 1973.
Case 7. Machiavelli, Niccolò, and Leo Paul S. De Alvarez. 1989. The Prince. Prospect Heights, Ill: Waveland Press.
Case 8. Lees-Maffei, Grace, and Rebecca Houze, eds. The Design History Reader. Berg, 2010.
Find an eBook:
Case 9. Rubin, Jeffrey, and Dana Chisnell. Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. Wiley Technical Communication Library, 2008.
Case 10. Rubin, Jeffrey, and Dana Chisnell. Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. Wiley Technical Communication Library, 2008.
Case 11. Laube, Matthew Bryan. Ancient Awakening. 2010.