MetaSearching and Beyond : Implementation Experiences and Advice from an Academic Library

In March 2003 the University of Mississippi Libraries made our MetaSearch tool publicly available. After a year of working with this product and integrating it into the library Web site, a wide variety of libraries interested in our implementation process and experiences began to call. Libraries interested in this product have included consortia, public, and academic libraries in the United States, Mexico, and Europe. This article was written in an effort to share the recommendations and concerns given. Much of the advice is general and could be applied to many of the MetaSearch tools available. Google Scholar and other open Web initiatives that could impact the future of MetaSearching are also discussed.

In March 2003 the University of Mississippi Libraries made our MetaSearch tool publicly available.After a year of working with this product and integrating it into the library Web site, a wide variety of libraries interested in our implementation process and experiences began to call.Libraries interested in this product have included consortia, public, and academic libraries in the United States, Mexico, and Europe.This article was written in an effort to share the recommendations and concerns given.Much of the advice is general and could be applied to many of the MetaSearch tools available.Google Scholar and other open Web initiatives that could impact the future of MetaSearching are also discussed.

M
any libraries are looking for ways to facilitate the discovery process for users.Implementing a onestop search product that does not require databasespecific knowledge is one of the paths librar ies are choosing. 1As these search engines are made available to patrons, the burden of design falls to the library as well as to the product developers.Most library users may be familiar with a few databases, but the vast majority of electronic resources remain unrevealed.Using a MetaSearch product, a single search is broadcast out to similar and divergent electronic resources, and search results are returned and typically mixed together.MetaSearch results are returned in realtime and link the user to the native interface.Although there are many products that support onestop searching, the University of Mississippi Libraries chose to purchase Innovative Interfaces' MetaFind product because it tied into a digital initiative partnership with Innovative.Some of the possibilities of the types of resources you can search include: The simplicity of Google searching is very appeal ing to users.In fact, users have come to expect this kind of empowering tool.At the University of Mississippi, students use and have been using Google for research.As Google Scholar went public, it became evident that university faculty also use it for the same reasons.
It was apparent from the University of Mississippi Libraries' 2003 LibQUAL+ survey results that users would like more personal control than the library was offering (table 1).Unintentionally elaborate mazes are created and users become lost in a quagmire of choices.As indicated by our LibQUAL+ survey results, our users want easytouse tools that allow them to find informa tion on their own, and they want information to be easily accessible for independent use.These are clearly two areas that many libraries are struggling to improve for their patrons.The question is how to go about it.Based on several changes made between 2003 and 2005, which included implementing a MetaSearch tool, the adequacy mean improved for both questions and for undergradu ates as well as graduate students and faculty (table 2).The adequacy mean compares the minimum level of ser vice that a user expects with the level of service that they perceive.In table 1, the negative adequacy mean figures indicate that the library was not meeting users' minimum level of service for these two questions or that the per ceived level of service was lower than the minimal level of service.n Design perspectives and tension Generally, there are conflicts within libraries regarding the question of how to improve access for patrons and allow for independent discovery.For those leading a MetaSearch implementation, these tensions are important to understand.In implementing new technologies, there are key development issues that may decrease internal acceptance until they are addressed.However, one may also find that there are some underlying fears regarding this technology.Although the following crosssubculture comparisons simply do not do justice to each of the valid perspectives, these brief descriptions highlight the types of perspectives one might encounter when considering or implementing a MetaSearch product.
Expert searchers prefer native interfaces and all of the functionalities of the native interface.They are typically unhappy with the "dumbeddown" or clunky searching of a MetaSearch utility.They would prefer for patrons to be taught the ins and outs of the database they should be using for their research.This presupposes that the students either know which database to use, will spend time inves tigating each database on their own, or that they will ask for assistance.However, there are clearly native interface functionalities-such as lim iting to full text-that, while wonderful to patrons, are not consistent across resources or a part of the MetaSearch standard.Users would cer tainly benefit if limiting to fulltext was ubiquitous among vendors and if there were some way to determine fulltext availability within MetaSearch tools.Results ranking is another issue that expert searchers may bring to the table.Currently, there is a NISO MetaSearch Initiative that is striving to standard ize MetaSearching. 2 Another downside for the expert searcher is that there is no browse function.
Those who are in administrative or manage rial positions working with electronic resources see MetaSearching as an opportunity to reveal these resources to users who might not otherwise discover them.For example, many users have learned to search EBSCO's Academic Search Premier not realizing that key articles on a local civil rights figure such as James Meredith are also available in America: History & Life, JSTOR, and LexisNexis.MetaSearching removes the need for the user to spend additional time choosing databases that seem relevant and searching them indi vidually.From a financial perspective, if a library is pay ing for these electronic resources, they should be using them as much as possible.And while the University of Mississippi Libraries generally target the undergraduate audience with our MetaSearch tool, the James Meredith search is a good example of how a MetaSearch tool might reveal other databases with information that a serious researcher could then further investigate by link ing through the citation to the native interface.
Those associated with library instruction may also be uncomfortable with MetaSearching.In fact within a short time of implementing the product, several instructors conveyed their fear that in making searching so simple, they would no longer have a job as the product developed.Generally, it seems that users are always in need of instruc tion although the type of instruction and the tools continue to change.It is an understandable fear and one that would be wise to acknowledge for those embarking on a MetaSearch implementation.While MetaSearch can be an empowering tool for users, you may also encounter some emotional reactions among library employees.From an information literacy point of view, Frost has noted that MetaSearching is "a step backward" and "a way of avoiding the learning process." 3 It is true that in providing an easy search tool, the library is not endeavoring to teach all students intermedi ate or advanced information retrieval knowledge or skills.However, it is important to provide tools that meet users at their level of expertise and as previously noted, this is an area identified in need of improvement.
For those working at public service points such as the Reference Desk, MetaSearching is an adjustment.Many times those working with patrons tend to use databases with which they are more familiar or in which they feel more confident.Federated search tools may reveal resources that are typically less used and therefore unfa miliar to library employees.Training may then become an issue worthy of addressing not just for the MetaSearch interface and design but also for the lessused resources.
For those involved in technical support, this product may range from exciting to exasperating.The amount of time your technical support personnel have to dedicate to your MetaSearch project should be a major factor when investigating the available products.Just like any other technological investment, you are either going to (1) purchase the technology and outsource manage ment or (2) obtain a lesser price from a vendor for the tool and invest in developing it yourself.There is also a middle ground, but this costshifting is important to keep in mind.Regardless of your approach, it is critical to include the technical support person on your imple mentation team and to keep in mind the kind of time investment that is available when reviewing prices.Along with developing this product, one may also find oneself investing additional time and money into infra structural upgrades such as the proxy server, network equipment, or DNS servers.
In addition to these perspectives, there is a general tension in library Web site design philosophies between how librarians would like patrons to use their services and what patrons want.The traditional design based on educating users and having users navigate to information "our way" has definitely curtailed over the past several years with attention being paid increasingly to usability.
As usability studies give librarians increasing informa tion, libraries are moving toward designing for our users based on their approaches and needs rather than how librarians would have them work.
Depending on where one's library is in this spectrum of design philosophy, implementing a MetaSearch tool may be harder or easier.Judy Luther surmised the situa tion well, "For many searchers, the quality of the results matter less than the process-they just expect the process to be quick and easy." 4 Moving toward this lofty goal is to some extent dictated by the abilities and inabilities of the technologies chosen.As a technologist, the general rule seems to be that the easier navigation is made for our users; the more complex the technical structure becomes.

n MetaSearch categories
In arranging categories of searches for a MetaSearch product, some libraries group their electronic resources by subject, and others use categories that reflect fulltext avail ability.The University of Mississippi Libraries use both.The most commonly used category is our fulltext category.This fulltext category was set as the default on our most popular search box located on our articles and databases Web page (figure 1).Since limiting to fulltext materials is not a standard, the category was defined by the percentage of fulltext they contain.This is an important distinction to understand because a user may receive results that are not fulltext, but the majority of results will likely be fulltext.At our library, if the resource contains more than 50 percent fulltext, it is included in the fulltext category.
Other categories included in this implementation are ready reference, library catalogs, digital collections, lim ited resources, publicly available databases, and broad subject categories.One electronic resource may be included in the fulltext category, a broad sub ject category such as "arts and humanities" and also have its own individual category in order to mix and match individual resources on sub ject guides using a tailormade search box.The limited resource category contains resources that should be searchable using the MetaSearch tool but that have a limited number of simultaneous users.If it were included in the default fulltext category that is used so much, it would tie up the resource too much.Investigating resources with only one or two simultaneous users at the begin ning of the project may help you avoid error messages and user frustration.
One might wonder, "Why profile limited resources then?"There may be specific search boxes on subject guides where librarians decide to add that individual but limited resource.It might also be necessary to shorten the timeout period for limited user resources.Along those same lines, having paypersearch resources profiled could also be expensive and is not recommended.Since the initial implementation, migrating away from per search resources has become a priority.
Within the first few months of implementation, the free resources such as PubMed and AskEric were moved to a new "publicly available" category.The reason is that since there is not any authentication involved, these results return very quickly and are always the first results a user sees.While they are important resources, our intent was really to reveal our subscription resources.This approach allows users to search these resources if specifically chosen but they are not included in the default fulltext category.This approach does still allow Subject Librarians to mix and match these free individual resources on subject guide search boxes.

n Response time
Of all of the issues with our MetaSearch tool, response time has been the most challenging.There are so many issues when it comes to tracking down sluggish response that it can be extremely difficult to know where to start.If one's MetaSearch software is not locally hosted, response time could involve the library network, campus network, offcampus network provider, and the vendor's network, not to mention the networks of all the electronic resources users are searching.When one adds the other variable of authentication, the picture becomes even more over whelming and difficult to troubleshoot.
For authentication, the University of Mississippi Libraries purchased Innovative's Web Access Management Module (WAM), which is based on the EZproxy software.As the use of our electronic resources from oncampus and offcampus has grown, the inci dence of increasing network issues has risen.In work ing with our campus telecommunications group, the pursuit of evergreater bandwidth has become a priority.Troubleshooting has included tracking down trouble some switch settings, firewall settings, as well as campus DNS and vendor DNS issues.If your network adminis trators use packet shapers, this may be another hurdle.Clearly, our MetaSearch product has placed a significant load increase on the proxy server.In looking at proxy statistics, 24 percent of total proxy hits were from the MetaSearch product (figure 2).With this in mind, one may find the load on one's proxy server increasing very dramatically during peak usage and may need to plan for upgrades accordingly.
Even with improvements and tweaks along the way, response time is still an issue and one of the highest hurdles in selling a MetaSearch product internally and externally.One MetaSearch statistical module includes response time information for individual resources along with usage data.The response time information would be very helpful in troubleshooting and in working with electronic resource vendors.Usage tracking is another criterion to consider in reviewing MetaSearch products.
n Response time and tailored search boxes During implementation, one of the first discussions to have is who will be the target audience for this product.At this institution, undergraduates were the target audi ence and more specifically, those looking for three to five articles for a paper.While our MetaSearch software has a master screen showing all of the resources divided into the main categories, facing users with over sixty check boxes was not a good solution (figure 3).This master screen is good for demonstrating categories to library staff, overall functionality of the technology, and also for quickly checking all of your resources for connectivity errors.From early conversations with students, keeping basic users far away from this busy screen is a good goal.Remember, the purpose is to give them an easy starting point.
The best way to keep users in a simple search box is to construct search boxes and handpick either individual resources or categories keep ing in mind the context of the Web page.For example, the articles and databases page has a simple search box that searches for articles.Subject guide boxes search individual electronic resources selected by the Subject Librarian.The University of Mississippi Libraries also have a large col lection from the American Institute of Certified Public Accountants (AICPA).The search box on that page searches our catalog, which contains AICPA books along with the AICPA digital collection.Some libraries are interested in developing a standard MetaSearch box to display as a widget or standing content area throughout their Web site.This is interesting and worth considering.However, matching the Web page content with appropri ate resources has been our approach.As the standards and technology develop, this may be worth further con sideration depending on usability findings.For the most commonly used search box on the articles and databases page (figure 1), the default category checked is the full text articles category.Donna Fyer stated that, "For the average end user, the less decision making, the better." 5 This certainly rings true for our users.
Originally, a simple MetaSearch search box was placed on the library homepage.The library catalog and the basic MetaSearch box were both displayed.This seemed confusing for users since both products have search capabilities.With the next Web site redesign, the basic MetaSearch box moved from the library homepage to the articles and journals Web page.This was a success ful place for the article quick search box to reside since the default was set to search the fulltext category.There were some concerns that users might be typing journal titles into the search box but these were rare instances and not necessarily inappropriate uses.The next rede sign eventually moved this search box to the articles and databases page, where it remains.For the articles and databases pages, the simple search box (figure 1) by default searches the fulltext category and searches the title keyword index.The index category with the label, "Article Citations," can also be checked by the user.The majority of MetaSearches begin with this search box and n Subject guide search boxes In addition to the "Article Quick Search" box, Subject Librarians slowly became interested in a search box for their subject guides as the possibili ties were demonstrated.In order to do this, the ven dor was asked to profile each resource with its own unique value in order to mix and match individual resources.While the idea of searching resources by subject category sounds useful and appealing, sometimes universal design begets universal dis cord.Even with a steering committee involved, it is hard for everyone to agree what resources should be in each of the main subject categories: arts and humanities, science and engineering, business and economics, and social science.Some libraries have put a lot of time and effort into creating a large number of subject categories.
The master search screen (figure 3) displays several of this library's categories but not the broad subject categories noted above.These general sub ject categories are brought out in the multipurpose interface called the "Library Search Engine" (figure 4).The Library Search Engine design is a collection of the categories and resources showing the full functionality of our MetaSearch tool.The subject categorization approach within our MetaSearch interface is a good way to show the multifunction ality of the product but remains relatively unused by patrons.By giving each resource its own value, Subject Librarians have the flexibility to select spe cific resources and/or categories for their subject guides.It is worth noting that it required additional setup from our vendor and was not part of the original implementation.
After a few months of testing with the initial implemen tation, willing Subject Librarians chose individual resources for their tailored search boxes.Once a simple search box has been constructed, it can be easily copied with minor modi fications to make search boxes for those requesting them.While progress was slow to add these boxes to Subject Guides, after about a year there was growing interest.
In setting these up, Subject Librarians have several choices to make.First of all, they choose the resources that will be searched.For example, the biology subject guide search box searches Academic Search Premier, BioOne, and JSTOR by default.BasicBIOSIS and PubMed are also avail able but are not checked by default.Users can check these search boxes if they also wish to search these resources.Choosing the resources to include in the search box as well as setting what resources are checked by default is the most important decision.The Subject Librarian is also encour aged to assist in evaluating the number of hits per resource returned.With response time being a critical factor, deter mining the number of hits per resource should involve testing and take into consideration the overall number of resources being searched.

n Relevance
Selecting the default index is another decision in setting up search boxes.Again, users are Googleoriented and tend to go with whatever is set as the default option.Out of the box, our MetaSearch tool defaults to the keyword index or keyword search.The issue of relevancy is a hot topic for MetaSearch products.This issue typically comes up in MetaSearch discussions.It is also listed as an issue in the NISO MetaSearch initiative.From the technical side of the equation, results are displayed to the user as soon as they are retrieved.This allows users to begin immediately exam ining the results.Adding a relevancy algorithm as a step would mean all of the results would have to be returned, ranked, and then displayed.With response time being a key issue, a faster response is more important than relevance.Another consideration is if the MetaSearch results are displayed to the user as interfiled or by electronic resource where the resource is returning results based on its own relevancy rankings.
One way to increase relevance is to change the default index from keyword to title keyword.For our students, bringing back keywords in the title made the results more relevant.This is the default index used for our article search on the articles and database Web page.Subject Librarians have the choice of indexes they prefer when blending resources.
One caveat in using title keyword is that there are resources that do not support title keyword searching.For other resources, title keyword is not an appropriate index.For example, Wilson Biographies does not have a title keyword search.It makes perfect sense that a biography database would not support title keyword searching.In these cases, the search may fail and note that the index is not supported.To accommodate this type of exception, the profile for Wilson Biographies needed to be changed to have the title keyword searchmapped to a basic keyword search.While this does not make the results as relevant as the other search results, it keeps any errors from appearing and allows results to be retrieved.
n Results per source and per page For MetaFind, there are also two minor controls that can work as hidden values unseen by the patron or as compo nents within the search box for users to manipulate.The first control is the number of hits to return per resource.If a Subject Librarian is only searching two or three resources in his tailored search box, he probably will want to set this number higher.If there are many resources, this number should be lower in order to keep response time reasonable.The second control is the number of results to return per page.In general, it is important to adjust these controls after testing the response for the resources selected.While users typically use the default settings, showing these two con trols gives the user a visual clue that the MetaSearch tool is not retrieving all of the results from the resource.Instead, it is only retrieving the first twentyfive, for example.

n Implementation advice
One of the most important pieces of advice is that it is extremely important to have a date in one's contract or RFP for all of the profiling to be completed if the vendor is doing the resource profiling.From this library's experi ence, the profiling of a resource can take a very long time, and this is a critical point to include in the contract.One might also consider adding cost and turnaround time for new resources after the initial implementation to the contract.The more resources profiled, the more useful the product.However, one also needs to pay attention to response time.If the plan is to profile one's own resources or connectors, librarians should be mindful of the time involved and ask other libraries with the same product about time investments.Being able to work with vendors who will provide an opportunity to evaluate the product "live" is preferable.
In deciding who to target for an implementation team, consider representatives from reference, collection development, and systems.It is also very important to include whoever manages electronic resource access/ subscriptions and a Web manager.In watching other pre sentations, exclusion of any of these representatives can seriously undermine the implementation.Buyin is essen tial to success.Additionally, giving librarians as many options as possible, such as control over what types of resources are in their search boxes as well as the number of hits per resource makes the product more appealing.

n Questions to ask
Once the implementation team is set, interviewing refer ences for the products under consideration is an impor tant part of the process.Unstructured conversations with references really allow librarians to explore together what the group wants and how its needs fit with the services the vendor offers.A survey of questions via email is another possibility.In choosing this method, be sure to leave some room for open comments.Regardless of the approach, it is important to spend some time asking ques tions.Provided are a list of recommended questions: n Is there a list or database of alreadyestablished pro files for electronic resources for this product?n How much time would you estimate that it took to implement the product?n Will you be able to edit all of the public Web pages yourself or will you be using vendor support staff to make changes?If the vendor support staff has to make some of the changes, how responsive are they?As with most technologies, there are upfront and hid den costs.It is important to determine what hidden costs are involved and if you have the resources to support all of the costs.Sometimes libraries choose the least expen sive product.However, this approach can lead librar ies down the path of hidden costs.For example, if the product is less expensive but your library is responsible for setting up new electronic resources, managing all of the pages, and finding ways to monitor and troubleshoot performance outside of the tools provided, the hidden expenditures in time and training may be more costly in the end than purchasing the premium MetaSearch tool.In essence, one must pay for the product one way or another.The big question is, Where are the resources to support the product?If one's library has more IT/Web personnel than money, the lowercosting product may be the way to go, but be sure to check with other librar ies to see if they have been able to successfully clear this hurdle.Additionally, if your library has more onetime money than yearly subscription money, this may dictate the details of the RFP, and your library may lean toward a purchase rather than an annual subscription.

n MetaSearch summary
Clearly, students want a simple starting place for their research.Implementing a MetaSearch tool to meet this need can be a hard sell internally for many reasons.At this institution, response time has been the overriding critical issue.Response has lagged due to server and network issues that have been difficult to track down and improve.However, authentication is truly the most time consuming and complex part of the equation.Some fed erated search tools are actually searching locally stored information, which helps with response.While these are not truly MetaSearch tools and are not performing real time searches, this approach may yield more stability with faster response.
Over the years in implementing new services such as the library Web site, ILLiad, electronic resources, and off campus authentication, new services are often adopted at a much faster rate by library users than by library employees.Typically, there will be early adopters who use the services immediately based on need.It then takes general users about a year to adopt a new service.III's MetaSearch technology has been available for the past four years.However, our implementation is evolving with each Web site redesign.Still, it is used regularly.
The University of Mississippi Libraries has been pro viding access to its electronic resources in two distinct ways: (1) providing URLs on Web pages to the native interface of the electronic resource and (2) MetaSearching.As the library moves forward in developing digital col lections and the number of electronic resources profiled for MetaSearching increases, it is possible that this kind of global discovery tool will compete in popularity with the library catalog.Providing such information mining tools to patrons will cause endless frustration for the library literate.Response times, record retrieval order, as well as licensing and profiling issues, are all obstacles to pro viding a successful MetaSearch infrastructure.Retrieval inconsistency and ad hoc retrieval order of records is very unsettling for librarians.However, this is the kind of tool to which Web users have become accustomed and certainly seems to fill a need that to date has been lacking where library electronic resources are concerned.

n Open Web developments
One other trend appearing is scholarly research discovery tools on the open Web.Enter Google Scholar along with other similar initiatives such as Windows Live Academic Search.Google Scholar BETA was released in November 2004 and very soon after began an initiative to work with libraries and their OpenURL resolvers. 6This bridging between an open Web tool and libraries is an interest ing development.A fair amount has been written about Google Scholar to date although the project is still in its beta phase.
What does Google Scholar have to do with MetaSearching?Good question.It remains to be seen how much scholarly information will become search able via Google Scholar.For now, the jury is still out as to whether Google Scholar will begin to encroach upon the traditional territory of the indexing and abstracting world.If sufficient content becomes available on the open Web, whether from publishers or vendors allowing their content to be included, then the authentication piece that directly effects response time may be overcome.In using Google Scholar or other such open Web portals, search ing happens instantly.When a user uses the OpenURL resolver to get to the fulltext, that is where authentication enters into the picture and removes the negative impact on searching.The tradeoff is that there are many issues involved in OpenURL linking and the standardization of the metadata needed to provide consistent access.
There are many parallels between what Google Scholar is attempting to offer and what the promises of MetaSearching have been.For MetaSearching, under graduate students looking for their three to five articles for a paper are considered our target audience.For in depth searching, MetaSearching does have limitations, but for the casual searcher looking for a few fulltext articles, it works well.Interestingly, similar recommen dations are being made for Google Scholar. 7However, opinions differ on this point.Roy Tennant went so far as to indicate it is a step forward in access to those users without access to licensed databases, but remained reserved in his opinion regarding the usefulness for those with access. 8oogle Scholar also throws in a few bonuses.While providing access to open access (OA) materials in our OPAC for specific collections such as the Directory of Open Access Journals, these same resources have not been included in our MetaSearch discovery tool.Google Scholar is searching these open repositories of scholarly informa tion, although there is some concern over the automatic inclusion of materials such as syllabi and undergraduate term papers within the institutional repositories. 9Google Scholar also provides a useful citation feature and rel evancy.Google Scholar recognizes the user's preference for fulltext access and provides a visual cue from the brief results when article fulltext is available.This func tionality is not currently available from our MetaSearch software but would be extremely helpful to users.On the downside, some of Google Scholar's linking policies make it difficult for libraries to extend services beyond full text articles to their users.Another notable development among subscription indexing services is the ability to reveal content to Web search engines.EBSCO's initiative is called Ebscohost Connection. 10 In implementing MetaSearching, libraries have debated about providing access to free versus subscrip tion resources.For our purposes, free resources were not included in the most commonly used search in the full text category.There are those who would argue against this decision, and they have very good points.In fact, it has already been noted that some libraries use Google Scholar to verify incomplete interlibrary loan citations quickly. 11In watching the development of Google Scholar, it seems possible that this free tool that uncovers free open access resources and institutional repository mate rials may not necessarily be a competitive product, but may be a very complementary one.
n Impact on the OPAC What will this mean for the "beloved" OPAC?For a very long time, users have expected more of the library catalog than it has provided.While the library catalog is typically appreciated by library personnel, its usefulness for finding materials other than books has been hard for general users to understand.Many libraries including the University of Mississippi have been loading records from their electronic resources in hopes of making the library catalog more useful.The current conversation regarding digital library creation also begs the question, "What is the library catalog?"Although the library catalog serves as a searchable inventory of what the library owns, it is simply a pointing mechanism, whether it points the user to a shelf, a building, or a URL.
In our endeavor to provide instant gratification and fulltext, as well as the user's desire for information regardless of format, the library catalog is beginning to take a backseat.It was clear four years ago in plan ning digital collections that a MetaSearch tool would be needed to tie together subscription resources, digital collections, publicly available resources, and the library catalog.It will be interesting to see whether patrons choose to use the formal tools provided by the library or the informal tools developing on the open Web, such as Google Scholar, to perform their research.More than likely, discovery and access will happen through many avenues.While this may complicate the big picture for those in library instruction, it is important to meet users on the open Web.
One's best intentions and designs are presented to users but they may choose unintended paths.Librarians should watch the paths they are taking and build upon them.Sometimes even one's best attempts fall short, as pointed out clearly in Karen Schneider's latest series, "How OPACs Suck." 12 Still it is important to acknowl edge design shortcomings and keep forging ahead.Dale Flecker, who spoke at the TAIGA Forum, recommended not to spend years trying to "get it right" before imple menting, but instead to consider ourselves in perpetual beta and simply implement and iterate. 13In other words, do not try to make the service perfect before implement ing it.Most libraries do not have the time and resources to do this.Instead, find ways to gain continual feedback and constantly adjust and develop.
Students are familiar with Internet search engines and do not want to choose between resources.Access to a simple resource discovery tool is an important service for users.Unfortunately, authentication, product design

Figure 1 .
Figure 1.MetaSearch tailored search box with full text category selected

n
Who is responsible for setting up each resource-the vendor or you?n How much time does it typically take to set up a new resource and what is the standard cost to add a new resource?

nn
Can you easily mix and match individual resources for subject guides, departmental pages, or other kinds of Web pages?Or do you only have the option to set up global categories?n Is your installation local or does the vendor host it?Are there response issues?n Is there an administrative module to allow you to maintain categories, resource values, and configura tion options?n How much time goes into managing the product monthly?And who manages the product at your library?What kind of statistical information does the vendor provide?n How satisfied are you with the training, implementa tion support, and technical documentation?n How does the vendor handle broken resources or subscription changes?

Table 2
compares the adequacy mean from 2005 with 2003 and indicates a notable, positive change in adequacy mean for each question and with each group.

Table 2 .
Positive change in LibQUAL adequacy mean from 2003 to 2005