New Trends and the Evaluation of Scholarship

Introduction

The advancement of information and communication technologies over the past decade, particularly the growth of the Internet, the World Wide Web (Web), and email, have had an impact on how scholarship is conducted and are re-defining many aspects of scholarly communication. Interdisciplinarity, collaboration, and disintermediation are three aspects of scholarly communication that are on the increase as a result of the advancement of information and communication technology. The trend towards increased interdisciplinary and collaborative scholarship, combined with the disintermediation of traditional journal publishers and publishing as more scholarship is made directly available electronically, make the evaluation of scholarship more difficult. This paper will briefly examine these trends, the advancement of new technology, and how they may effect the evaluation of scholarship.

Backgound

Changes in scholarly communication and the evaluation of scholarship are discussed in this paper in the context of several concepts. Operationalizing the key concepts used in this discussion results in the following definitions. Scholarship is both the work, and the result of the work, performed by scholars to produce new information to be added to a body of knowledge. The addition of scholarly work to a body of knowledge is accomplished by communicating, usually in writing, either in print or digital format, the findings produced by scholarship to the wider, usually academic, community. Borgman (2000) defines scholarly communication as “…the study of how scholars in any field…use and disseminate information through formal and informal channels” (p. 144). Evaluating scholarship entails examining the written results of scholarly work, but it also includes examination of the methods and procedures used to produce that work.

Interdisciplinarity refers to the increasing trend of scholars to participate in research and publication activities with other scholars outside of their own academic discipline. Webster’s online defines discipline as “a branch of knowledge or teaching; the subject matter of instruction” (Webster’s, 2003). “Interdisciplinarity is a term which refers to the integration of concepts across different disciplines” (Wikipedia, 2003). The most complete definition found was one quoted in Lattuca (2002) and specifies the different types of interdisciplinary actions that are at the core of interdisciplinarity.

Interdisciplinary – An adjective describing the interaction among two or more different disciplines. This interaction may range from simple communication of ideas to the mutual integration of organizing concepts, methodology, procedures, epistemology, terminology, data, and organization of research and education in a fairly large field. An interdisciplinary group consists of persons trained in different fields of knowledge (disciplines) with different concepts, methods, and data and terms organized into a common effort on a common problem with continuous intercommunication among the participants from the different disciplines (p. 712).

The continuous intercommunication among researchers referred to in the definition above is central to another concept, collaboration. Collaboration can be defined simply as, “to work together, especially in a joint intellectual effort” (Webster’s, 2003). There are perhaps some scholars, in certain disciplines, who would view interdisciplinary collaboration more along the lines of the secondary definition, “to cooperate treasonably, as with an enemy…” (Webster’s, 2003).

Disintermediation “is giving the user or the consumer direct access to information that otherwise would require a mediator…” (SearchCIO, 2003). Webopedia (2003) gives the following, business oriented, definition for disintermediation.

Removing the middleman. The term is a popular buzzword used to describe many Internet-based businesses that use the World Wide Web to sell products directly to customers rather than going through traditional retail channels. By eliminating the middleman, companies can sell their products cheaper and faster. Many people believe that the Internet will revolutionize the way products are bought and sold, and disintermediation is the driving force behind this revolution.

Applied to scholarship and scholarly communication, disintermediation means using the Internet or other electronic means to disseminate scholarly work quickly and widely (sell products directly) rather than relying on the expensive print journals (traditional retail channels) as the initial, or only, method of communicating the results of research. Putting these concepts together as outlined in the introduction results in the following thesis statement. Information and communication technologies are re-defining scholarly communication; there is a trend towards interdisciplinarity, collaboration, and disintermediation and these trends make evaluation of scholarship difficult.

Review of Literature

The print format has been the principle means of disseminating scholarly research for several centuries and it has continued to grow, even in the last 20 years. From the middle of the twentieth century to 1984, the number of periodicals grew from 20,000 to 66,000. From 1984 to 1995 it increased to 145,000 titles, most aimed at the academic community (Meszaros, 2002). The last 20 years has also seen the rapid advance of information and communication technologies. Investment in technology that makes communication easier and cheaper “has increased rapidly in the United States and other developed economies” (Hamermesh & Oster, 2002, p. 539). Technical change and deregulation dropped telephone rates by 50%, leading to higher phone and fax usage. Fax use increased 20% just between 1996 and 1997 (Hamermesh & Oster, 2002). Advancing technology and inexpensive telecommunication, coupled with the growth and acceptance of the Internet, led to an explosion in the use of the Web, especially the use of email. “Electronic communication…have quickened the pace of scholarly exchange to be nearly instantaneous and have democratized the scholarly community into a global network” (Meszaros, 2002, p. 32).

Discussing the impact of the Internet and the Web on scholarly communication, Meszaros (2002) writes, “besides the shift in the mechanisms of information exchange, the Internet has expanded the scholar’s world both within and outside the academic community. Academics rarely isolate themselves within one discipline anymore, as they come to appreciate the benefits of interdisciplinary exchanges and collaborations” (p. 36). The advance in communication and information technology “permits a depth of interaction and connections never before possible among academic disciplines” and “the increased speed of communication provides a more immediate exchange of ideas” (Meszaros, 2002, p. 36). This interaction and communication is perhaps most advantageous for those scholars separated by large distances. No longer forced to wait for annual conferences or the appearance of print publications, scholars are able to “…produce original research, host interactive discussions, and otherwise provide an electronic scholarly environment” (Meszaros, 2002, p. 39). Discussing the results of a study examining the role of technology in economics scholarship, Hamermesh & Osterman (2002) noted that only 5% of research in the 1970s was conducted collaboratively between researchers separated by long distances, but during the 1990s it had increased to 20%.

Bordons & Gomez (2000) state that science policy makers who assume that cooperation will yield an increase in benefits have in part fostered the trend towards increased interdisciplinary collaboration. Another impetus toward collaboration is that cooperation between over-specialized scientists is necessary in order to solve complex scientific problems. “…The ever-increasing specialization of scientists, together with the need to develop multidisciplinary approaches to solve scientific problems, is an underlying reason for the increasing collaboration between researchers” (Bordons & Gomez, 2000, p. 199). Collaboration is facilitated by advances in telecommunication technology that “…enable the high speed diffusion of knowledge and ideas, while modern transport facilities enhance the mobility of scientists and improve scientific communication” (Bordons & Gomez, 2000, p. 199).

Writing about the impact of technology on research and publication in the natural sciences, Atkins & Shrode make the following statement.

The Web makes is possible for researchers to access data from any geographic location, regardless of their personal status. In this respect, an individuals existing reputation in the field is less important now than in earlier eras. Electronic dissemination of information fosters communal efforts, reducing barriers to communication and diminishing limits imposed by tightly organized invisible colleges. Researchers at the beginning of their careers can benefit from access to the same resources used by their more advanced colleagues who have established successful records in their fields (p. 104).

The communal efforts and reduced barriers are akin to collaboration and interdisciplinarity, and the lack of the need for status or established careers is the democratization referred to by Meszaros. The third trend identified in the thesis statement, disintermediation, “relates to the role of the intermediary in acting between information (or other products) and its end-users. It is the finding of information by an end-user without the need for a third party” or the “diversion of information from centralized physical repositories to alternate sources available directly through computers and computer networks” (Fourie, 1999, p. 10). While Fourie is discussing disintermediation in terms of its affect on librarians, the statement is equally applicable to the trend of scholarly communication being diverted from print journals to electronic format for direct and more current access by interested parties.

Affect of Trends on Evaluation

While it has been shown that information and communication technologies are re-defining scholarly communication, part of this redefinition entails providing the means for a concerted effort to transform the current system of scholarly publishing. “The current system of scholarly publishing has become too costly for the academic community to sustain” (ARL, 2000, p. 1). Recognizing the value of electronic media to help bring about a change, the ARL statement goes on to say that “electronic capabilities should be used, among other things, to: provide wide access to scholarship, encourage interdisciplinary research, and enhance interoperability and searchability” (ARL, 2000, p. 3). In a similar vein in an article discussing the cost of journals and the power of publishers, Frazier writes that “the most important thing that librarians can do to change the rules of the game is to invest in bold new experiments in scholarly communication…in investing in these new forms of scholarly communication, we are steadily building the publishing infrastructure so that future scholars may never have to publish in an expensive commercial journal in order to be academically successful” (Frazier, 2001, last section). The key phrase is ‘academically successful’ and that is where evaluation becomes a key point. The ARL statement recognizes this, stating that “the system of scholarly publication must continue to include processes for evaluating the quality of scholarly work…” (ARL, 2000, p. 4).

The evaluation of scholarship is an important component of scholarly communication, particularly where academic success is concerned. Evaluative judgments as to the quality and quantity of scholarship are often key factors in promotion and tenure decisions for scholars. While some argue that the trend of collaboration has led to “…increases in productivity, visibility, and quality of research” (Bordons & Gomez, 2000, p. 207), others state that collaboration made possible by technological advances “…did not generate higher quality output per unit” (Hamermesh & Oster, 2002, p. 554). Differentiating between core (old, established, and accepted) and frontier (newly created and published) knowledge, Cole (2000) writes that what separates the one from the other is the evaluation process. Much frontier knowledge will be deemed unimportant and not of a quality necessary to begin a move to the core. To date, print journals, especially the more prestigious visible ones, have played an important part in the evaluation of scholarship.

Of the numerous methods available for the evaluation of scholarship, bibliometrics provides several that can be used. These include Lotka’s, Zipf’s, and Bradford’s laws as well as citation analysis. “Collaboration among companies that traditionally provided indexing and abstracts for journal articles and scholarly publishers creates systems that allow scientists to move easily from an article citation to its full text” (Atkins & Shrode, 2002, p. 122). As databases make research results more accessible and easier to find, they also provide the data necessary for bibliometric evaluation such as citation analysis. In some ways, this is a disadvantage of the trend towards disintermediation and increased electronic publishing. Research that is made available electronically, bypassing the traditional print publication path, never enters the databases needed for bibliometric methods of evaluation. It also bypasses the traditional peer-review evaluation system of the print journals. This does result in making evaluation of scholarship more difficult.

Scholarly communication through print journals, while adding to a body of knowledge, also serves to evaluate the work of contributors through the selection process. “Publication is understood to be the primary channel through which individual faculty demonstrate their worthiness for tenure, promotion, grants, and fellowships” and “the peer review mechanisms that underlie the decision of any publisher to accept an article or full length manuscript help to certify the value of any given contribution as well as contributor to the field” (Publish & Perish, 1998, p. 3). Peer review functions as a conservative force that determines out of all submitted material that which is of high enough quality to be published (Kahn, 1992, sect. 3). In doing this, peer review is serving an evaluation method on its own, as it offers “…a vetting, gatekeeping and quality control function (through the initial selection of articles) and quality improvement (through refereeing and revision)” (Lally, 2002, p. 83).

As important as peer review is to the print publication process and to the academic success of scholars, it has its own limitations and shortcomings (van Raan, 2000). There is often subjectivity and bias involved, and the vast amounts of research being produced in so many different specializations make it difficult for individual reviewer to adequately and accurately assess the quality of submitted research. Electronic publication and the availability of even more research via the Web only compounds the problem of too much information, but it can help to eliminate subjectivity and bias as a world wide audience is immediately able to evaluate and provide feedback on the research. However, evaluation through print publication is still seen as the most important factor in assessing scholarly performance. Electronic publication makes this form of evaluation more difficult, and some hope that it will lead to a change in evaluation standards for scholars, resulting in more emphasis being placed on teaching, discovery, collaboration, and integration (Meszaros, 2002). There has not been an acceptance of common standards however, and many scholars are hesitant to forego print publication since it could affect their careers.

The re-defining of scholarly communication and the trends affecting scholarship are having wide impacts. “In consequence, everything from peer review and tenure consideration, to the integrity of the published record, to the ownership and control of intellectual property, to new publishing models and the economic well-being of commercial publishers, is being reconsidered” (New Horizons, no date, p. 1). This reconsideration is due, at least in part, to the difficulties encountered in evaluating scholarship in today’s climate of interdisciplinary and collaborative electronic communication. While the Web has made scholarship available faster and to a wider audience it has its own drawbacks. Electronic data is not permanent and is subject to change, degradation, and loss. Preservation of scholarship published in electronic format is an ongoing concern. Many scholars are also worried about plagiarism and intellectual theft (Meszaros, 2002). The citations that exist in print publication often take the form of hyperlinks in digital format. “A reader can follow the path directly from a document to those documents it cites, and from those documents to ones cited therein, as long as the links are available online” (Borgman, 2000, p. 148). It is quite common, however, for those links not to be working as papers are moved, sites and addresses are changed, data is not migrated to new platforms and software, or servers are simply no longer maintained. The lack of permanence of scholarship in electronic form is another factor that makes evaluation difficult.

Conclusion

Advances in communication and information technology has made many of the changes taking place in scholarly communication possible, but much of the cause for the change has been the high prices of print journals and the monopoly like power of a few large publishers. The revolt against the print publishers has in part fueled the growth of direct electronic publishing of research results by scholars. The growth of electronic means for scholarly communication (disintermediation), combined with the trend towards collaboration and interdisciplinarity, has made evaluation of scholarship more difficult in the traditional sense. What is needed are updated methods and standards for evaluating scholarship in this new environment that emphasize other aspects of scholarly work besides the publishing of articles in high prestige print journals. New evaluation techniques with less emphasis on print publication will help to further alleviate the cost problem associated with print journals, will make life more enjoyable for scholars, and may possibly enhance the education of students studying under those scholars.

References

ARL. (2000). Principles for emerging systems of scholarly publishing. In the Association of Research Libraries ‘Issues in Scholarly Communication’ website. Retrieved 2 May 2003 from http://www.arl.org/scomm/tempe.html.

Atkins, D. P., & Shrode, F. G. (2002). Technology’s impact on research and publication in the natural sciences. In C. F. Thomas (Ed.), Libraries, the Internet, and scholarship: Tools and trends converging (pp. 103-132). New York: Marcel Dekker, Inc.

Bordons, M., & Gomez, I. (2000). Collaboration networks in science. In B. Cronin & H. B. Atkins (Eds.), The web of knowledge: A festschrift in honor of Eugene Garfield (pp. 197-213). Medford, NJ: Information Today, Inc., ASIS Monograph Series.

Borgman, C. L. (2000). Scholarly communication and bibliometrics revisisted. In B. Cronin & H. B. Atkins (Eds.), The web of knowledge: A festschrift in honor of Eugene Garfield (pp. 143-162). Medford, NJ: Information Today, Inc., ASIS Monograph Series.

Cole, S. (2000). The role of journals in the growth of scientific knowledge. In B. Cronin & H. B. Atkins (Eds.), The web of knowledge: A festschrift in honor of Eugene Garfield (pp. 109-142). Medford, NJ: Information Today, Inc., ASIS Monograph Series.

Fourie, I. (1999). Should we take disintermediation seriously? The Electronic Library, 17(1), 9-16.

Frazier, K. (2001). The librarian’s dilemma: Contemplating the costs of the “big deal.” D-Lib Magazine, 7(3). Retrieved from http://www.dlib.org/dlib/march01/frazier/03frazier.html.

Hamermesh, D. S., & Oster, S. M. (2002). Tools or toys? The impact of high technology on scholarly productivity. Economic Inquiry, 40(4), 539-555.

Kahn, B. (1992). Scholarly Communication in the network environment: Issues of principle, policy, and practice. Retrieved 2 May 2003 from http://www.eff.org/IP/kahin_scholarly_communication.

Lally, E. (2001). A researcher’s perspective on electronic scholarly communication. Online Information Review, 25(2), 80-87.

Lattuca, L. R. (2002). Learning interdisciplinarity: Sociocultural perspectives on academic work. The Journal of Higher Education, 73(6), 711-739.

Meadows, J. (2000). The growth of journal literature: A historical perspective. In B. Cronin & H. B. Atkins (Eds.), The web of knowledge: A festschrift in honor of Eugene Garfield (pp. 87-107). Medford, NJ: Information Today, Inc., ASIS Monograph Series.

Meszaros, R. L. (2002). The Internet, scholarly communication, and collaborative research. In C. F. Thomas (Ed.), Libraries, the Internet, and scholarship: Tools and trends converging (pp. 31-44). New York: Marcel Dekker, Inc.

New Horizons in Scholarly Communication. (no date). Scholarly publishing in the digital age. A publication of the Librarians Association of the University of California. Retrieved 2 May 2003 from http://libweb/ucsc.edu/scomm/pubhome.html.

Publish & Perish. (1998). To publish and perish. Policy Perspectives: Co-sponsored by the Association of Research Libraries, the Association of American Universities, and the Pew Higher Education Roundtable, 7(4), March 1998. Retrieved 2 May 2003 from http://arl.cni.org/scomm/pew/pewrept.html.

SearchCIO. (2003). Retrieved 2 May 2003 from http://searchcio.techtarget.com/sDefinition/0,,sid19_gci211962,00.html.

van Raan, A. F. J. (2000). The pandora’s box of citation analysis: Measuring scientific excellence-the last evil? In B. Cronin & H. B. Atkins (Eds.), The web of knowledge: A festschrift in honor of Eugene Garfield (pp. 301-319). Medford, NJ: Information Today, Inc., ASIS Monograph Series.

Webopedia. (2003). Retrieved 2 May 2003 from http://www.webopedia.com/TERM/D/disintermediation.html.

Webster’s online dictionary. (2003). Retrieved 2 May 2003 from http://dictionary.reference.com/search?q=discipline and http://dictionary.reference.com/search?q=collaboration.

Wikipedia, the free encyclopedia. (2003). Retrieved 2 May 2003 from http://www.wikipedia.org/wiki/interdisciplinarity.