Evaluating Digital Scholarship, Promotion & Tenure Cases

Evaluating the work of a scholar traditionally involves several elements:

1. Reading and judging the work,
2. Looking to outside experts in the same area for their assessment of the work,
3. Taking note of the work’s formal peer review, from book and journal editors,
4. Considering citation of the research in the field at large, and
5. Sometimes, considering the impact that this work has had on the general public.

Each of these steps has a parallel in the evaluation of electronic scholarship, but not all forms of evaluation are always available.  And some issues that would not arise in evaluating scholarship in print are important to consider when evaluating electronic scholarship.

A number of institutions and organizations have made serious attempts to grapple with these issues, and though UVA does not have an institution-wide policy statement on the matter, it might be useful for the committee to consider the Modern Language Association’s guidelines on the subject (available on the web at http://www.mla.org/guidelines_evaluation_digital/), or those of some other colleges and universities—for example, the report of Mount Holyoke College’s “Guidelines for Evaluating Faculty Research, teaching and Community Service in the Digital Age” (http://www.mtholyoke.edu/committees/facappoint/guidelines.shtml).  The Mount Holyoke document, based on a year-long survey of the ways in which these issues are handled at other educational institutions, makes some useful points. 

The following is a discussion of the five elements of evaluation (above) and considerations to be made as they apply to digital scholarship.

1. Reading and judging the work
   
   It is indeed difficult to evaluate digital research and teaching materials through paper surrogates, so it is important for the committee to see the project, if possible, on the web.
   
   
2. Outside opinions
   
   While the committee will be primarily concerned with the comments in the traditional letters of review they solicit from outside scholars, it is perhaps worthwhile to consider unsolicited comments as well.  Digital projects often have “comment forms” through which people visiting the site (including colleagues in the field) may provide insightful, however informal, observations of the work.
   
   
3. Formal peer review
   
   This, and citation, are the two criteria—beyond personal judgment of the work—that have been most important, traditionally, in promotion and tenure.  Most originally digital publications by faculty are not peer reviewed, because most journals have not ventured into electronic publication (except as an alternative method for distributing printable content), and university presses have not yet risen to the challenge of publishing originally digital research, in any discipline.  Some projects do, therefore, assemble their own editorial boards, to review content before publication.  However, it is not clear that an invited editorial board provides an equivalent to independent peer review.  The absence of formal peer review is a problem common to much of the work of this medium.
   
   
4. Citation
   
   Note if the site been adopted, endorsed, and linked by any applicable official sites, library-based subject-collections of webs resources, scholarly association, and/or colleges and universities.
   
   
5. Public impact

   How many people have visited the site?  How does that figure compare to a traditional print publication in the same area of scholarship?

   Other considerations

   “Work with technology is often collaborative.  It is not uncommon for instructors on different campuses to link their courses, for example.  It is also not uncommon for people working with technology to work closely with others in different areas of the campus—e.g., faculty in history using the geographic information systems workstation and software in teaching and research.” 

   — “Guidelines for Evaluating Faculty Research, teaching and Community Service in the Digital Age”  Mount Holyoke College

Digital research seems to inculcate collaboration, and it opens up new and important collegial roles for graduate research assistants.  Who has collaborated with the faculty member under review?  What graduate students were involved and what was their role in the project?  But, also of consideration is the faculty member’s role of leadership in the project.  Has the faculty member been involved in each aspect of the project, and at each stage of its development?

 

(This text was taken from a 3 Dec 2001 letter by John Unsworth evaluating the digital scholarship of a faculty member up for promotion that year. Mr. Unsworth was director of the Institute of Advanced Technology in the Humanities at the time.)