Evaluating Research Results

A week ago I taught students in a library-instruction session how to find books, articles, and and library information. They worked to find resources for a research project required for their class. I tried an active-learning exercise that I had not tried previously, which allowed the students to work in groups of two and analyze abstracts and elements of an article citation.

Like I do before most of my instruction sessions I practice many searches in the databases, particularly related to the the subject of the class. Generally, the instructors send me a few ideas of topics in which the students may have some interest. After finding a few results from a search that I had conducted, I looked at the abstracts and the document information attached to the articles. Recently, I had been feeling that I have not emphasized information-literacy skills as much as I would prefer. With this in mind, I decided to choose four or five abstracts from the results list of a search and hand them out to students in the class, so they could conduct their own analysis of the results.

By focusing their attention on one abstract and its accompanying citation elements, the students could practice using the evaluation criteria outlined in the CRAAP Test. Using the acronym CRAAP can help researchers remember five important points when evaluating many sources (Yes, our library webpage uses this test only in context of evaluating websites, but it can be useful for all types of resources.). Here are the five points:
- Currency
- Relevance
- Authority
- Accuracy
- Purpose

In groups of two, students read the citation and the abstract before discussing among themselves the different criteria. For this particular class that focused on physical education and sports history, I found some articles that talked about the Olympics and its history. It seems that when individuals begin a research project they should have a central question in mind, one that interests them, and one that is appropriate for the scope of their project. Students received the following question as a touchstone against which they could measure the article abstract with the five points listed above: "How have the modern Olympics evolved since 1896?"

Students needed about three minutes, but not more than five to evaluate their article citation and abstract. One student acted as spokesperson for their group, so that individual talked about whether the article they received would be a good source to answer the question listed above. When students have an opportunity to discuss in a small group, they tend to be more willing to share with the larger group afterward.

During the discussion we talked about the currency of an item and determined that an older article could be valuable for those doing historical research, but researchers and teachers generally prefer newer articles for their currency on the topic. Fields in the health sciences demand newer resources as new knowledge continually grows. Furthermore, we talked about the relevancy of an article, so if an article claims to talk about the changes from amateur to professional participation in the Olympics and we want to learn about female athletes in the modern Olympics, then it would not be useful. We know not to spend time reading that article and go to find another that addresses our concern. Likewise, an article about antique Greek coins depicting athletes in the ancient Olympics would not be pertinent to our inquiry.

When it came to authority, one abstract described how the Olympics went at the first modern iteration in 1896. It also stated that the article was only two pages long and written for Sports Illustrated. The students believed that anyone writing for this magazine would know their stuff--that they would be experts in sports. It provided a good opportunity to talk about the audience of this serial publication; typically it consists of the general public. Truthfully, writers for Sports Illustrated do need to be authorities in some regard, which may mean that they have watched sporting events their entire lives, participated in the sports extensively, or attended many such events in person. These writers may only have a bachelor's degree and may not be scholarly by any means. The key points I tried to emphasize were that the article only took up two pages of space, which meets the needs of a general audience but hardly scratches the surface for those who want to really gain an understanding of the 1896 Olympics.

Additionally, the fact that the authors of this article published in Sports Illustrated means that a less rigorous editing process took place. Undoubtedly, the editors at the magazine demand quality and accuracy; however, imminent deadlines do not permit the kinds of painstaking efforts required for in-depth fact checking, such as those taken by authors publishing in a peer-reviewed journal or a scholarly monograph publication. Admittedly, I would probably use a Sports-Illustrated article if it related directly to my topic, but I would also be sure to include other more thoroughly researched sources that may be more authoritative and accurate.

Looking back on the experience, we may not have discussed the criterion "purpose" at any great length, but it is one that writers need to remember. If a writer and research can identify the reasons for which an author wrote a particular article or book, they can use that as leverage to bolster or discredit their argument. For example, an activist in a well-known environmental group like Greenpeace or Friends of the Earth would be biased in opposition to group in the other direction.

Overall, the experience succeeded, and it got the students thinking critically about the resources--something we want to promote. In the future I might want to demonstrate the search that led me to the article titles and abstracts. It may be good to make students aware of my mental processes, so they can know how they might approach a similar search. Let them know which questions I am seeking to answer along the way. Ideally, I would like to illicit more responses from the students; I fear that I may have been too quick to talk after students gave their brief summaries of the abstracts. What kinds of questions would be best to ask after students have talked about their sources? Is it best to engage that particular group in the discussion or the class as a whole?