Finding evidence that answers a question is only the first part of the research process. You also have to evaluate the quality and credibility of your research. Inevitably, as we’ve already seen in this chapter, you do this as you consider the origins of your research—primary versus secondary research, scholarly versus popular sources, the Internet, and so forth. But evaluating the quality and credibility of your research is more subtle and complicated than just determining the source of the evidence. Consider again the example from the beginning of this chapter about deciding which computer to buy. One of the things you would have to weigh is the credibility of the information you received from your friends compared to the information you received from a salesperson at the computer store. You can probably count on your friends to be trustworthy and honest, but they might not know much about computers. Conversely, while a salesperson might know a lot about computers, you may be uncertain to what extent you can trust him to give you the best advice. The salesperson wants to sell you a computer, which means that his motivations might be consciously or unconsciously influencing the information he is providing you.
Who should you trust? We have all been in situations like this, and there is no easy way to answer that question. Chances are, you’ll make your computer decision based on your interpretation of the evidence and based on what you perceive to be the reliability and credibility of your different sources. If someone else were faced with the same computer decision and the same evidence, they might make a different choice. That is why there are different kinds of computers on the market and that is why different people can do the same sort of research about “the best” computer and why they can arrive at different conclusions.
Academic research is not much different in the sense that different researchers, considering the same or similar evidence, often arrive at different conclusions. Academic research rarely provides clear answers in the sense of definitively knowing the “rights” and “wrongs” about some issue. Not all academics think that computer hacking is wrong (or right), that the solution to commercial over-fishing is strict international control, or that F. Scott Fitzgerald’s novel The Great Gatsby depicts the connection between material goods and the American dream. Rather, there are debates about these issues, differences of interpretation and opinion that result from different researchers looking at the same evidence.
Furthermore, the debates about differences of opinion on how to interpret evidence are good and healthy because these discussions further our understanding of complex issues. If we all agreed that something was true, then there would be no point in conducting research and writing about it. Indeed, if we all agreed about everything and had all of our questions answered as well as we thought possible, there would be no point to education at all!
Ultimately, there is no easy formula for evaluating the credibility and reliability of research. But there are some basic questions you should ask about your all of your evidence to ensure it is reliable and credible:
Who wrote or said it?
Is there an author named with the evidence?
If your evidence does not name the author, it might still be reliable, especially if you have confidence about where the evidence was published. However, most credible and reliable publications tell readers who wrote the articles they contain.
On Web pages and other Internet-based sources, it can sometimes be tricky to find the name of the Web page’s author. Many web sites don’t name an author, which, given the nature of the Web, should send up red flags for you as a researcher regarding the credibility of the evidence. But like print publications, more credible Web pages will include the name of the page’s writer. Be sure to look for the writer’s name throughout the particular page (including the bottom) and related pages within the Web site.
What are the qualifications of the author?
Does he or she seem to be an expert in the field?
Have he or she written about this topic before?
Are there other experiences that seem to uniquely qualify him or her as a reliable and credible source on this topic?
Many academic publications will give a lot of detail about their authors, including their degrees and academic training, the institution where they work (if they are a college professor or instructor), and other publications they have had in the past. Popular sources tend to include less information about their writers, though they too will often indicate in a byline (where the writer’s name is listed in a magazine or newspaper article) if the writer is a reporter, contributing editor, or editor for a particular subject.
Credible web sources will also describe the qualifications of the source’s author or authors. If you can find an author’s name on a Web site but you can’t find anything about their qualifications on their research subject, you should be suspicious about what that research has to say.
Have you come across the writer based on some of the other research you have done?
After you have conducted a bit of research on your topic, you might find yourself coming across the same authors writing similar articles in different publications. You might also find different publications referring to the author or her work, which would suggest that the author is indeed reliable and credible in her field. After all, if other articles and writers refer positively to a particular writer or her articles again and again, then it seems likely that the often-referred-to writer is credible.
Understanding and trusting the expertise of the author of your evidence is probably the most crucial test of credibility and reliability of that evidence.
Simply put, academics find evidence that comes from an author who is a credible expert to be much more persuasive than evidence that does not come from an expert.
For example, while my mom is a reliable source of information regarding many different topics, it would do you little good for me to interview her for an academic research project about the problems of over-fishing. Mind you, I value my mom’s thoughts and wisdom, and she might have some things to say about the effects of decreased catches of fish that I find insightful. However, because my mom doesn’t have any expertise about commercial fishing and because she doesn’t know anything more (or less) about it than most people, most of the readers of my research project won’t be persuaded by what she has to say.
On the other hand, my mother was a hospice work for many years, working with terminally ill patients and their families. If I were conducting research about the advantages and disadvantages of hospice care for terminally ill patients, my mom might be a very interesting and credible source.
What do you think motivated the writer?
Is the writer identified with a particular organization or group that might have a specific interest in the subject of the writing?
This can often be the source of conscious or unconscious bias. An obvious example: a writer who is identified as a member of the National Riflemen’s Association, which represents a variety of Americans particularly interested in protecting the right to own guns, will certainly have a different view on gun ownership than a member of The Center to Prevent Handgun Violence, an organization working to enact gun control legislation.
You need to be particularly careful with Web-based sources of research when considering the writer’s affiliation with different groups or organizations. There have been numerous incidents where Web page writers falsely claimed their Web pages were affiliated with particular groups or causes.
Does the writer identify himself or herself with an explicit political group or party?
Considering a writer’s politics is particularly important when thinking about the credibility of a Web site. Besides the ease with which a writer can misrepresent themselves or others, the low cost and wide reach of the Web has also made it an attractive forum for hate groups, terrorists, and other “fringe” political movements. This doesn’t automatically mean the information you find on reactionary or radical Web sites is wrong; however, writers with particularly strong and extreme politics frequently present information that is biased to the point of inaccuracy.
Of course, while it is important to consider why a writer wrote about her subject and to think about how her motivations impact how she wrote about his or her subject, having a particular bias or motivation doesn’t automatically lead to a lack of credibility or reliability.
Where was it published?
Was the piece of writing published in an academic or non-academic source? A book, a journal, a magazine, etc.? I’ve already discussed this a great deal in this chapter; generally speaking, academic sources are considered more credible than non-academic sources, and print-based sources are generally considered more credible than web-based sources.
But there are some more subtle tests of credibility and reliability concerning where a piece of research was published. For example, single-authored or co-authored scholarly books on a particular subject might be more regarded as more credible than a scholarly journal article because books go into much greater detail on topics than journal articles.
Are you familiar with the publication? If you are a new researcher to a particular field of study this can be a difficult question to answer since you might not have heard of some of the more well-known and credible publications known in that field. But once you get to know the field better (which will inevitably be the case as you conduct more research on your topic), chances are you will begin to realize certain publications are seen by experts in the field as more credible than others.
When was it written?
Last, but far from least, the date of publication can dramatically effect the credibility of your research. Obviously, this is especially important for date-sensitive research topics. If you were writing a research project about the Internet and the World Wide Web, chances are any research older than about 1990 or so would be of limited use since the Web literally did not exist before 1990.
But other potentially less obvious topics of research have date sensitive components to them. For example, if you were doing research on cigarette smoking or drunk driving, you would have to be careful about evaluating the credibility of research from the 1970s or 1960s or earlier since cultural “norms” in the United States for both smoking and drinking have changed a great deal.
Knowing (or rather, not knowing) the date of publication of a piece of research is yet another thing to be worried about when evaluating the credibility of Web-based sources. Many Web sites do not include any information about the date of publication or the date when the page was last updated. This means that you have no way of knowing when the information on that dateless page was published.
The date of publication is a key piece of information, the sort of thing that is always included in more print sources. Again, just because the date of publication or update is missing from a Web site does not automatically discount it as a credible source; however, it should make you suspicious.
Exercise 1.5
Working alone or collaboratively in small groups, consider a variety of different types of research—articles from scholarly and non-scholarly sources, newspaper articles, books, web sites, and other types of evidence. Using the criteria discussed here, how would you rate the quality and credibility of your research? Which of your sources seems the most reliable? Are there any pieces of evidence that, upon closer examination, do not seem credible or reliable?
Evidence Quality and Credibility Checklist
Who wrote or said it?
- The writer’s name
- Qualifications
- Expertise in the field
- Previous publications on the topic
- Unique experiences of the writer
Why did the source write or say it?
- Association with an organization or group
- The writer’s stated or implied politics
Where (what source) was it published?
- Academic/scholarly source versus non-academic/popular source
- Prior knowledge of publication
When was it published or said?
And when it comes to evidence from the ‘net and World Wide Web…
- It’s still important to know who wrote it, why you think they wrote it, where you found it online, and when was it published.
- If you don’t know the answers to the who/why/where/when questions, you should be skeptical of the evidence.
- Don’t be fooled by Web sites that “look” real, because…
- Anybody can publish information on the Web, no matter what that information is. Unlike most scholarly and many non-scholarly publications, Web writers don’t have to have the work reviewed by editors and publishers to reach an audience.
- The Internet and the World Wide Web are still good places to find research. You just have to be a bit more careful with them.