Finding and Evaluating Sources

Daniel Libertz

There’s so much information out there! When you’re starting any kind of research, it can be overwhelming just to think about where to start and how to move from one step to another: what to search for, where to search for it, what to do when you get a list of results, how to decide if a source is useful or credible, and so on.

This chapter offers some practical advice/suggestions for both finding and evaluating sources:

• Choosing search engines for academic and popular sources

• Generating keywords for relevant results

• Using Boolean operators for relevant results (i.e., words you can use to help narrow down search results)

• Looking at results, online or in print

• Using “lateral reading” and “frame checking” for evaluating sources

Search Engine Choices and Navigating Search Engines

Some search engines can find you anything. Some search engines are more targeted for specific kinds of information.

Websites like Google and Bing have a search bar where you can enter search terms and the results will be based on a combination of factors: the kinds of links people tend to click based on using similar words typed into the search bar, your own history browsing the internet, payment made to the company that produces the search engine that makes certain results appear higher in results rankings, and so on. Knowing that the results you receive are curated by a series of procedures automatically applied by proprietary software systems is part of developing what Abigail Bakke calls “algorithmic literacy.” Bakke explains that part of understanding the value and credibility of sources you find are not only the source itself but how you find it, “to take a step back and consider the processes…use[d] to locate…sources” (3).

Because free websites like Google and Bing are motivated by profit generated from advertising, the algorithms used to generate search results will always be incentivized by that profit motive. Sometimes those incentives lead to useful results, but not always, especially if other, less accurate results help to generate more advertising revenue. For example, some companies pay to have their results higher up—see “Google Ads”.

Free search engines are also not designed to find academic sources (i.e., sources that have been reviewed by academic experts in academic journals or books)—though there are some free search engines that do so, like Google Scholar. In contrast, search engines that are free for college students through the college library but not free for the general public are designed to return academic source results. Moreover, since there is no profit motive (especially none driven by ad revenue), the quality of results will also be generally better relative to search engines like Google, Bing, and other similar search engines.

So, if you want academic sources, Google or Bing aren’t going to be your best options. Just start right with the library search engine at https://library.baruch.cuny.edu/. The easiest way to use this search engine is to click “Define Your Search” and choose “Articles” or “Books + Articles + Videos.” It will also return magazine and newspaper articles, but you can use the checkmarks to the left of the results to uncheck any types of sources you don’t want.

If you want sources that are not academic, you can still use the library search engine to return sources like newspaper and magazine articles, but it might not return results from other types of sources like organization websites. You might also want sources that offer very general introductions to a topic, which academic sources do not always provide. In that case, using something like Google or Bing can be helpful. But, you should still consider your own algorithmic literacy here: be wary of any of the results that are returned at the top of the rankings and be careful to notice when results are explicitly noted as advertisements. Explore several pages of results and refine your keywords (see next section) to make sure you get high-quality results.

Keyword Generation and Boolean Operators for Relevant Results

Sometimes overusing a thesaurus can make for some bad writing (you don’t need to say, for example, that e.g., “From time to time, overdoing a lexicon can cause certain unpleasant prose”), but not with generating keywords. A keyword is not just any term you type into a search engine, but a term that you are specifically choosing because you think it will return the results you seek. So, for instance, if you want to find more information about how plastic trash in the water affects fish, you could use a combination of keywords: “fish”, “plastic.” Maybe you find nothing, but even if you find results, trying new keywords might reveal some other great results that wouldn’t turn up otherwise. You could try any combination of: “marine life,” “sea-life,” “pollution,” “microplastics”, etc. Or maybe you want to know more about the history of how religion has influenced art. Maybe you start with “art” and “religion,” but eventually get more specific: “literature,” “painting,” “sculpting.” Maybe you try words besides religion like: “spirituality,” “belief,” “faith,” “God,” and so on. Trying a bunch of options can give you a better chance of finding the best sources for your particular interests in your writing.

You can also use what are called “Boolean operators,” which are words and punctuation you can insert into search engines that can help you narrow results - like the word AND. Let’s think through some examples. If I search “fish” on Google (on December 26, 2023 on my personal computer browser), I get 1,500,000,000 results. If I search “fish” AND “plastic,” I get fewer (but still a lot!) of results (139,000,000 results). If I search “fish” AND “plastic” AND “pollution”, I get 32,100,000 results. Fewer results! And you’ll get (usually) more relevant sources filtered at the top of the results (of course, you won’t look at all 32,100,000 results). All of this makes it easier to sift through results.

The use of “AND” is one Boolean operator, but there are other options that can also make search results more accurate and manageable. You can also use “OR” for returning results. For instance, if you are researching the COVID-19 pandemic, you might want to use keywords like the following: “SARS-CoV-2” OR “COVID-19.” Perhaps sometimes some sources use one term or the other but you want results for both. Only results that contain one or the other term will be included in the list of results.

You can also use “NOT,” which can be useful when you are using a term that has two (or more) different meanings. For instance, perhaps you are researching grading practices in schools. The term “grading” is also a term used in engineering and construction about making a level surface. So, you could use keywords using “NOT” in this way: “grading” NOT “level.” Only results with grading but NOT the word “level” will be included.

The use of quotation marks is very useful when you want an exact phrasing. For instance, if you are researching social media, using quotations around “social media” will ensure that you will only get results that contain the words “social” and “media” when they are right next to each other. You won’t get any results that have the word “social” in one paragraph and then several paragraphs later the word “media”.

You can also use parentheses to help narrow results by combining Boolean operators, like this: (pollution or deforestation) AND “climate change.” Doing this returns documents containing: pollution and climate change; deforestation and climate change; pollution and deforestation and climate change; but does not return pollution or deforestation when climate change is not mentioned (thanks to Baruch student Maria Frants who showed me this!).

Not all search engines support all Boolean operators, but the use of AND and the use of quotations, at minimum, are usually supported.

Looking at Results and Evaluating Sources

Once you get a list of results, you could just click on the first two results and call it a day. But don’t do that! Especially if you are using a free search engine, many of the early results are there not because they are the best results but because they are advertisements or engineered to be there. Skim the titles, read the preview text, click on a bunch at a time, and skim through some of the ones you clicked. Apply these steps to the first several pages of results.

Once you have a few good candidates, try to evaluate them further. Just because something is relevant to your research question does not make it credible or of good quality. The sources could be inaccurate; they could contain facts but make exaggerated or faulty interpretations of those facts; they could be written by people not qualified to offer useful commentary on the subject, and so on. While it is impossible to assess any source as 100% credible and high quality, there are approaches you can use to help you determine better from worse sources.

One approach is lateral reading. This concept comes from Sam Wineberg and Sarah McGrew’s 2017 research that studied fact checkers’ evaluation processes. To read laterally, readers open new tabs in their browser to learn more about the author, the publication (e.g., newspaper, journal, book, website), the organization or people who fund the publication (e.g., donors, businesses, government agencies, universities), and other information that helps us to understand the writer’s motives and qualifications. You might look at “About Us” pages on websites, resumes or CVs of authors, and news articles or Wikipedia entries about authors, publications, and funders. People who read laterally also know search results are influenced by algorithms that determine how results are ordered. So, all of the skills you use in finding sources should also be used in reading laterally. Learning about the author, the publication, the publisher, the organization who funds the publication and publisher, the consensus on the topic from other experts, and other contextual information can help you decide on how qualified the writer and the publication are for offering useful knowledge about a topic.

Another approach is called frame checking. This concept comes from Dana Cloud’s contention that fact-checking can sometimes be misguided. Cloud essentially argues that spending a lot of time on the trees (facts) can miss the forest (values). Every piece of writing is framed in some way, meaning that writers make some information prominent instead of other information. Something can be true but be framed in multiple ways, and missing out on how a fact is framed can lead someone to get stuck on an issue because a writer and reader may be coming from places of different values. For example, two sources can both have the same valid interpretation of a monthly jobs report–that there has been an increase in jobs since the last month. One source might contain interviews, descriptions, and word choice that create a “healthy economy” frame. Another source might contain interviews, descriptions, and word choice that create a “failure to meet expectations” frame (e.g., projections might have predicted more jobs to have been created). A source framed in a way that is at odds with the kinds of research questions you want to address may not be a source that will be helpful to you. Before offering an example, let’s go through the steps of frame checking (Cloud, 59-64):

1. Figure out what stasis the source is operating in. A stasis refers to questions or issues about a topic that all parties agree or disagree about; see Blankenship, “Stasis Theory” in chapter 5):

a. Question of fact or conjecture: Does it exist? Did it happen? Who did it?

b. Question of definition: How can the act or event be defined?

c. Question of quality: What is the character of the act: right or wrong, good or bad? What characteristics might make us evaluate it differently?

d. Question of jurisdiction: Who has authority over this action? What should be done about it?

2. Determine where the writer is trying to get you to focus. Ask questions like: where is my attention directed? How? What images or word choices make me think or feel a certain way?

3. Think about what is missing. Ask questions like: what is left out? Are there other perspectives that are not present here?

4. Add up the answers from the first three steps: what is the story here? What would you call this kind of story? In other words, what would you “name” this frame(s) operating in the source?

For example, let’s “frame check” this poster about libraries:

For questions of stasis, a reasonable one to choose here would be of quality. Are libraries good or bad? For what purpose? Here, the poster is making a direct association between “greater knowledge” and “library use,” with the implication that viewers of the image will agree that having greater knowledge is a good thing. Our attention is directed to the words in larger font and to the images: a globe, representing being “worldly”, a pair of glasses, associated with being well-read, and a graduation cap, associated with being knowledgeable. Another reasonable answer to the question of stasis could be definition, as in what are libraries (i.e., places to be more knowledgeable). We could also ask about what is missing here. Do libraries only share knowledge? Are they places for other things, like entertainment or community events? What about the process of becoming knowledgeable? Should we consider the requisite skills someone might need to become knowledgeable using the library’s resources? Why is that not brought up here?

Putting this all together, what frames could we discover here? Here are some options: the “autodidact frame” (a library is a place for self-taught people), the “public resource” frame (libraries are public goods that we should take advantage of more often), the “cosmopolitan” frame (libraries are places for people who want to know many subjects), or the “intellectual” or even “snobby” frame (i.e., libraries are places for people who see themselves as smart). This frame checking work can help you figure out what perspective this source is coming from and thus help you decide on its usefulness. For instance, naming the frames involved might lead you to believe that this promotes the idea of libraries as worthy of people who see themselves as intelligent. It could be a good example to use if you were writing about cultural understandings of the purposes of libraries or could lead you to a research question on why libraries in certain time periods and locations might promote some values (individualistic understandings of intelligence) at the expense of other values (community building).

Getting a sense of the frames operating in the source can help you think about the source’s motive and what might be missing from the source, leading to evaluations of the source and helping to point you to new questions and sources. For example, a source stuck in the question of jurisdiction about social media for teenagers might not be valuable to you if you care only about the effects of social media on maintaining healthy relationships. Or, a source that examines the closure of a department store in a poor neighborhood that only gets perspectives from the department store and from a local government official is missing perspectives from people who live in the neighborhood. If you care about effects on people who live in the neighborhood, this source may not be helpful and you might want to seek out other sources. A source that talks about exercise that has descriptions and images of young women who all have the same body type (e.g., skinny) may suggest an understanding of what it means to be healthy that is at odds with your research questions. Going through each step of frame checking can help you start to see how the source is framed in ways that might help you question an author’s credibility or think differently about how useful the source may be for you and what other sources you may want to find in addition or instead.

Conclusion

Finding and evaluating sources takes just as much analytical skill as the kind of analysis that you will do within the writing of any assignment. You have to pay critical attention to the search engine you choose, how you use the search engine, how you generate keywords and use Boolean operators, the kinds of results you received, how you adjust keywords and Boolean operators to produce more relevant results, how you look through lists of results to select possible sources to use, and how you evaluate sources through tactics like lateral reading and frame checking. It takes a lot of brain power. But it pays off - you can be more confident in the information you are using to enrich your writing.

Works Cited

Bakke, Abigail. “Everyday Googling: Results of an Observational Study and Applications for Teaching Algorithmic Literacy.” Computers and Composition, vol. 57, 2020, pp. 1-16.

“Be Just a Google Search Away.” Google Ads, https://ads.google.com/intl/en_us/home/campaigns/search-ads/, Accessed 26 December 2023.