(Image by Tero Vesalainen on Shutterstock)

VIENNA — In today’s digital world, determining what information to trust online has become increasingly challenging. According to researchers, the total volume of false news remains relatively small—accounting for just 0.15% of all news consumption. Still, its impact on public discourse and social division continues to raise concerns. A new study published in the Journal of Quantitative Description: Digital Media examines one of the most prominent tools researchers use to identify untrustworthy news sources: NewsGuard.

Since its launch in 2019, NewsGuard has emerged as a widely used database for researchers studying online misinformation, with more than 12,000 websites having been evaluated by the company. The service employs professional journalists and editors who rate news sources based on nine journalistic criteria, generating trustworthiness scores from 0 to 100.

While initially focused on U.S. media outlets, NewsGuard has expanded to evaluate sources across multiple countries, including the United States, United Kingdom, Germany, France, Italy, Canada, Austria, Australia, and New Zealand.

A team of researchers led by Jula Luehring at the University of Vienna and the Complexity Science Hub conducted an extensive analysis of NewsGuard’s database from 2019 to 2024, examining how its ratings have evolved over time and how effectively it serves as a tool for studying misinformation across different countries. Their findings reveal both NewsGuard’s reliability and the potential pitfalls of oversimplifying its nuanced ratings system.

One of the study’s most striking findings concerns U.S.-based news sources, which consistently received lower trustworthiness scores compared to outlets from other nations. This pattern persisted across NewsGuard’s entire rating history, suggesting fundamental differences in journalistic standards and practices between American media and their international counterparts.

“Smaller (hyper-)partisan sources tend to lack editorial practices and transparency measures. Since these are key criteria for NewsGuard, judging based on NewsGuard criteria results in an objectively lower overall trustworthiness,” Luehring explains in a statement.

The research validates earlier findings from a 2023 study published in PNAS Nexus by Lin et al., which demonstrated strong alignment between NewsGuard’s ratings and those of other fact-checking organizations. The new study found that NewsGuard’s ratings have remained relatively stable since 2022, particularly for sources in the United States, France, Italy, Germany, and Canada. When changes in ratings did occur, they were often due to updates in transparency measures or editorial policies rather than dramatic shifts in content quality.

Another intriguing finding relates to political orientation. While only about a third of sources are labeled for political leaning, right-leaning outlets tend to receive lower trustworthiness scores than left-leaning ones. The average score for right-leaning sources was 26.4 out of 100, compared to 60.6 for left-leaning sources. However, this pattern should be interpreted cautiously due to the limited number of politically labeled sources

“It seems unlikely that NewsGuard has an inherent bias against conservative sources, both in selecting and giving them lower ratings,” suggests Luehring. “Instead, the U.S. media system is flooded with right-wing sources that tend to not adhere to professional editorial practices.”

The researchers discovered that untrustworthy sources aren’t confined to fringe topics or conspiracy theories. In fact, many low-rated outlets cover mainstream topics like political news, local news, and health information. This finding challenges the common assumption that unreliable information is primarily limited to extreme or sensational content.

NewsGuard’s selection process relies heavily on web traffic data, ensuring comprehensive coverage of widely-read sources. “We manually checked sources in the US, UK, and Germany, and found that the database misses almost no news sites with substantial traffic. The sites it does miss are not systematically biased towards any political ideology,” Luehring explains.

A crucial caution emerged from the research regarding how NewsGuard’s ratings are used in academic studies. The researchers found that using a simple binary classification of “trustworthy” versus “untrustworthy” based on a cutoff score of 60 can lead to significant variations in research results over time. They strongly recommend using NewsGuard’s full 100-point scale to capture the nuanced reality of source reliability.

“Tools to measure the trustworthiness of sources—like the NewsGuard database—are therefore crucial to quantify the spread of untrustworthy information in online environments,” adds co-author Jana Lasser, a professor at the University of Graz and associate faculty member at CSH. “However, researchers must use these tools thoughtfully to avoid oversimplifying complex media landscapes.”