University of Kwazulu-Natal

Fact Checking Fact Checkers A Data Driven Approach

Bonisiwe Shabane
-
fact checking fact checkers a data driven approach

This study examined four fact checkers (Snopes, PolitiFact, Logically, and the Australian Associated Press FactCheck) using a data-driven approach. First, we scraped 22,349 fact-checking articles from Snopes and PolitiFact and compared their results and agreement on verdicts. Generally, the two fact checkers agreed with each other, with only one conflicting verdict among 749 matching claims after adjusting minor rating differences. Next, we assessed 1,820 fact-checking articles from Logically and the Australian Associated Press FactCheck and highlighted the differences in their fact-checking behaviors. Major events like the COVID-19 pandemic and the presidential election drove increased the frequency of fact-checking, with notable variations in ratings and authors across fact checkers. College of Information Sciences and Technology, The Pennsylvania State University, USA

College of Information Sciences and Technology, The Pennsylvania State University, USA College of Information Sciences and Technology, The Pennsylvania State University, USA College of Information Sciences and Technology, The Pennsylvania State University, USA If politicians utter dubious statements on the campaign trail and fact-checkers are around to hear them, will those fact-checkers rate the statements with agreeing skepticism? If the checkers are Snopes and PolitiFact, and the claim is similar, that answer is nearly always yes, Penn State University researchers found. This consistency helps build public trust in fact-checking and fact-checkers, the researchers said.

“‘Fact-checking’ fact checkers: A data-driven approach,” a 22-page October research article from the Harvard Kennedy School Misinformation Review, examined practices of U.S. fact-checking organizations Snopes, PolitiFact and Logically, along with The Australian Associated Press. Sian Lee, Aiping Xiong, Harseung Seo and Dongwon Lee of Penn State University’s College of Information Sciences and Technology did the peer-reviewed research. The Penn State researchers found U.S. fact-checking spikes during major news events. In recent years, that was during the COVID-19 pandemic and the 2020 presidential election.

Further, the researchers said, misinformation’s spread can mislead and harm people and society. Misinformation and disinformation spread rapidly on social media, threatening public discourse, democratic processes, and social cohesion. One promising strategy to address these challenges is to evaluate the trustworthiness of entire domains (source websites) as a proxy for content credibility. This approach demands methods that are both scalable and data-driven. However, current solutions like NewsGuard and MBFC rely on expert assessments, cover only a limited number of domains, and often require paid subscriptions. These constraints limit their usefulness for large-scale research.This study introduces a machine-learning-based system designed to assess the quality and trustworthiness of websites.

We propose a data-driven approach that leverages a large dataset of expert-rated domains to predict credibility scores for previously unseen domains using domain-level features. Our supervised regression model achieves moderate performance, with a mean absolute error of 0.12. Using feature importance analysis, we found that PageRank-based features provided the greatest reduction in prediction error, confirming that link-based indicators play a central role in domain trustworthiness. This highlights the importance of highly referenced domains in reliable news dissemination. This approach can also help fact-checkers and social media platforms refine their credibility assessment strategies.The solution’s scalable design accommodates the continuously evolving nature of online content, ensuring that evaluations remain timely and relevant. The framework enables continuous assessment of thousands of domains with minimal manual effort.

This capability allows stakeholders (social media platforms, media monitoring organizations, content moderators, and researchers) to allocate resources more efficiently, prioritize verification efforts, and reduce exposure to questionable sources. Ultimately, this facilitates a more proactive and effective response to misinformation while also supporting robust public discourse and informed decision-making. With so much information at our fingertips, the need for accurate and reliable data has never been greater. Relying on data-driven fact-checking can help us better understand what’s real and reputable over what’s false and misleading, but in addition to a technology-based approach, we also need to develop our own critical thinking... But as the case with any kind of skill development, where do you even begin? In this article, we’ll take a closer look at what data-driven fact-checking actually means, how it’s implemented and how you can develop your own critical thinking skills alongside the developments in technology to accurately...

With the rise of fake news, misinformation and information overload, it’s easy to turn passive and just accept whatever comes your way as fact. It’s also a very risky strategy that if left unchecked, can have real-world consequences. By developing data-driven fact-checking skills, you empower yourself to understand truth from fiction and make more informed decisions by pulling from the most accurate information around you. Developing your data-driven fact-checking skills isn’t something that can happen overnight, but the more you familiarize yourself with the process and use it when evaluating the credibility of an article, a picture or a... That means: Not all sources are created equal.

You often can’t tell how credible a specific source is just by looking at the title or the author. Take a deeper look at the author’s credentials and any affiliations that they’re part of which might reveal human biases within the article. Research output: Contribution to journal › Article › peer-review This study examined four fact checkers (Snopes, PolitiFact, Logically, and the Australian Associated Press FactCheck) using a data-driven approach. First, we scraped 22,349 fact-checking articles from Snopes and PolitiFact and compared their results and agreement on verdicts. Generally, the two fact checkers agreed with each other, with only one conflicting verdict among 749 matching claims after adjusting minor rating differences.

Next, we assessed 1,820 fact-checking articles from Logically and the Australian Associated Press FactCheck and highlighted the differences in their fact-checking behaviors. Major events like the COVID-19 pandemic and the presidential election drove increased the frequency of fact-checking, with notable variations in ratings and authors across fact checkers. Research output: Contribution to journal › Article › peer-review N1 - Publisher Copyright: © 2023, Harvard Kennedy School. All rights reserved. N2 - This study examined four fact checkers (Snopes, PolitiFact, Logically, and the Australian Associated Press FactCheck) using a data-driven approach.

First, we scraped 22,349 fact-checking articles from Snopes and PolitiFact and compared their results and agreement on verdicts. Generally, the two fact checkers agreed with each other, with only one conflicting verdict among 749 matching claims after adjusting minor rating differences. Next, we assessed 1,820 fact-checking articles from Logically and the Australian Associated Press FactCheck and highlighted the differences in their fact-checking behaviors. Major events like the COVID-19 pandemic and the presidential election drove increased the frequency of fact-checking, with notable variations in ratings and authors across fact checkers. The Evolution and Future of Fact-Checking in a Misinformation-Ridden World A decade ago, the fight against misinformation coalesced around a simple yet powerful idea: establish a global network of fact-checkers to debunk false claims and promote accurate information.

This nascent movement, driven by organizations like the International Fact-Checking Network (IFCN), gained traction, earning the support of major platforms and becoming an integral part of public discourse. However, the landscape shifted dramatically with the advent of the COVID-19 pandemic and the contentious 2020 US presidential election. These events thrust fact-checking into the global spotlight, simultaneously increasing its importance and exposing its vulnerabilities. While platforms like Meta initially invested heavily in these efforts, the underlying model struggled to keep pace with the evolving nature of online misinformation. The limitations of the traditional fact-checking model became increasingly apparent. While effective in debunking individual claims, it proved too slow and resource-intensive to combat the sheer volume and velocity of misinformation spreading online.

Furthermore, fact-checkers faced increasing harassment and accusations of censorship, highlighting the precarious position they occupied in the online information ecosystem. The reliance on platform funding also created a dependency that left fact-checking organizations vulnerable to shifts in platform priorities, a vulnerability starkly revealed by Meta’s recent decision to discontinue its third-party fact-checking program in... This move signaled a broader industry trend towards crowdsourced solutions like X’s (formerly Twitter’s) Community Notes, shifting the responsibility of verification away from professional journalists. While crowdsourcing offers scalability, it also presents significant challenges. Community-based systems are susceptible to manipulation, brigading, and the dominance of popular narratives, even if those narratives are factually incorrect. This raises serious concerns about the reliability of such systems, particularly when dealing with critical issues like public health, elections, and security.

The expertise and methodological rigor of professional fact-checkers are essential components of a robust verification system, and cannot be easily replicated by crowdsourced opinions. The focus must shift from simply gathering diverse perspectives to ensuring the accuracy and credibility of the information itself. The current model of fact-checking, largely reliant on manual processes and journalistic investigations, is ill-equipped to handle the speed and scale of today’s misinformation ecosystem. The rise of AI-generated content, sophisticated disinformation networks, and virality-driven platforms demands a more technologically advanced approach. Furthermore, the financial sustainability of fact-checking organizations remains a significant concern. The dependence on grant funding and short-term platform partnerships creates instability and hinders long-term planning.

Meta’s decision to withdraw funding underscores the inherent fragility of this model and the need for a more sustainable approach. In the rapid-fire world of internet news, breaking news reporters face the dual challenge of disseminating information quickly while ensuring every detail is correct. Today, business intelligence and data analytics have become indispensable tools for fact-checking, providing reporters with the power to verify claims rapidly and efficiently. With the complexity of modern news and the pressures of an always-on news cycle, data-driven analysis has emerged as a cornerstone in maintaining journalistic integrity. This article explores how advanced data techniques and systems like DataCalculus empower reporters to cross-check facts, analyze trends, and ultimately rebuild trust in the era of digital news. The landscape of journalism has shifted dramatically over the past decade.

The traditional newsroom, once heavily reliant on intuition and eyewitness reports, now finds itself in an era of data abundance. Breaking news reporters are no longer just storytellers; they have become data analysts, piecing together narratives with the aid of real-time data streams. This integral role of data analytics ensures that every news story is cross-verified against multiple sources and historical trends. For those looking to transform raw information into powerful insights, platforms like DataCalculus offer intuitive solutions that turn diverse data sets into actionable reports with just one click. Today’s news cycle is relentless. With news breaking every minute on social media and online platforms, reporters are expected to verify stories as they happen.

This pressure often creates a gap between speed and accuracy. Several challenges face fact-checkers: the surge of misinformation and unverified claims, the time-consuming process of manual verification, and the difficulty of connecting disparate data points from various sources. The interplay between these challenges forces journalists to rely on well-structured data frameworks and algorithms to sift through the noise. Emerging techniques in business intelligence allow reporters to quickly aggregate data from legislative records, financial filings, social media feeds, and other digital repositories. Additionally, robust platforms like Data Dictionary let reporters explore datasets at the granular level, ensuring that no piece of information is overlooked. Using advanced business intelligence and data analytics as part of a fact-checking strategy provides an immense advantage.

Data-driven approaches help in systematically verifying the authenticity of every detail, where every piece of evidence is supported by numbers and trends. By integrating quantitative methods, newsrooms can reduce errors and build confidence among readers. This integration ranges from scrutinizing social media posts to verifying the authenticity of official statements using complex datasets.

People Also Search

This Study Examined Four Fact Checkers (Snopes, PolitiFact, Logically, And

This study examined four fact checkers (Snopes, PolitiFact, Logically, and the Australian Associated Press FactCheck) using a data-driven approach. First, we scraped 22,349 fact-checking articles from Snopes and PolitiFact and compared their results and agreement on verdicts. Generally, the two fact checkers agreed with each other, with only one conflicting verdict among 749 matching claims after ...

College Of Information Sciences And Technology, The Pennsylvania State University,

College of Information Sciences and Technology, The Pennsylvania State University, USA College of Information Sciences and Technology, The Pennsylvania State University, USA College of Information Sciences and Technology, The Pennsylvania State University, USA If politicians utter dubious statements on the campaign trail and fact-checkers are around to hear them, will those fact-checkers rate the ...

“‘Fact-checking’ Fact Checkers: A Data-driven Approach,” A 22-page October Research

“‘Fact-checking’ fact checkers: A data-driven approach,” a 22-page October research article from the Harvard Kennedy School Misinformation Review, examined practices of U.S. fact-checking organizations Snopes, PolitiFact and Logically, along with The Australian Associated Press. Sian Lee, Aiping Xiong, Harseung Seo and Dongwon Lee of Penn State University’s College of Information Sciences and Tech...

Further, The Researchers Said, Misinformation’s Spread Can Mislead And Harm

Further, the researchers said, misinformation’s spread can mislead and harm people and society. Misinformation and disinformation spread rapidly on social media, threatening public discourse, democratic processes, and social cohesion. One promising strategy to address these challenges is to evaluate the trustworthiness of entire domains (source websites) as a proxy for content credibility. This ap...

We Propose A Data-driven Approach That Leverages A Large Dataset

We propose a data-driven approach that leverages a large dataset of expert-rated domains to predict credibility scores for previously unseen domains using domain-level features. Our supervised regression model achieves moderate performance, with a mean absolute error of 0.12. Using feature importance analysis, we found that PageRank-based features provided the greatest reduction in prediction erro...