University of Kwazulu-Natal

Google S Deepmind Introduces Ai System Outperforming Human Fact Checke

Bonisiwe Shabane
-
google s deepmind introduces ai system outperforming human fact checke

Google DeepMind's latest research on large language models (LLMs) provides compelling evidence that these AI systems can exceed human performance when it comes to fact-checking long-form content. The findings, detailed in a new paper, mark a significant milestone in the development of more truthful and reliable AI. The study introduces LongFact, a benchmark dataset comprising thousands of fact-seeking questions across 38 topics, generated using GPT-4. To evaluate the factual accuracy of LLM responses to these questions, the researchers propose the Search-Augmented Factuality Evaluator (SAFE). This method uses an LLM to break down a long-form response into individual facts, queries Google Search to find supporting evidence for each fact, and determines the overall factuality of the response through multi-step... The researchers also propose extending the F1 score as an aggregated metric for long-form factuality.

This metric, called F1@K, balances the percentage of supported facts in a response (precision) with the percentage of provided facts relative to a hyperparameter K, which represents a user's preferred response length (recall). While the study demonstrates the potential of LLMs as highly capable fact-checkers, the authors acknowledge some limitations. SAFE relies on the underlying LLM's capabilities and the comprehensiveness of Google Search results. Additionally, the proposed F1@K metric assumes no repetition of facts in the model's response. Despite these caveats, the research presents a promising step towards more truthful AI systems. As LLMs continue to improve, their ability to assess and ensure the factual accuracy of generated text could have far-reaching implications for combating misinformation and increasing trust in AI applications.

Google DeepMind’s ‘Superhuman’ AI System is making waves in the field of fact-checking, cost efficiency, and accuracy. In a recent study, researchers from DeepMind found that their artificial intelligence system, known as SAFE (Search-Augmented Factuality Evaluator), outperformed human fact-checkers when evaluating the accuracy of information generated by large language models. The study, titled “Long-form factuality in large language models,” introduces SAFE as a method that uses a large language model to break down generated text into individual facts. It then uses Google Search results to determine the accuracy of each claim. The researchers compared SAFE’s assessments with those of human annotators on a dataset of 16,000 facts and found that SAFE’s judgments matched the human ratings 72% of the time. Even more impressively, when there were disagreements between SAFE and human raters, SAFE’s judgment was correct in 76% of cases.

While the researchers claim that LLM agents can achieve “superhuman” rating performance, some experts are questioning the definition of “superhuman” in this context. AI researcher Gary Marcus suggests that “superhuman” may simply mean better than an underpaid crowd worker, rather than a true expert fact checker. Marcus argues that benchmarking SAFE against expert human fact-checkers is crucial to truly demonstrate its superhuman performance. One clear advantage of SAFE is its cost-efficiency. The researchers found that using the AI system was about 20 times cheaper than employing human fact-checkers. As the volume of information generated by language models continues to increase, having an economical and scalable way to verify claims becomes increasingly vital.

The DeepMind team also used SAFE to evaluate the factual accuracy of 13 top language models across four families. They found that larger models generally produced fewer factual errors. However, even the best-performing models still generated a significant number of false claims. This highlights the risks of relying too heavily on language models that can fluently express inaccurate information. Automatic fact-checking tools like SAFE could play a key role in mitigating these risks. In an era where misinformation can spread with the speed and reach of the internet, a groundbreaking study from Google’s DeepMind offers a promising solution.

The study, recently published on the arXiv pre-print server under the title “Long-form factuality in large language models,” reveals that an artificial intelligence system, known as the Search-Augmented Factuality Evaluator (SAFE), has demonstrated the... At the core of this research is the development of SAFE, a method that leverages a large language model to dissect generated text into discrete facts. These facts are then verified using Google Search results to evaluate the truthfulness of each statement. The DeepMind team eloquently explains that “SAFE employs an LLM to segment a long-form response into individual facts and assesses the veracity of each through a multi-step reasoning process. This involves initiating search queries on Google Search and determining if a fact is corroborated by the search outcomes.” The research team conducted rigorous comparisons between SAFE and human annotators across a dataset comprising approximately 16,000 facts.

Results showed that SAFE’s evaluations were in line with those of human raters 72% of the time. More intriguingly, in instances of disagreement between SAFE and the human evaluators (a sample of 100 cases), SAFE’s decisions were deemed accurate in 76% of those cases. This led the researchers to claim that “LLM agents can achieve superhuman rating performance.” However, the use of the term “superhuman” has sparked debate within the AI community. Gary Marcus, a renowned AI researcher and critic of inflated claims, suggested that this designation might be misleading, equating the AI’s performance more to that of an underpaid crowd worker than a skilled human... Marcus’s critique underscores a crucial aspect of the study: the need for benchmarking SAFE against expert human fact-checkers to truly validate claims of superhuman performance. The qualifications, compensation, and methodologies of the human raters are fundamental for a comprehensive understanding of the results.

A groundbreaking study conducted by Google's DeepMind research unit unveils an astonishing finding: an artificial intelligence system surpasses human fact-checkers in assessing the accuracy of information produced by extensive language models. Published on the pre-print server arXiv under the title "Long-form factuality in large language models," the paper introduces a revolutionary approach named Search-Augmented Factuality Evaluator (SAFE). Leveraging a vast language model, SAFE dissects generated text into discrete facts and harnesses Google Search results to ascertain the veracity of each assertion. This breakthrough heralds a new era in fact-checking, demonstrating the remarkable capabilities of AI in enhancing accuracy and efficiency in information evaluation. The 'superhuman' performance of AI sparks a fervent debate In a head-to-head comparison, researchers set SAFE against human annotators using a dataset containing approximately 16,000 facts.

Remarkably, SAFE's evaluations aligned with human ratings 72% of the time. Notably, in 100 instances of disagreement between SAFE and human raters, SAFE proved correct 76% of the time. Despite the paper's claim that "LLM agents can achieve superhuman rating performance," some experts are scrutinizing the definition of "superhuman" in this context. Gary Marcus, a prominent AI researcher known for his skepticism of exaggerated claims, took to Twitter to suggest that the term "superhuman" in this context might simply imply outperforming underpaid crowd workers rather than... He argued that this characterization could be misleading, akin to labeling 1985 chess software as "superhuman." Marcus highlights a crucial point: for SAFE to genuinely exhibit superhuman capabilities, it must be compared against highly... Understanding the qualifications, compensation, and methodology of the human raters is essential for accurately interpreting the study's findings.

How it saves Cost and Benchmarks Leading Models In a groundbreaking study, Google’s DeepMind research unit has unveiled an artificial intelligence system that outperforms human fact-checkers in assessing the accuracy of information produced by large language models. This innovative system, known as the Search-Augmented Factuality Evaluator (SAFE), leverages a multi-step process to analyze text and verify claims using Google Search results. In a recent study titled “Long-form factuality in large language models,” published on arXiv, SAFE showcased remarkable accuracy, aligning with human ratings 72% of the time and outperforming human judgment in 76% of disagreements. Nevertheless, the concept of “superhuman” performance is sparking lively discussions, with some experts debating the comparison against crowdworkers instead of expert fact-checkers. One of SAFE’s significant advantages is its cost-effectiveness.

The study revealed that utilizing SAFE was approximately 20 times cheaper than employing human fact-checkers. With the exponential growth of information generated by language models, having an affordable and scalable method for verifying claims becomes increasingly crucial. The DeepMind team utilized SAFE to evaluate the factual accuracy of 13 leading language models across four families, including Gemini, GPT, Claude, and PaLM-2, on the LongFact benchmark. Larger models generally exhibited fewer factual errors, yet even top-performing models still generated significant false claims. This emphasizes the importance of automatic fact-checking tools in mitigating the risks associated with misinformation. While the SAFE code and LongFact dataset have been made available for scrutiny on GitHub, further transparency is necessary regarding the human baselines used in the study.

Understanding the qualifications and processes of crowdworkers is essential for accurately assessing SAFE’s capabilities.

People Also Search

Google DeepMind's Latest Research On Large Language Models (LLMs) Provides

Google DeepMind's latest research on large language models (LLMs) provides compelling evidence that these AI systems can exceed human performance when it comes to fact-checking long-form content. The findings, detailed in a new paper, mark a significant milestone in the development of more truthful and reliable AI. The study introduces LongFact, a benchmark dataset comprising thousands of fact-see...

This Metric, Called F1@K, Balances The Percentage Of Supported Facts

This metric, called F1@K, balances the percentage of supported facts in a response (precision) with the percentage of provided facts relative to a hyperparameter K, which represents a user's preferred response length (recall). While the study demonstrates the potential of LLMs as highly capable fact-checkers, the authors acknowledge some limitations. SAFE relies on the underlying LLM's capabilitie...

Google DeepMind’s ‘Superhuman’ AI System Is Making Waves In The

Google DeepMind’s ‘Superhuman’ AI System is making waves in the field of fact-checking, cost efficiency, and accuracy. In a recent study, researchers from DeepMind found that their artificial intelligence system, known as SAFE (Search-Augmented Factuality Evaluator), outperformed human fact-checkers when evaluating the accuracy of information generated by large language models. The study, titled “...

While The Researchers Claim That LLM Agents Can Achieve “superhuman”

While the researchers claim that LLM agents can achieve “superhuman” rating performance, some experts are questioning the definition of “superhuman” in this context. AI researcher Gary Marcus suggests that “superhuman” may simply mean better than an underpaid crowd worker, rather than a true expert fact checker. Marcus argues that benchmarking SAFE against expert human fact-checkers is crucial to ...

The DeepMind Team Also Used SAFE To Evaluate The Factual

The DeepMind team also used SAFE to evaluate the factual accuracy of 13 top language models across four families. They found that larger models generally produced fewer factual errors. However, even the best-performing models still generated a significant number of false claims. This highlights the risks of relying too heavily on language models that can fluently express inaccurate information. Au...