Wikidata:ProVe

ProVe: Automated PROvenance VErification of Knowledge Graphs against Textual Sources

The ProVe UI for Wikidata reference verification

Introduction

ProVe is a tool for helping editors improve the references of Wikidata items. In general, Wikidata item statements should be verifiable are referenced by a source (e.g. a book, scientific paper, etc). However, a large number of Wikidata item statements do not provide any reference and, even when they do, such references are of varying quality which tends to decay over time. ProVe helps editors to keep references up-to-date by providing information about the quality of the references of Wikidata items, using advanced AI techniques to check references against claims. In its user interface, ProVe introduces a reference quality indicator (located at the top left of the page) with a ProVe score between -1 and 1 (see below). An expandable table gives an overview of the quality of references for each of the item's statements. These are categorised based on one of the four following instances stances with respect to the claim: missing, supportive, refuting, or inconclusive. See, for example the analysis of the references of the item Q44 (Beer), with the tool installed as per instructions below, for a demonstration.

ProVe's reference quality score represents the overall level of support (from -1 to 1) for a Wikidata entity's references, with 1 indicating strongest support and -1 representing strongest refutation. Values hovering around 0 indicate a low level of references overall, or a balanced level of support and refutation. Visually, a battery status icon represents this with positive values in green and negative values in red.

Behind the scenes, ProVe uses an automated pipeline to analyse the level of support of a Wikidata statement (triple) by the corresponding documents used as references. This involves several steps: verbalisation of the triple into a sentence, retrieval of the documents references as source, extraction and selection of of the most relevant passages within the document texts, actual claim verification using the passages, and finally aggregation of intermediate results and computation of an overall score giving the corresponding support stance and level.

ProVe aims to assist authors in identifying areas where reference quality can be improved and to inform data consumers about the level of support provided by the references of a specific Wikidata item.

ProVe is designed to access external web pages via reference URLs to verify the appropriateness of their content. It uses a Chrome web crawler driver to access non-Wikimedia sites without archiving the accessed pages. ProVe automatically scans documents searching for relevant parts (segments) which are later used for text entailment.

IMPORTANT NOTE: To compute scores, the ProVe widget communicates with a non-wikimedia site/server for processing claims and references. At the moment, this cannot be avoided as ProVe needs to employ language models and high performance computing which are not available for tools within Wikimedia Foundation servers. We are working towards realising an architecture that overcomes this, so that we can provide a Wikimedia-site only version.

User Guide

ProVe Installation Guide

ProVe adds visual components to each Wikidata entity page, helping you assess the quality of references. Follow these steps to enable ProVe on your Wikidata account.

1. Log into your Wikidata account: Make sure you're logged in to your Wikidata account.

2. Create or access your common.js file: Visit your common.js to open or create your common.js file. This will take you directly to your file, allowing you to edit it.

3. Add the ProVe script to your common.js: In the common.js file, paste the following line of code:

importScript('User:1hangzhao/ProVe.js');

// IMPORTANT NOTE: To compute scores, the ProVe widget communicates with a non-Wikimedia site/server for processing claims and references. This cannot be avoided as ProVe needs the support of LLMs/heavy processing. We are working towards having a Wikimedia-site-only version.

4. Save your changes: After pasting the script, click the "Publish changes" button to save the file.

5. Refresh the page: Once you’ve saved the changes, refresh the Wikidata page you're viewing. You should now see ProVe components activated and ready for use.

6. Add your name to the user list (optional). After installing, we would be grateful if you write down your username here. This will help us show the value of the work we are doing. Thanks!

How to Use ProVe

1. Visit an item of interest, for example, Q44 (Beer)
2. At the top right corner, ProVe will show a Reference score summarising the quality of references for the item (closer to 0 means lower quality, and closer to 1 means higher quality). You can click on "Recompute" if you want to recompute this score after making changes to references (it may take some time).
3. At the top left corner, ProVe will show a "Check the quality of references" button. By clicking on it, ProVe will display a summary table with the following:
   1. The number of statements with and without references. Statements without references are a priority, so you'll be encouraged (by a "Start adding references to unreferenced statements" button) to add references to the first unsupported statement.
   2. "Red" statements with references to sources whose authoritativeness could not be checked
   3. "Yellow" statements with references that are possibly not relevant to backing the statement
   4. "Green" statements with references that most likely support the statement

Users can click on the "Link" anchor to any statement to scroll automatically to the reference in question and start editing it if needed.

(Re)Computing reference quality

Computing ProVe scores and assessing possible issues with references is an expensive process. Some items, instead of showing a score and the reference quality table directly, might show a "Prioritise" button, meaning that ProVe does not have a pre-computed score/reference table yet for that item. If you are interested in that item, you can click that button and the item will be given priority in the queue. Usually it only takes a couple of minutes to show updated scores and the table of references (simply refresh the page after a few minutes).

How ProVe Works

Reference Score Calculation

An item in Wikidata can be associated with several "properties", resulting in corresponding claims. These are triples (subject, predicate, object) describing a property of the item. A claim needs to provide a reference supporting it. This reference can be internal, i.e., pointing to a Wikidata object, or external, i.e., a link to a document outside Wikidata. ProVe does not verify internal references. For external references, ProVe provides a ProVe Score (PS) which is calculated according to how well supported the corresponding pairs of (claim,reference) are.

The ProVe Score for an item $i$ is calculated using the following formula:

$PS(i)=\frac{1}{n}\sum _{j=1}^{n}labe{l}_j$

In the formula above, $labe{l}_i$ is a value in $\{-1,0,1\}$ corresponding to the evaluation of the support for the $j$-th (claim,reference) pair of the Wikidata item $i$. $labe{l}_i$ is 1, if the reference is found to support the claim; $labe{l}_i$ is -1, if the reference is found to refute the claim; otherwise, $labe{l}_i$ is 0 (this includes inconclusive results, and the stance for irretrievable references).

Steps:

1. Retrieve Results: Collect the results for the target item.

2. Classify Results: Each result is classified into one of three categories:

1. SUPPORTS: The item supports the target claim.
2. REFUTES: The item refutes the target claim.
3. NOT ENOUGH INFO: The item provides insufficient information to make a conclusion.

3. Calculate Proportions:

1. SUPPORTS: It increases the reference score to 1. If all claims are determined as 'SUPPORTS', then the reference score should be 1.
2. REFUTES: It decreases the reference score to -1. If all claims are determined as 'REFUTES', then the reference score should be -1.
3. NOT ENOUGH INFO: It doesn't affect the score directly, but it increases the denominator. Therefore, if a Wikidata item has many 'NOT ENOUGH INFO' values compared to other labels, the absolute value of the reference score will be minimized, approaching zero.

4. Normalize the Score: The total reference score is normalized by dividing the sum of weighted negative results by the total number of results.

Frontend Components

ProVe extends the Wikidata entity pages by adding two visual components:

• A ProVe Score indicator, ranging from -1 (high level of refuting references) to 1 (high level of references). Values close to 0 indicate balanced support/refutal and/or high livel of inconclusive/irretrievable references.
• An expandable table summarising the support stance computed for each of the external references, plus a link to the properties missing references.

ProVe Reasoning

Main algorithm

• The ProVe reasoning process is conducted sequentially for each Wikidata claim belonging to an item. ProVe processes an item by inputting its Q-id and returns 'No external URLs' if the item doesn't have any available external URLs. ProVe is designed to verify only external URLs, not internal URLs like connections among other Wikimedia and Wikidata resources.

ProVe Workflow

• Wikidata Parser: First, ProVe collects and filters Wikidata claims for an item with the requested Q-id. We have a great interest in checking the quality of external references, so ProVe filters out unnecessary Wikidata claims such as those having no URL or only internal URLs. For URLs encoded in a format different from common URLs, it transforms them into commonly accessible URLs for browsers. Then, it collects HTML documents according to these accessible URLs.

• Verbalization: In this function, ProVe transforms a set of Wikidata claims with URLs from RDF format to natural language sentences by adopting a verbalization model based on pre-trained T5. This function is designed to transform Wikidata claims into natural sentences for comparison with sentences from the HTML documents of URLs attached to the Wikidata claims. It runs for all Wikidata claims obtained in the previous Wikidata parsing process.

• Sentence Segmentation: This function cleans HTML documents obtained in the Wikidata parsing process to have only clean text without HTML tags and scripts. Then, it splits the cleaned HTML documents into a set of sentences using a Python library (pysbd). Segmented sentences will be compared to the Wikidata claims, respectively.

• Top-N Sentences Selection: ProVe creates multiple pairs for the Wikidata claims of the Wikidata Item. Each pair consists of one verbalized sentence from the Wikidata claim and a set of segmented sentences from the HTML document obtained by accessing the URL of the Wikidata claim. This function selects the top-N semantically similar sentences from the set of segmented sentences compared to the verbalized sentence, based on a fine-tuned BERT model.

• Text Entailment: This function determines if the Wikidata claim is supported or not by the N selected sentences, based on a fine-tuned BERT model with the FEVER dataset. The result for each selected sentence will be 'SUPPORTS', 'REFUTES', or 'NOT ENOUGH INFORMATION'. Finally, it returns a final result for the Wikidata claim by aggregating these text entailment results for the N selected sentences.

System architecture

ProVe inference server is currently hosted at King's College London due to the need of GPU resources to run the language model components. It consists of several parts:

• Main ProVe Process: A set of Python scripts for the ProVe process. All source code is available in ProVe's GitHub repository: https://github.com/King-s-Knowledge-Graph-Lab/RQV

• DBs and Resources: The second part relates to databases and resources used by ProVe to store results of reference evaluations. ProVe uses the Wikidata API to collect Wikidata claims and pre-trained machine learning models with datasets and configuration parameters.

• API Server: The third part is an API host server using Flask. It returns the requested data from external sources by interacting with the databases and functions of ProVe. Details of the APIs are written at the bottom of this page.

ProVe Tools

ProVe's User Interface

This is a widget displayed at the top of a Wikidata item's editing page that shows ProVe's reference evaluation results for the item.

The ProVe UI for Wikidata reference verification

When the widget is loaded, Prove will first add an information/control row. On this row, if the item's references have already been previously checked, the item's 'Prove score' is displayed. Hovering over the score will show an infobox with more detailed information. Next to the score indicator, a control button can be used to request ProVe to calculate or re-calculate item scores. A display button allows detailed reference results to be displayed or neatly put away so as not to interfere too much with the layout of the editing page.

Underneath the information/control row ProVe will display a grey box with a summary of the status of all the references for statements about the item. This will include the total number of statements with internal references (which are not checked); the total number of statements with missing references (and a link to the first missing reference for easy addition); and information about statements with external references which were checked by ProVe.

Underneath the grey box, filtering switches are displayed which can control what types of references are displayed below within a table: refuting, inconclusive, supportive or irretrievable.

A comprehensive table is then displayed with individual results for all references checked (according to the filtering switches), including links to the references and statements. This table can be hidden/displayed using the display button. This table has four main columns. The first column displays the statements for the item (with links to where the statement can be edited within the editing page); the second column displays either the passage within the referenced document attributed to the evaluation result, or the HTTP error code if the document could not be retrieved; the third column shows the support stance evaluated by ProVe; and the fourth column displays the link to the actual external reference.

You can find more additional information and a tutorial in the Google Slides presentation (please not the slides display an older version of the user interface): Empowering Wikidata editors and content with the Wikidata Quality Toolkit, Wikimania 2024 Hackathon

To implement and activate the ProVe gadget, please visit the Installation Guide at the top of this page.

We are working to publish the ProVe as an official Wikidata gadget.

ProVe Dashboard (Under Development)

• It will display statistics related to ProVe's work, such as (tentative):

 - The number of processed items
 - The number of processed triples
 - The number of processed references
 - User requests for Wikidata items
 - A reference score line graph over time

ProVe API

• The formal API documentation for the ProVe project is accessible on Swagger. Click the link to view the complete interactive documentation, where you can explore the API endpoints, parameters, and responses.

WikiProject Reference Verification

We would like to encourage editors to use ProVe to improve the quality of references of some pre-selected items that seem to need priority in checking the quality of their references. We have a WikiProject on Reference Verification that we encourage you to visit and use to try ProVe. Your feedback for improving the tool, and improving the overall quality of Wikidata references, will be really appreciated!

Additional Resources

Presentation: Empowering Wikidata editors and content with the Wikidata Quality Toolkit, Wikimania 2024 Hackathon