Wikidata:Tools/Paulina/Software Documentation

← back to tool main page

This page details Paulina's software architecture, as well as useful information to contribute to the project in a technical way, and steps to install, run and customize the application.

Paulina is a web application hosted on Toolforge, developed in Python using the Flask web framework. Data is obtained through some of the main methods for accessing and editing Wikidata data: the MediaWiki Action API, the Wikibase REST API, and the Wikidata Query Service. You can visit the code repository on Wikimedia's GitLab and the issue tracker on Phabricator.

Requirements

Python 3.13 or later.
Python third-party modules: Flask, Flask-Babel, Flask-Caching, Flask-Limiter, gunicorn, pytest, pytest-cov, pytest-flask, python-dateutil, requests, Requests-OAuthlib.
A basic Flask WSGI webservice. Learn how to set it up in Toolforge following steps 1 and 2 of the Wikitech's Flask tool step-by-step guide. In other hosting environments the steps are different. For example, the steps for setting up a Flask application in PythonAnywhere are these.

Creating a local development environment

Note: these instructions are for Linux. On other operating systems, the steps can be slightly different.

On your terminal:

1. Clone the repository to a local folder:

git clone git@gitlab.wikimedia.org:toolforge-repos/paulina.git

or

git clone https://gitlab.wikimedia.org/toolforge-repos/paulina.git

2. Go to the newly created folder:

cd paulina

Paulina uses uv as project manager, since it facilitates the management of the virtual environment and the versions of Python and packages.

3. Install uv with:

curl -LsSf https://astral.sh/uv/install.sh | sh

4. Check you are already inside the paulina folder. Run:

uv sync

The uv sync command creates your virtual environment in the .venv folder, installs the Python version specified in pyproject.toml, and downloads and installs all the dependencies.

5. Configure your environment variables or `config.py` file. (For more details about this file, go here)

By default, config.py looks for environment variables (envvars) and, if it doesn't find them, it fallbacks to dummy values. You can set your own variables in your environment (for example, using an .env file) or you can modify your config.py file locally (if you choose this last option, please don't push your modified config.py file to the repository!).

The only mandatory variables in the config.py file are SECRET KEY and USER_AGENT:

SECRET_KEY = "YourSecretKeyHere"

The secret key is a random string to secure the session cookie. The session cookie is necessary so that the language choice is persistent throughout the session. Replace YourSecretKeyHere with an actual key.

USER_AGENT = "Paulina/1.0 (https://paulina.toolforge.org/; your-email@example.com)"

The user agent is for the Wikimedia API calls made by the tool. Replace the placeholder email with your actual contact email for compliance with Wikimedia's user agent policy.

If you'd like to run the application in DEBUG mode (which is preferable in a development environment), set:

DEBUG = True

When Flask runs in debug mode, it shows detailed error messages and it auto-reloads on code changes.

If you want to use the Wikidata editing features locally, you need to create OAuth credentials using the following steps (so these 3 steps are optional):

Step 1: Register Your OAuth Consumer
Go to: https://meta.wikimedia.org/wiki/Special:OAuthConsumerRegistration/propose/oauth2

Fill out the form:

Application name: Paulina Local Dev - YourWikimediaUsername
Application version: 2.0
Application description: Local development instance of Paulina for testing Wikidata editing features.
This consumer is for use only by [username]: Leave UNCHECKED
OAuth "callback" URL: http://localhost:5000/oauth-callback
Check the following for applicable grants:
- Basic rights
- Edit existing pages
- Create, edit, and move pages

Click "Propose consumer"

Step 2: Copy Your Credentials
After registration, you'll receive:

Client ID (Application key)
Client Secret (Application secret)

It is important to copy both immediately as the secret is only shown once. Test consumers expire after 30 days.

Step 3: Update your envvars or your config.py with the OAuth Credentials:

CLIENT_ID = "your-client-application-key-here"
CLIENT_SECRET = "your-client-application-secret-here"
REDIRECT_URL = "http://localhost:5000/oauth-callback"

Important: Keep CLIENT_SECRET secret. This is your OAuth 2.0 application secret from Wikimedia and it is confidential.

REDIRECT_URL is the callback URL where Wikimedia returns users after authentication. You always use http://localhost:5000/oauth-callback for your local environment.

6. That’s it. Your local environment has been created. Paulina can run in test mode or production mode.

For test mode, run:

uv run app.py test

For production mode, run:

uv run app.py

uv will activate your virtual environment automatically. If you don't use uv, you have to activate the virtual environment before executing the app.

Test mode uses test.wikidata.org while production mode uses the actual wikidata.org when sending data from Paulina to Wikidata.

For testing ‘Add Author’ and ‘Add Work’ features, test mode is always recommended for local development to safely create new items without affecting real data.

For testing other Wikidata editing features, run Paulina in production mode and use Wikidata Sandbox to test.

Attention: Don't confuse test mode with debug mode. Test mode/production mode is used to configure the Wikidata instance where edits made with Paulina are executed. Debug mode is something different: when set to True, you get tracebacks of errors, and the application in your local browser is updated in real time without needing to restart the local server every time you add or change code.

Application files

This is the Paulina software repository file tree:

paulina
├── .gitignore
├── .gitlab-ci.yml
├── .python-version
├── app.py
├── babel.cfg
├── config.py
├── filters.py
├── languages.py
├── LICENSE
├── messages.pot
├── oauth.py
├── pdclasses.py
├── pyproject.toml
├── README.md
├── search.py
├── squeries.py
├── uv.lock
├── wikidata_api.py
├── wikidata_constants.py
├── static
│   ├── css
│   │   └── main.css
│   ├── fonts
│   │   └── ...
│   ├── images
│   |   └── ...
|   ├── js
|       └── ... 
├── templates
│   ├── 404.html
│   ├── about.html
│   ├── add-author.html
│   ├── add-work.html
│   ├── author.html
│   ├── countries.html
│   ├── country.html
│   ├── home.html
│   ├── layout.html
│   ├── no-results.html
│   ├── oauth-callback.html
│   ├── results.html
│   ├── term.html
│   ├── work.html
│   └── works-list.html
├── tests
│   ├── conftest.py
│   ├── test_error_handling.py
│   ├── TEST_PLAN.md
│   ├── test_routes.py
│   ├── test_search.py
│   └── test_templates.py
└── translations
    ├── ar
    │   └── LC_MESSAGES
    │       ├── messages.mo
    │       └── messages.po
    └── ...

app.py

app.py is the root file of the software.

This file:

Initializes the main application.
Initializes Babel, the application responsible for internationalization (i18n).
Initializes Flask-Caching, the application needed for caches.
Loads the config file.
Initializes Flask-Limiter for rate limiting API requests.
Determines Wikidata mode (test vs production) from command-line arguments.

app.py also defines all the routes, with the HTML templates that will be displayed for each URL and the data that will populate those templates.

It also includes a small function that allows you to handle special types of data.

config.py

See Configure your environment variables or config.py file.

search.py

search.py includes 3 functions. The two main ones are:

search_author()
search_work()

These two functions receive the user's input and connect to the Wikidata API to perform a search for Wikidata items with CirrusSearch (more about this).

search_author() includes the filter haswbstatement:P31=Q5|P31=Q21070568 (for human beings) in the CirrusSearch search itself, while search_work() takes the results from CirrusSearch and filters them with a query that leaves only the items that have creator (P170), author (P50), director (P57) and/or other similar properties.

The search_filters() function is a helper function that is called from the two previous functions to include the filters chosen by the users in the search.

filters.py

filters.py maps the filters displayed in the "advanced search" section of the search form to their respective Wikidata IDs. This module is imported by app.py, where the inject_filters() function injects the filters into the layout.html template.

pdclasses.py

pdclasses.py contains the different types of objects (classes) that Paulina has: Author, Work, Country and Term. Each type of object has its own attributes that describe the entity and are shown in the page dedicated to that entity. The classes also have methods that perform actions with the entities (for example, calculating the age of an author, consulting his works and inferring whether they are in the public domain).

Objects are created from the Q ID of a Wikidata item. The function retrieve_item(), used by all classes, connects to the Wikibase REST API and requests the information of the Wikidata item item. Depending on the case, it requests the complete information of the item or only the labels. Some attributes and methods require extra information, not contained in the individual Wikidata item, so they run queries.

Other functions in this module, build_field() and build_date(), allow handling special data types, such as unspecified values and dates, and also identify the preferred value of a property if it exists.

squeries.py

squeries.py is imported by others that use Wikidata queries. This module contains the templates of all the SPARQL queries in the application:

squeries.py also includes a function, retrieve_query(), that connects to the Wikidata Query Service and requests the data in JSON format. This function has one required argument, which is the name of the query template, and accepts one or more keyword arguments that correspond to the variables that customize the template.

languages.py

languages.py deals with language-related issues. First, it includes a list of supported languages taken from Wikidata ((only two-character codes). Paulina can potentially take and display data from Wikidata in all those languages.

The module also includes a dictionary of already translated languages (see Translations), with their name and code. These selected languages are displayed in the language selector of the web application.

Finally, languages.py has a function that takes the language selected by the user (included in the "language" URL parameter) and stores it in a session cookie (see Configure your environment variables or config.py file) that has the sole purpose of making the language choice persistent throughout the session. If the user did not choose a language, the language is taken from the browser's Accept-Language HTTP header. If that header does not exist, the default language is English.

oauth.py

oauth.py handles OAuth 2.0 authentication with Wikimedia, allowing users to log in with their Wikimedia accounts and make authenticated edits to Wikidata. The module uses requests-oauthlib to manage OAuth tokens securely and stores the authenticated session in Flask's session storage. oauth.py features:

save_token_to_session(new_token): A callback function that saves refreshed OAuth tokens to the Flask session. It is automatically called by requests-oauthlib when a token is refreshed and updates the session with the new token data (access_token, refresh_token, expires_at)

get_oauth_session(token=None, state=None): It creates an OAuth2Session object with automatic token refresh capability by retrieving the token from the Flask session if not provided, and returns a ready-to-use OAuth session for making authenticated requests. It configures the OAuth2Session with client credentials, redirect URL, and scopes.

token_refesh(): Proactively refreshes OAuth tokens before they expire to maintain user sessions. It checks if a token exists in session and refreshes it if expiration is in less than 5 minutes (300 seconds).

wikidata_api.py

wikidata_api.py handles all interactions with the Wikibase REST API for creating and editing items. The WikidataAPI Class contains different functions for processing forms, creating items, editing statements and structuring payloads to meet Wikibase REST API format. It features:

create_item(): Creates new Wikidata items with labels, descriptions, and statements.
update_statement_field(): Updates individual statement fields on existing items, handling both single-value and multi-select fields with reference support.
build_statement(): Constructs properly formatted statement objects for the API.
build_reference(): Creates reference objects with URL and retrieval date.
process_author_form(): Validates and processes author creation forms.
process_work_form():Validates and processes work creation forms.
build_author_statements_from_form() and build_work_statements_from_form(): Converts form data into Wikidata statements for authors and works.

Finally, wikidata_api.py also has some utility functions to help structure and validate form values.

generate_author_description() and generate_work_description(): Auto-generates Wikidata standardized descriptions when the user leaves the description field empty. Following the Wikidata standard for descriptions, it ensures the sentence starts with a lower-case letter and does not end with a period.
validate_date_format(): Formats date to Wikidata standard and validates date inputs with support for partial dates (year only and year-month).

wikidata_constants.py

wikidata_constants.py defines all Wikidata property IDs (PIDs) and item IDs (QIDs) used throughout the application, and includes test mode configuration. This is essential because test.wikidata.org uses different QIDs and PIDs from wikidata.org. It features:

WIKIDATA_TEST_MODE: Boolean flag to switch between test.wikidata.org and production Wikidata.
Property IDs: Constants like `PID_AUTHOR`, `PID_OCCUPATION`, `PID_DATE_OF_BIRTH` map to their Wikidata property IDs.
Item IDs: Constants like `QID_HUMAN` map to specific Wikidata item IDs.
FORM_GENDERS: Different gender types with both their test QIDs and production QIDs.
WORK_TYPES: Configuration array defining different work types (books, films, albums, etc.) with their required and optional fields.

This centralization makes it easy to adapt the entire application between test and production environments by changing a single variable.

babel.cfg

babel.cfg is the i18n Babel application configuration file.

messages.pot

messages.pot is the Babel-generated template for translations.

pyproject.toml

pyproject.toml is a standard configuration file that contains the minimum required Python version and a list of the required third-party modules listed in Requirements. They can be installed by running from the root folder of the application:

uv sync

.python-version and uv.lock

.python-version and uv.lock are uv specific configuration files that help uv reproduce the exact same environment in every machine.

.gitignore

.gitignore is a file used to specify untracked files when working with the Git version control system.

.gitlab-ci.yml

.gitlab-ci.yml is a configuration file for GitLab CI/CD pipelines. It automates testing and deployment of the code pushed to the repository.

static

The folder called /static/ includes four subfolders:

css: contains the main.css file, which includes a set of styles that modify Bootstrap styles, the CSS framework used by Paulina.
fonts: Paulina uses the Lato typeface. This folder includes all the variants used of this typeface.
images: contains all the images used in the application, including several versions of the logo and icons.
js: contains all reusable JavaScript components and functionalities. These include:

combobox.js	Implements accessible combobox widgets for dropdown selectors with keyboard navigation support.
duplicate-detection.js	Prevents duplicate item creation by searching Wikidata for existing authors or works before allowing form submission. It features: Real-time duplicate detection as users type Displays potential matches with links to existing Wikidata items Allows users to confirm creation if no match exists
form-utils.js	Provides reusable form validation utilities used across author and work forms. The FormUtils Object contains the following functions: `validateDateInput()`: Validates date components (year, month, day) with detailed error messages `combineDateParts()`: Combines separate date inputs into YYYY-MM-DD format `normalizeDate()`: Handles partial dates (e.g., "0" month/day for unknown values)
image-preview.js	Provides live preview of Wikimedia Commons images when users paste image URLs into author/work creation forms. It has support for extracting filenames from different Commons URL formats.
references.js	Manages reference attachments for edited fields, allowing users to cite sources for their Wikidata contributions. The ReferenceManager Class: Adds/removes reference input fields dynamically Enforces maximum reference limits per field Collects reference URLs and retrieval dates Formats references for API submission
wikidata-search.js	Provides autocomplete search functionality for Wikidata entities (items, countries, languages) with support for both single-select and multi-select fields.The WikidataItemSearch Class: Fetches search results from Wikidata's Action API as users type Displays results with labels and descriptions Handles single-select (nationality, gender) and multi-select (occupation) fields Prevents duplicate selections Stores selected QIDs in hidden inputs for form submission

templates

The folder called /templates/: contains all the HTML templates used in the application. These templates are used to create web pages dynamically with the Jinja web templating engine included in Flask. For example, there is an author.html template that is used to create the information page for any author.

There is also a basic template, called layout.html, that includes the <head> metadata (the favicon, Open Graph meta tags, the link to Bootstrap CDN, among other things), the header, search form, footer, and other basic aspects present in all pages. This layout template is inherited by all other templates.

Jinja allows, among other things, to include information stored in variables in the html file and to execute loops to create html code dinamically. This technique is used, for example, in the results.html template to display an indefinite number of search results, and in the works-list.html template to display the list of works by an author. Learn more about Jinja in the Jinja Template Designer Documentation.

Authenticated users can create new author and work items on Wikidata directly from Paulina. On author.html and work.html pages, authenticated users can click edit icons next to individual fields to update them directly. Changes are saved immediately to Wikidata with optional reference citations.

Missing information can also be added through a batch editing feature that displays all empty fields on author/work pages in a collapsible panel. Users can fill multiple fields at once and publish all changes.

In some cases, JavaScript logic was added directly in templates rather than standalone modules because it needs access to Flask/Jinja2 template variables, server-side data, and translated strings from Flask-Babel.

Additionally, small page-specific scripts that aren't reused across multiple pages are more maintainable when kept with their templates, avoiding the overhead of managing separate files for minor functionalities.

tests

The folder called /tests/ include a set of files containing different tests that can be executed using pytest. Tests can be executed going to the root folder and running:

uv run pytest -v --cov=. --cov-report=term-missing

translations

The folder called /translations/ contains subfolders with the translations into different languages. The structure of these subfolders is a i18n standard and is automatically generated by the Babel application. For each language there is a messages.po file containing the translated strings in plain text, and a messages.mo file containing the same strings compiled into a binary file.

Learn more about how to use Babel with Flask in the chapter i18n and L10n of Miguel Grinberg's excellent Flask tutorial.

Translations are done collaboratively on Weblate (see Translations).

Paulina Local

Warning: Paulina Local is not up to date with the changes in the main repository and its development pace is slower.

Paulina Local is a fork of the main Paulina repository. It contains adaptations with respect to the main repository to facilitate the implementation of local or thematic versions of the tool.

The most important difference with respect to the main repository is the addition of a file in the root folder of the app, local_settings.json, which contains the custom information. In the main version of Paulina, the logo, the title of the site and the texts of the home page, among other data, are hardcoded in the html templates. In Paulina Local, however, this data is taken from local_settings.json.

local_settings.json can be modified by hand from the version available in the Paulina Local repository. Another possibility is to delete the file when installing the application on the server. If the file is not on the server, the first time we enter the site the interface will redirect us to a web form where we can fill in this data, after which the file will be created automatically.

Another important issue in local or thematic versions is the need to adapt the main search results. For example, the first implementation of Paulina Local, called Dominio Público Uruguay (available at https://dominiopublico.uy), is a national portal focused on Uruguay's cultural heritage. Thus, the main search for authors and works must be filtered to show only results for Uruguayan authors and works.

This is achieved by modifying the search.py file, in particular the search_author() and search_work() functions.

For example, in the main Paulina application, search_author() contains this line with a filter for human beings:

human_being_filter = "%20haswbstatement:P31=Q5|P31=Q21070568"

In the Dominio Público Uruguay repository, that line was changed. In addition to the human being filter, now it contains two more filters: 1) a filter for items that have IDs of catalogs of authors from Uruguay, and 2) a filter for Uruguayan nationality:

predefined_filters = ("%20haswbstatement:P31=Q5|P31=Q21070568" # Is a human being
                     "%20haswbstatement:P12595|P2558|P6156" # Has an ID from BNU, autores.uy or MNAV
                     "%20haswbstatement:P27=Q77") # Has Uruguayan nationality

As for the search_works() function, at certain point this function calls the query query_for_works_from_search_results, available in the squeries.py module. Compared to the main Paulina application, the Dominio Público Uruguay repository has an extra line added to that query:

?author wdt:P27 wd:Q77 .

meaning "author of the work has country of citizenship (P27) -> Uruguay (Q77)".

These or other changes can be made to filter by default the main search results of the Paulina Local custom implementation by country, region, type of work, language, etc.

Contribute

To contribute to Paulina, you need to have registered accounts on Wikimedia's GitLab and Phabricator. If you don't have them, read the instructions to create them.

Issues: Request a feature, report a bug or suggest other tasks on the Paulina Phabricator project.

Merge requests: Merge requests can be submitted to Paulina's repository on Wikimedia's GitLab.

Translations

You can help translate Paulina on Weblate.

License

This software is under a GNU Affero General Public License Version 3.

Code repositories

Paulina (main application)
Paulina Local (fork of the main app for custom implementations)