Wikidata:WikiProject Authority control

From Wikidata
Jump to navigation Jump to search

Wikidata pays a lot of tribute to authority control, linking to all kinds of datasets and databases with various IDs. The holy grail of every GLAM worker Sum of All People, with links to their Works is coming about!

But we’re just at the start of a lot of work in that direction. The purpose of this project is to try and coordinate such work.

I know a few things (ViafBot, Mix-n-Match) and I'd like to help with some things, but I don't know what others are doing. --Vladimir Alexiev (talk) 08:08, 27 January 2015 (UTC)

Data Sources[edit]

The report Name Data Sources for Semantic Enrichment shows that when it comes to name data sources, maybe the two that matter are VIAF and Wikidata.

  • Their name coverage is fairly orthogonal: VIAF has more name variations and permutations, Wikidata has more translations (Venn diagram of names for Cranach).
  • VIAF is much bigger: 35M persons/orgs. Wikidata has 2.7M persons and maybe 1M orgs.
  • Only 0.5M of Wikidata persons/orgs are coreferenced to VIAF, with maybe another 0.5M coreferenced to other datasets, either VIAF-constituent (eg GND) or non-constituent (eg RKDartists). So coreferenced part between the two is still quite small (30%) and a lot of work remains!
  • A lot can be gained by leveraging coreferencing across VIAF and Wikidata: finding errors in Authority files, finding merge candidates in Wikidata, promulgating identifiers...
  • Wikidata has great tools for crowd-sourced coreferencing.

Please comment!

Resolve against VIAF links[edit]

Wikidata authority IDs are obtained from various places, and hand-edited. But http://viaf.org/viaf/data/viaf-20150115-links.txt.gz has links to IDs of VIAF participants (constituent national libraries) and Wikipedia.

Stats:

  • 2Gb unzipped size
  • 27684634 subjects (27.7M)
  • 1.67 links per subject
  • 46248396 links, of which 377650 enwiki links.
  • Link breakdown:
  27684634 VIAF    total subjects
  10531522 DNB     Germany
   9154093 LC      LC (NACO)
   7655649 ISNI    ISNI
   2555033 NTA     Netherlands
   2508374 SUDOC   France (Sudoc)
   2036493 BNF     France (BnF)
   2018647 XR      xR OCLC file
   1351105 NUKAT   Poland (NUKAT)
   1032862 NDL     Japan (NDL)
   1016708 NLA     Australia
    844024 NLP     Poland (Nat lib)
    743215 NKC     Czech
    689827 LAC     Canada
    570840 NLI     Israel
    562244 BNE     Spain
    473518 NSK     Croatia
    377650 WKP     Wikipedia
    373078 PTBNP   Portugal
    320898 BAV     Vatican
    232327 JPG     Getty (ULAN)
    220304 RERO    Swiss (RERO)
    187073 SELIBR  Sweden
    169028 ICCU    Italy
    158515 LNB     Latvia
    144299 BNC     Catalunya
    101500 DBC     Denmark (DBC)
     73421 BIBSYS  Norway
     45633 SWNL    Swiss (Nat lib)
     37004 EGAXA   Egypt
     33727 NSZL    Hungary
     11000 LNL     Lebanon
      9953 IMAGINE
      5723 VLACC   Belgium (Flemish)
      1228 PERSEUS Perseus
       997 RSL     Russia
       408 NLB     Singapore
       267 XA      xA OCLC file
       209 SRP     Syriac

Now let's count some IDs in Wikidata, using WDQ API and working down the list:

  • VIAF: 504736
    • would be nice to cross-check this against the 377650 enwiki links
    • Wikidata has 13M items and VIAF has 27.7M subjects, so I would expect at least 3-4M common subjects. This means that we have co-referencing for only 15% of the possible items! A lot of work remains
  • GND: 335883
    • 37% of all VIAF items have GND id, while in Wikidata the ratio is 66%. This means that GND co-referencing is more advanced than VIAF co-referencing
  • VIAF or GND: 567240
    • Everything in VIAF has a GND, but this shows that in Wikidata, 62504 with GND id don't have a VIAF id. We can assign VIAF id to these easily!
  • LCNAF: 210845
    • VIAF or LCNAF: 506136
    • Again, we can leverage LCNAF ids to assign 1400 VIAF ids easily
  • RKDartists: 21760: most of these can be added to VIAF for free!
  • NTA: 335883
  • SUDOC: 103120

More importantly: we could cross-check all 36 ID's in VIAF against Wikidata to:

  • add the missing ones,
  • flag different ones with a qualifier "questionable".
unsigned comment by user:Vladimir Alexiev 16:33, 23 January 2015‎
I imported the NTA stuff some time ago. Could easily do the same for other properties, but I don't know how good the data is and if I need to do some transformations. Please start a wikiproject. Would love to comment over there and get everything imported. Multichill (talk) 21:20, 26 January 2015 (UTC)
User:Multichill Thank you sir/lady, will do. Chill :-)
Some remarks:
  • VIAF contains "name authorities" (historically persons first, later then corporate bodies, finally "geographic" items, works and "expressions" too. However "Subject Headings" ("topical terms" as in LCSH against LCNAF) are not part of VIAF, although some of the authority files (like GND) include them. Thus "X completely included in VIAF" does not hold for all constituent files of VIAF.
  • Please be careful with respect to the ODC-BY license which reigns the VIAF dumps: Whereas the constituent authority files (GND for sure) are CC-0, VIAF definitely is not. (My interpretation: Import on a case-by-case basis is o.k., everything more must make certain that the license is met. Thus it fr.wikipedia would present SUDOC numbers fetched from Wikidata which here would have been collected by bulk-matching existing GND numbers against VIAF, french wikipedia would be obliged to present VIAF attribution or VIAF identifiers next to the SUDOC numbers.
  • OCLC some years ago made a matching between VIAF and en.wikipedia.org and afterwards donated the data such that VIAF numbers could be imported into English Wikipedia (afterwards some cross-check against the VIAF numbers in corresponding articles of de.wikipedia was performed) from where they moved on to wikidata. I doubt that they ever repeated the matching and believe that "wikipedia" linking in VIAF still reflects the original matching. Therefore in a sense there is no need to compare wikidata to VIAF again. Unless of course this can magically restricted to VIAF numbers which never appeared in en.wikipedia. -- Gymel (talk) 12:58, 27 January 2015 (UTC)

New situation[edit]

Have you read this?

Yes, this is a very nice announcement since it means that VIAF will source Wikidata actively, which hopefully will close the gap between the two. I think this makes it even more important to leverage VIAF and other authority IDs in Wikidata. --Vladimir Alexiev (talk) 09:57, 29 May 2015 (UTC)

Update Mar 2017:

select (count(*) as ?c) {
  ?x wdt:P214 ?viaf
  filter exists {?x wdt:P31 wd:Q5}
}

Try it!

  • My guesstimate is that out of 4.5M humans on WD, half are in VIAF. So we only have 35% of the possible links. --Vladimir Alexiev (talk) 14:57, 26 March 2017 (UTC)

RKDArtists Coreferencing[edit]

RKDartists is an important Authority that does not yet participate in VIAF. There are already 21760 RKDartist id's on Wikidata. These could be imported to VIAF for free!

British Museum Coreferencing[edit]

The BM has several thesauri that are not co-referenced to anything in the world. I think they'd see it as a major win if the community helps them to co-reference.

This could be followed by importing the 2.5M cultural objects of the BM.

ULAN Coref Relations[edit]

ULAN does record possible matches and mismatches in their editorial system: ULAN Artists Whose Identity May be Associated or Confused With Another (608 pairs).

Looks like this:

x x_name x_bio rel y y_name y_bio
ulan:500071106 Master of 1515 Portuguese painter, active 1515 gvp:ulan1005_possibly_identified_with ulan:500025279 Afonso, Jorge Portuguese painter and court artist, born ca. 1470-1475, died before 1540
ulan:500042027 Master of the Madre de Deus Retablo Portuguese painter, active 16th century gvp:ulan1005_possibly_identified_with ulan:500025279 Afonso, Jorge Portuguese painter and court artist, born ca. 1470-1475, died before 1540
ulan:500032055 Monogrammist A. M. Spanish artist, active 19th century gvp:ulan1005_possibly_identified_with ulan:500038287 Aguirre, Marcial Spanish sculptor, 1841-1900

Here's to proper coreferencing! --Vladimir Alexiev (talk) 18:07, 12 March 2015 (UTC)

Match Persons not Disambiguation Pages[edit]

We should match persons to persons, not disambiguation pages to persons or other disambiguation pages.

Wikipedias, GND and RKD all have disambiguation pages (in GND they are called "undifferentiated names"). 13 Feb 2015:

Do you agree with my reasoning:

  • Jane said "any match is better than none"
  • I countered "A correct match is better than none"
  • the only way to make sure it's correct is to examine more data about the person, which will necessarily lead you to a real person page.
  • Look at the ULAN data above: that's good data that gives you some basis for decision. A name alone does not.

--Vladimir Alexiev (talk) 19:34, 12 March 2015 (UTC)

Symbol support vote.svg Support @Vladimir Alexiev, Randykitty, Ghuron: A GND Tn (Thesaurus name = undifferentiated) is not a stable disambiguation page. A Tn is a placeholder. It can be deleted, it can be upgraded into a Tp (Thesaurus person), or changed into a redirect. Works connected with a Tn will be checked by the library or archive who owns them and afterwards might be delinked. The database Online GND (OGND) includes only Tp numbers. --Kolja21 (talk) 00:04, 28 March 2015 (UTC)

Coreference AAT[edit]

AAT is a crucial thesaurus in cultural heritage.

    select (count(*) as ?c) {
      ?x a skos:Concept; skos:inScheme aat: }

I think that's BAD. I'm sure going to need that coref for the Europeana Food and Drink Classification Scheme that will be based on Wikidata and AAT:

Update: the AAT-Wordnet coreference described below is brought into Wikidata. AAT is actively coreferenced on Mix-n-Match: 12985 (32%) matched, 3293 (8%) awaiting confirmation, 1553 (3.8%) confirmed no-matches, and 22543 (55.7%) awaiting matching. So it's way better than 2 years ago. Help coreference this pivot thesaurus that is of immense importance for Cultural Heritage! --Vladimir Alexiev (talk) 15:04, 19 September 2017 (UTC)

Coreference AAT with Mix-n-Match[edit]

The Wikidata coref tool Mix-n-Match has mostly been used for people until now. But I hope it can be used for concepts as well.

I made an export that includes AAT URL, preferred English label (without qualifier), parents (ascendants to root) and scope note (description). Could also add alternative labels, and labels in other languages (Dutch, Spanish, Chinese).

select ?id (str(?lab) as ?label) ?parents (str(?scopeNote) as ?note) {
  ?x a gvp:Concept; dc:identifier ?id; gvp:prefLabelGVP/gvp:term ?lab;
     gvp:parentString ?parents.
  optional {?x skos:scopeNote [dct:language gvp_lang:en; rdf:value ?scopeNote]}
}

I saved as XML then converted to TDV: aat.rar.

    rset --results tsv aat.xml > aat.tdv

Also see https://meta.wikimedia.org/wiki/Talk:Mix%27n%27match#Coreference_AAT !!!

Coreference AAT through BabelNet[edit]

Mix-n-Match has good automatic matching, but that works for people.

So let's check what other vocabs that are coref to AAT may be coref to Wikidata: According to Michiel Hildebrand's famous CH LOD diagram:

CH LOD, cultural heritage linked open data (thesauri only)
  • Wordnet. No such prop in Wikidata
  • I'd guess Wiktionary is coref to Wordnet, but Wikidata got no site links to Wiktionary
  • RKD Concepts. There's prop "RKDartists" and "RKDimages" but none for concepts
  • Rijksmuseum Concepts. There's "Rijksmonument" but none for concepts
  • Joconde: aha! There's Joconde ID (P347), and it has 2275 instances, so that's better. Joconde is 18% coref to Wikidata but I don't know how much to AAT, maybe I can gain 1k here.
    • Looked at the results: nope, Joconde are all paintings, not concepts
  • Bibliopolis: never heard of it, and nope
  • SVCN: never heard of it, and nope

Then it dawns on me.

AAT-Wordnet coref[edit]

Ok, so off to look for that AAT-Wordnet coref. - Why yes, it's part of http://semanticweb.cs.vu.nl/europeana/skos/browse/ - I got a file from somewhere that says

<aat_wordnet20_mappings>
  a void:Linkset;
    dcterms:title "AAT-Wordnet 2.0 mappings by Anna Tordai (baseline)" ;
    lib:source <http://semanticweb.cs.vu.nl/lod/getty/aat/> ;
    void:dataDump <bl_aat_wn.rdf> , <bl_norm_aat_wn.rdf> , <bl_sing_aat_wn.rdf> .

(Note: you can get those files from URLs like: http://semanticweb.cs.vu.nl/europeana/api/export_graph?graph=http://semanticweb.cs.vu.nl/lod/getty/aat/bl_sing_aat_wn.rdf&mimetype=text/plain&format=turtle)

These are called "baseline" (i.e. mostly literal matches). A quick conversion to Turtle and a line count:

$ wc -l bl*
   2300 bl_aat_wn.ttl
   4369 bl_norm_aat_wn.ttl
   4303 bl_sing_aat_wn.ttl
  10972 total

Run a query at http://semanticweb.cs.vu.nl/europeana/user/query (specify entailment=None or else!):

prefix getty:  <http://purl.org/vocabularies/getty/> 
prefix aat:  <http://purl.org/vocabularies/getty/aat/> 
select * {?x skos:inScheme getty:aat; skos:closeMatch ?y}

It returns 4592 (see below why).

AAT-Wordnet Overlaps[edit]

There is significant overlap between the files:

$ cat bl* | sort| uniq | wc -l
4596
$ cat bl* |cut -d " " -f 1 | sort| uniq | wc -l
4581

The following AAT concepts have 2 matches:

aat:bleachers
aat:boxcars
aat:cleavers
aat:feudalism
aat:groats
aat:jackstraws
aat:lats
aat:leotards
aat:morocco
aat:ninepins
aat:quoits
aat:shekels
aat:stairs

We need to reconcile them manually, eg

aat:bleachers  skos:closeMatch  <http://www.w3.org/2006/03/wn/wn20/instances/synset-bleacher-noun-1> .
aat:bleachers  skos:closeMatch  <http://www.w3.org/2006/03/wn/wn20/instances/synset-bleachers-noun-1> .

The AAT definition is:

  • aat:bleachers vp:descriptiveNote "Use for benchlike tiered seating for spectators at, for example, outdoor sporting events, usually without weather or sun protection, affording less advantageous views than grandstands; may also be used for similarly constructed, often telescoping, indoor seating."@en .
  • Inspection at Wordnet 3.1 shows that the second one is right.

That's 4.6k matches, or 11% of AAT.

AAT-Wordnet2 Representation[edit]

The coref looks like this:

    aat:wrought_iron skos:closeMatch <http://www.w3.org/2006/03/wn/wn20/instances/synset-wrought_iron-noun-1> .

And there's another file aat.ttl with rep like:

aat:wrought_iron aat:parentPreferred aat:iron_alloy .
aat:wrought_iron vp:id "300011012" .
aat:wrought_iron vp:labelPreferred "wrought iron"@en .
aat:wrought_iron vp:labelNonPreferred "iron, wrought"@en .
aat:wrought_iron vp:labelNonPreferred "wrought-iron"@en .

This is quite old rep. The new rep uses numeric URL: http://vocab.getty.edu/aat/300011012 (and a bunch more data). So we need to construct a numeric URL.

AATNED-Cornetto Mapping[edit]

Cornetto is NL Wordnet and AATNED is NL AAT. I got another file saying:

<aatned_cornetto_mappings>
  a void:Linkset ;
    dcterms:title "AATNED-Cornetto mappings by Anna Tordai (baseline)";
    lib:source <http://semanticweb.cs.vu.nl/lod/rkd/aatned/> ;
    void:dataDump <bl_aatned_cn.rdf.gz> , <bl_norm_aatned_cn.rdf.gz> , <bl_sing_aatned_cn.rdf.gz> .

Eg we have this for AAT 300191645 "salinity":

bl_aatned_cn.ttl: aatned:zoutheid  skos:closeMatch  cornetto:synset-zoutheid-1-noun .
cornetto-wn20.ttl: cornetto:synset-zoutheid-1-noun cornetto:eqNearSynonym instances:synset-brininess-noun-1 .
cornetto-wn30.ttl: cornetto:synset-zoutheid-1-noun cornetto:eqNearSynonym wn30:synset-brininess-noun-1 .
aatned.ttl: aatned:zoutheid core:notation "300191645" .

The number of AATNED-Cornetto matches is as follows:

> cat bl*|sort|uniq> bl_aatned_all.ttl
> wc -l bl_aatned_all.ttl
6917 bl_aatned_all.ttl
> cat bl_aatned_all.ttl|cut -d " " -f 1 | sort| uniq | wc -l
6857

There are more matches than AAT-Wordnet. There are also overlaps: 60 AATNED concepts (0.9%) have two Cornetto matches.

We need to merge AATNED-Cornetto with AAT-Wordnet. The correlation is simply by id, eg

aatned.ttl: aatned:zwerfkeien core:notation "300011671"
aat.ttl: aat:boulder vp:id "300011671"

I guess the overlaps between them are quite big, eg for wrought_iron:

aatned.ttl: aatned:smeedijzer core:notation "300011012"
bl_aatned_all.ttl: aatned:smeedijzer  skos:closeMatch  cornetto:synset-smeedijzer-1-noun .
cornetto-wn20.ttl:cornetto:synset-smeedijzer-1-noun cornetto:eqNearSynonym instances:synset-wrought_iron-noun-1 .
cornetto-wn30.ttl:cornetto:synset-smeedijzer-1-noun cornetto:eqNearSynonym wn30:synset-wrought_iron-noun-1 .
DBpedia-Wordnet3 coref[edit]

The other problem is bigger:

    bn:s00081730n skos:exactMatch dbpedia:Wrought_iron, lemon-WordNet:wn30-14802262-n

It doesn't look like Wordnet3 and Wordnet2 share any IDs; we'll deal with that in next section.

Lets first do some queries at http://babelnet.org/sparql/ to see what we can see. Look for DBpedia-Wordnet matches:

SELECT * WHERE {
  ?x skos:exactMatch ?y, ?z 
  filter(strstarts(str(?y),"http://dbpedia.org/resource/"))
  filter(strstarts(str(?z),"http://lemon-model.net/lexica/pwn/"))
} LIMIT 30

Download: https://www.dropbox.com/s/92gq5r1qm3yytkp/WN3toDBP.csv?dl=1.

It has 47607 rows like this (there's a decent chance this will cover the 6k AAT matches):

"http://babelnet.org/rdf/s00075206n","http://dbpedia.org/resource/Sundowner_(drink)","http://lemon-model.net/lexica/pwn/wn30-07913081-n"
"http://babelnet.org/rdf/s00039711n","http://dbpedia.org/resource/Sonora_(genus)","http://lemon-model.net/lexica/pwn/wn30-01736256-n"
"http://babelnet.org/rdf/s00070026n","http://dbpedia.org/resource/Sealskin","http://lemon-model.net/lexica/pwn/wn30-04160261-n"
Wordnet3-Wordnet2 coref[edit]

Since Wordnet3 and Wordnet2 don't share any IDs, we can try to use Wordnet2-Wordnet3 coref made by Jacco van Ossenbruggen and Marc van Assem (VU University Amsterdam) in May 2010 with this VOID (manifest):

<wn30-wn20-mappings-jacco>
	a void:Linkset ;
	dcterms:title "synset-level mappings from Wordnet 3.0 to 2.0, created by jacco's code" ;
	lib:source <http://purl.org/vocabularies/princeton/wn30/> ;
	void:dataDump
		<label-child-matches.ttl.gz> ,
		<label-childparent-matches.ttl.gz> ,
		<label-instance-matches.ttl.gz> ,
		<label-meronym-matches.ttl.gz>,
		<label-neargloss-matches.ttl.gz> ,
		<label-parent-matches.ttl.gz> ,
		<label-unique-matches.ttl.gz> ,
		<nearlabel-matches.ttl.gz> ,
		<glossmatches-m.ttl.gz> .

<wn30-wn20-mappings-sense>
	a void:Linkset ;
	dcterms:title "synset-level mappings from Wordnet 3.0 to 2.0, created by Mark using the Princeton WordSense mappings" ;
	lib:source <http://purl.org/vocabularies/princeton/wn30/> ;
	void:dataDump
		<synset-matches-based-on-multiple-sense-mappings-princeton.ttl.gz> ,
		<synset-matches-based-on-single-sense-mappings-princeton.ttl.gz> .

It's a complex affair consisting of many steps, but the major step (contributing 87% of all matches) is glossmatches-m.ttl that looks like

    wn30:synset-wrought_iron-noun-1 terms:replaces instances:synset-wrought_iron-noun-1 .

And looking at wordnet-synset.ttl, we find the required wn30 ID:

    wn30:synset-wrought_iron-noun-1 wn20schema:synsetId 114802262 .
AAT-Wikidata Sheets[edit]

After much querying and manual cleaning (over a day of effort), I made some sheets in this google folder:

  • AAT-DBpedia-Babelnet.xlsx: 3324 potential matches, fairly clean, but need checking by more people
  • AAT-DBpedia-Babelnet-80-judged.xlsx: example of correct & incorrect matches
  • AAT-Wikidata-25-judged.xlsx: example of correct & incorrect matches on Mix-n-Match

T.seppelt (talk) 21:00, 18 February 2016 (UTC) Vladimir Alexiev (talk) 11:59, 13 March 2017 (UTC) GerardM (talk) 15:58, 26 March 2017 (UTC) Jonathan Groß (talk) 17:52, 26 March 2017 (UTC) Andy Mabbett (Pigsonthewing); Talk to Andy; Andy's edits Jneubert (talk) 13:47, 29 April 2017 (UTC) Framawiki (please notify !) (talk)

Pictogram voting comment.svg Notified participants of WikiProject Authority control: I need your help!

User:Zolo
Jane023 (talk) 08:50, 30 May 2013 (UTC)
User:Vincent Steenberg
User:Kippelboy
User:Shonagon
Marsupium (talk) 13:46, 18 October 2013 (UTC)
GautierPoupeau (talk) 16:55, 9 January 2014 (UTC)
Multichill (talk) 19:13, 8 July 2014 (UTC)
Susannaanas (talk) 11:32, 12 August 2014 (UTC) I want to synchronize the handling of maps with this initiative
Mushroom (talk) 00:10, 24 August 2014 (UTC)
Jheald (talk) 17:09, 9 September 2014 (UTC)
Spinster (talk) 15:16, 12 September 2014 (UTC)
PKM (talk) 21:16, 8 October 2014 (UTC)
Vladimir Alexiev (talk) 17:12, 7 January 2015‎ (UTC)
Sic19 (talk) 21:12, 19 February 2016 (UTC)
Wittylama (talk) 13:13, 22 February 2017 (UTC)
Armineaghayan (talk) 08:40, 10 March 2017 (UTC)
Hannolans (talk) 18:36, 16 April 2017 (UTC)
User:Martingggg
Zeroth (talk) 02:21, 4 June 2018 (UTC)
User:7samurais
User:mrtngrsbch
Pictogram voting comment.svg Notified participants of WikiProject Visual arts: Yours too!

  • Do some checks (add your initials in column "check")
  • Add Q numbers to the sheet
  • Merge WD items that already have AAT ID (P1014) (there are 8477) to the sheet to compare the matches (or remove them from the sheet if you're quite confident)

I could post the sheet as QuickStatements, but I think there are still 10% incorrect matches, especially for Styles and Periods (see Wikidata talk:WikiProject Visual arts/Item structure/Art movements. --Vladimir Alexiev (talk) 16:02, 7 March 2017 (UTC)

AAT-LCSH coreferencing[edit]

445 AAT-LCSH coreferences made by Getty editors.

400 of them are on the Getty LOD site (see query below), 45 are newly extracted

select * {
  ?x skos:exactMatch|skos:closeMatch ?y.
  ?x skos:inScheme aat:
  filter not exists {?y skos:inScheme aat:}}

Geonames Feature Code[edit]

T.seppelt (talk) 21:00, 18 February 2016 (UTC) Vladimir Alexiev (talk) 11:59, 13 March 2017 (UTC) GerardM (talk) 15:58, 26 March 2017 (UTC) Jonathan Groß (talk) 17:52, 26 March 2017 (UTC) Andy Mabbett (Pigsonthewing); Talk to Andy; Andy's edits Jneubert (talk) 13:47, 29 April 2017 (UTC) Framawiki (please notify !) (talk)

Pictogram voting comment.svg Notified participants of WikiProject Authority control Kopiersperre Jklamo ArthurPSmith S.K. Givegivetake fnielsen rjlabs ChristianKl Vladimir Alexiev User:Pintoch Parikan User:Cardinha00 User:zuphilip MB-one User:Simonmarch User:Jneubert Mathieudu68 User:Kippelboy User:Datawiki30

Pictogram voting comment.svg Notified participants of WikiProject Companies Beat Estermann
Vladimir Alexiev
Ilya
Fralambert
Sadads
Astinson
Strakhov
Zeromonk
Spinster
Wittylama
Daniel Mietchen
Susannaanas
Sic19
Jason.nlw
Carlojoseph14
YULdigitalpreservation
MB-one
Ouvrard
MartinPoulter
Missvain
VIGNERON
Ainali
Birk Weiberg
Pmt
Mauricio V. Genta
Smallison
ProtoplasmaKid
2le2im-bdc
Rodrigo Tetsuo Argenton
Ivanhercaz
VisbyStar

Pictogram voting comment.svg Notified participants of WikiProject Cultural heritage User:Zolo
Jane023 (talk) 08:50, 30 May 2013 (UTC)
User:Vincent Steenberg
User:Kippelboy
User:Shonagon
Marsupium (talk) 13:46, 18 October 2013 (UTC)
GautierPoupeau (talk) 16:55, 9 January 2014 (UTC)
Multichill (talk) 19:13, 8 July 2014 (UTC)
Susannaanas (talk) 11:32, 12 August 2014 (UTC) I want to synchronize the handling of maps with this initiative
Mushroom (talk) 00:10, 24 August 2014 (UTC)
Jheald (talk) 17:09, 9 September 2014 (UTC)
Spinster (talk) 15:16, 12 September 2014 (UTC)
PKM (talk) 21:16, 8 October 2014 (UTC)
Vladimir Alexiev (talk) 17:12, 7 January 2015‎ (UTC)
Sic19 (talk) 21:12, 19 February 2016 (UTC)
Wittylama (talk) 13:13, 22 February 2017 (UTC)
Armineaghayan (talk) 08:40, 10 March 2017 (UTC)
Hannolans (talk) 18:36, 16 April 2017 (UTC)
User:Martingggg
Zeroth (talk) 02:21, 4 June 2018 (UTC)
User:7samurais
User:mrtngrsbch
Pictogram voting comment.svg Notified participants of WikiProject Visual arts


GeoNames feature code (P2452) is applied only 33 times (see disscussion and archive, while there are 669 codes on Geonames. I'll ask Magnus to add the Geonames list to Mix-n-Match.

http://www.geonames.org/ontology/mappings_v3.01.rdf has the following mappings:

     32 dbo    http://dbpedia.org/ontology/
      5 frgeo  http://rdf.insee.fr/geo/
     79 lgdo   http://linkedgeodata.org/ontology/
     31 schema http://schema.org/

Can we use them somehow to push this coreferencing further?

I'm a little wary about importing these wholescale, because geonames in not a CC0 database. It's one thing to be providing external links to GeoNames, it's another to be importing data.
I did look at these values recently for English places with geonames links that are marked as both village (Q532) and civil parish (Q1115575) (see eg Abberton, Worcestershire (Q3137539) for an example), to identify which Geonames link corresponded to which role; but I purposely decided not to add a GeoNames feature code (P2452) statement.
It might be useful to be able to map the codes to Q-numbers here, to facilitate sanity checking of exisiting or proposed co-references. However, even then there are difficulties -- for example, I found that PPLA3 or PPLA4 at Geonames didn't necessarily match to distinctions we would want to make in a instance of (P31) here. Jheald (talk) 14:02, 13 March 2017 (UTC)

Supplement Wikidata items with properties from authorities (GND in particular)[edit]

Data from DifferentiatedPersons of GND can be used to fill missing properties of according items, e.g.,

  • date of birth/death (directly from gnd:dateOfBirth and gndo:dateOfDeath, for entries following YYYY or YYYY-MM-DD - everything else to be skipped)
  • affiliation (P1416) can be obtained by a join of gndo:affiliation to wd organizations (may be sparse currently, but can be repeated later on)
  • country (country (P17) or country of citizenship (P27)??) requires translation from gndo:geographicAreaCode, which refers to a customized code table derived from ISO 3166 (not part of GND) (table (pdf), rules)
  • aliases - require filtering of gndo:variantNameForThePerson, which carry no language tag, re. script and presumed language (would Lingua::Identify work here?)

For appropriate source statements see project chat -- Jneubert (talk) 06:35, 21 May 2017 (UTC) (with thanks to User:MisterSynergy and User:ChristianKl)

I am a big fan of standards but the ISO 3166 is used for modern countries, it does not as a consequence give the "nationality" of people who did precede a country. Thanks, GerardM (talk) 07:03, 21 May 2017 (UTC)
NB yes there are some, but at Wikidata we know about many more former countries. GerardM (talk) 07:13, 21 May 2017 (UTC)

A "sibling" of Mix-n-Match now imports birth/death dates from authority files: https://www.wikidata.org/wiki/User:Magnus_Manske/Mix%27n%27match_date_import. Also see discussion about this in relation to Getty ULAN: https://groups.google.com/forum/#!topic/gettyvocablod/TkdelW9RP1g --Vladimir Alexiev (talk) 09:38, 2 October 2017 (UTC)

Property proposal for applying SKOS mapping relations to "external identifiers"[edit]

In order be able to map a thesaurus more completely, and - more general - to make Wikidata fit as a linking hub for knowledge organiziation systems, I've proposed a new property which allows to qualify individual links by properties of type "external identifier" as in-exact (close/broad/narrow/related) match.

Please feel free to comment at https://www.wikidata.org/wiki/Wikidata:Property_proposal/mapping_relation_type.

Cheers, Jneubert (talk) 12:27, 28 August 2017 (UTC)

Grant proposal soweego[edit]

T.seppelt (talk) 21:00, 18 February 2016 (UTC) Vladimir Alexiev (talk) 11:59, 13 March 2017 (UTC) GerardM (talk) 15:58, 26 March 2017 (UTC) Jonathan Groß (talk) 17:52, 26 March 2017 (UTC) Andy Mabbett (Pigsonthewing); Talk to Andy; Andy's edits Jneubert (talk) 13:47, 29 April 2017 (UTC) Framawiki (please notify !) (talk)

Pictogram voting comment.svg Notified participants of WikiProject Authority control

There is a new grant proposal soweego for authority control. See discussion at https://meta.wikimedia.org/wiki/Grants:Project/Hjfocs/soweego#Endorsements.

I've considered it seriously but I think it doesn't address the main problem, see a list of 11 considerations (which can also be read as a sort of programme for next important steps for WD authority control). Please express your opinion there. --Vladimir Alexiev (talk) 10:19, 2 October 2017 (UTC)

Where is this discussion with 11 considerations? Thanks, GerardM (talk) 12:22, 2 October 2017 (UTC)

Remove Obsolete Getty Vocabularies IDs[edit]

Getty Vocabs have 8k obsolete subjects (identifiers). You can find them at http://vocab.getty.edu/sparql with a query like

select * {
  ?old a gvp:ObsoleteSubject; skos:inScheme aat: .
  optional {?old dct:isReplacedBy ?new}
}

Here are the numbers: I can put up the files somewhere: obsolete-AAT-2106.tsv, obsolete-TGN-1016.tsv, obsolete-ULAN-5574.tsv.

T.seppelt (talk) 21:00, 18 February 2016 (UTC) Vladimir Alexiev (talk) 11:59, 13 March 2017 (UTC) GerardM (talk) 15:58, 26 March 2017 (UTC) Jonathan Groß (talk) 17:52, 26 March 2017 (UTC) Andy Mabbett (Pigsonthewing); Talk to Andy; Andy's edits Jneubert (talk) 13:47, 29 April 2017 (UTC) Framawiki (please notify !) (talk)

Pictogram voting comment.svg Notified participants of WikiProject Authority control Any takers to replace any old values of AAT ID (P1014) View with SQID, TGN ID (P1667) View with SQID, ULAN ID (P245) View with SQID with the new values from the respective files?

@Magnus Manske: rdf:type gvp:ObsoleteSubject should be removed from Mix-n-Match consideration.

Done, but only changed 5 entries for the ~2100 obsolete ones, the others were either already N/A or matched with an item. --Magnus Manske (talk) 15:22, 21 February 2018 (UTC)
Data should not be removed, but - if necessary - marked as deprecated, with a qualifier giving the reason for deprecation. Andy Mabbett (Pigsonthewing); Talk to Andy; Andy's edits 11:02, 26 February 2018 (UTC)

Swedish National Library moves direction Linked data[edit]

Monday june 11 the Swedish National Library is moving in direction Linked data and release a new system called LIBRIS XL ==>

  1. We have a new identifier LIBRIS-URI Libris-URI (P5587) in Wikidata

Open issues are if we need to "upgraded" to support the new identifier in other systems. I have found

  1. User:Magnus_Manske/authority_control.js
    1. tool to read from VIAF.org and populate WD
  2. User:Magnus_Manske/Mix'n'match_date_import
  3. wdmapper

See also:

- Salgo60 (talk) 14:07, 10 June 2018 (UTC)

Status update
A project wmse-library-data-2018 done by Alicia_Fagerving_(WMSE) (talkcontribslogs) is converting and adding about 60 000 identifiers to Wikidata....
I sent in today a question to VIAF asking about plans they have to start supporting Libris-URI (P5587) - Salgo60 (talk) 06:54, 16 September 2018 (UTC)

VIAF Games[edit]

T.seppelt (talk) 21:00, 18 February 2016 (UTC) Vladimir Alexiev (talk) 11:59, 13 March 2017 (UTC) GerardM (talk) 15:58, 26 March 2017 (UTC) Jonathan Groß (talk) 17:52, 26 March 2017 (UTC) Andy Mabbett (Pigsonthewing); Talk to Andy; Andy's edits Jneubert (talk) 13:47, 29 April 2017 (UTC) Framawiki (please notify !) (talk)

Pictogram voting comment.svg Notified participants of WikiProject Authority control Ash Crow
Dereckson
Harmonia Amanda
Hsarrazin
Jura
Чаховіч Уладзіслаў
Sascha
Joxemai
Place Clichy
Branthecan
Azertus
ToJack
Jon Harald Søby
PKM
Pmt
Sight Contamination
MaksOttoVonStirlitz
BeatrixBelibaste
ajayi adeniyi
Moebeus
Dcflyer
Pictogram voting comment.svg Notified participants of WikiProject Names User:Zolo
Jane023 (talk) 08:50, 30 May 2013 (UTC)
User:Vincent Steenberg
User:Kippelboy
User:Shonagon
Marsupium (talk) 13:46, 18 October 2013 (UTC)
GautierPoupeau (talk) 16:55, 9 January 2014 (UTC)
Multichill (talk) 19:13, 8 July 2014 (UTC)
Susannaanas (talk) 11:32, 12 August 2014 (UTC) I want to synchronize the handling of maps with this initiative
Mushroom (talk) 00:10, 24 August 2014 (UTC)
Jheald (talk) 17:09, 9 September 2014 (UTC)
Spinster (talk) 15:16, 12 September 2014 (UTC)
PKM (talk) 21:16, 8 October 2014 (UTC)
Vladimir Alexiev (talk) 17:12, 7 January 2015‎ (UTC)
Sic19 (talk) 21:12, 19 February 2016 (UTC)
Wittylama (talk) 13:13, 22 February 2017 (UTC)
Armineaghayan (talk) 08:40, 10 March 2017 (UTC)
Hannolans (talk) 18:36, 16 April 2017 (UTC)
User:Martingggg
Zeroth (talk) 02:21, 4 June 2018 (UTC)
User:7samurais
User:mrtngrsbch
Pictogram voting comment.svg Notified participants of WikiProject Visual arts Beat Estermann
Vladimir Alexiev
Ilya
Fralambert
Sadads
Astinson
Strakhov
Zeromonk
Spinster
Wittylama
Daniel Mietchen
Susannaanas
Sic19
Jason.nlw
Carlojoseph14
YULdigitalpreservation
MB-one
Ouvrard
MartinPoulter
Missvain
VIGNERON
Ainali
Birk Weiberg
Pmt
Mauricio V. Genta
Smallison
ProtoplasmaKid
2le2im-bdc
Rodrigo Tetsuo Argenton
Ivanhercaz
VisbyStar

Pictogram voting comment.svg Notified participants of WikiProject Cultural heritage

Anyone interested in Authority Control, please read this section. It provides info about VIAF volumetrics, WD-VIAF volumetrics (very low!), correlation VIAF code - WD entity&prop, and an idea how to proliferate identifiers.

VIAF Volumetrics[edit]

We at Ontotext are developing a VIAF reconciliation service, and here are some statistics about their dump 2018-07 (note: this dump is missing links, see below, so it may be missing entities too):

Type Count Comment
skos:Concept 54,741,718 Contributor (per-source) node, 1.7x per entity (cluster)
foaf:Document & ont:InformationResource 32,254,690 Entities (clusters), or RDF docs describing each cluster
schema:Person 21,034,181 persons
schema:Organization 4,359,408 organizations. Pure orgs (not places): 4,249,192
schema:Place 972,703 places
schema:Place & schema:Organization 110,216 Eg viaf:233665742 Chile: country and its government
schema:CreativeWork 4,110,185 creative work (uniform title)
bib:Agent 2,163,020 "placeholder URI in need of further matching"
wd:Q1387388 2,148,663 "chimera: mythological hybrid creature combining body parts of more than one real species. VIAF URI corrupted by an 'undifferentiate name' and should be treated as unusable"
pto:Pseudonym 4,639 A person with a pseudonym

Wikidata to VIAF Coreferencing[edit]

Number of WD entities with VIAF id, per class:

class count
human 1131171
commune of France 9139
municipality of Germany 8631
band 8316
business 7574
ortsteil 6066
comune of Italy 5580
human settlement 4061
city 3814
organization 2811
university 2620
museum 2454
city of the United States 2187
municipality of Switzerland 2184
river 2105
town 2068
village 2016
church 1984
book 1977
municipality of Spain 1971
municipality of Austria 1914
film 1774
association 1575
art museum 1499

WD has 4,479,012 humans. So about 5.2% of VIAF persons are coreferenced, and 25.2% of WD persons are coreferenced.

WD has 1,443,285 organizations, of which 121,404 are coreferenced, so 2.9% of VIAF orgs are co-referenced, and 8.4% of WD orgs. (Query fixed by @ArthurPSmith:)

select (count(*) as ?c) {
  ?x wdt:P214 []; wdt:P31/wdt:P279* wd:Q43229; 
}

I believe a similar percent of places are coreferenced.

--Vladimir Alexiev (talk) 16:15, 22 August 2018 (UTC)

VIAF Source Codes[edit]

To do coreferencing work, we need to correlate VIAF source codes (eg JPG) to WD props (eg P245). We can find some of them with this query, and I found the others at the list of sources on http://viaf.org/.

select ?entity ?entityLabel ?prop ?propLabel ?code ?countryLabel {
  ?prop wdt:P1552 wd:Q26921380.
  optional {
    ?prop wdt:P1629 ?entity
    optional {?entity wdt:P17 ?country}
  }
  optional {
    ?prop wdt:P3303 ?formatter 
    filter(regex(?formatter,"https://viaf.org/processed/(.*)\\|\\$1"))
    bind(replace(?formatter,"https://viaf.org/processed/(.*)\\|\\$1","$1") as ?code)
  }
  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en". }
} order by ?propLabel

The sources are listed in the next section.

--Vladimir Alexiev (talk) 16:15, 22 August 2018 (UTC)

VIAF Links per Source[edit]

viaf-links-count:

  • 2016-12: 86,698,786 links
  • 2018-07: 54,741,718 links
  • 2018-08:

This decrease is strange. The 2018-07 VIAF dump is a fluke, don't use it

  • LC dropped from 16M to 10M, and DNB from 26M to 9M.
  • VIAF dropped Wikipedia links (not fatal since WD has all possible WP links)
  • VIAF lists Wikidata on their website, but such links are not in the dump (checked RDF data)
  • VIAF dropped these sources: GeoNames, IMAGINE, ORCID

2018-08 fixes these problems.

  • Also, VIAF added some sources: BLBNB, DE663, ERRR
2016-12 2018-07 2018-08 code entity property country
359497 366280 369712 B2Q Q39628 P3280, P1823 Canada>Quebec
369985 369985 369987 BAV Q213678 P1017 Vatican
286192 300088 BLBNB Q948882 P4619 Brazil
104455 1795284 1797032 BIBSYS Q4584301 P1015 Norway
365936 185314 370676 BNC Q8342938 P1273 Spain>Catalonia
412032 205976 412034 BNCHL Q19896851 P1890, P1966 Chile
620739 642770 646521 BNE Q750403 P950 Spain
4562204 2395213 4806862 BNF Q193563 P268 France
29007 29007 29009 BNL Q856651 !!! Luxembourg
54935 79128 79991 CYT Q618340 P1048 Taiwan
111686 111686 DE663 Q2178828 !!? intl (RISM: musical sources)
282952 162634 327462 DBC Q12307383 P3846 Denmark
26490921 8866855 17785894 DNB Q36578 P227 DE, AT, CH
46201 52350 52350 EGAXA Q501851 P1309 Egypt
57791 57794 ERRR Q609471 !!! Estonia
178835 179726 180164 FAST Q3294867 P2163 intl (OCLC)
152003 165845 GeoNames Q830106 P1566 intl (places)
343166 173713 347426 ICCU Q576951 P396, P5485 Italy
9941 9964 IMAGINE !!?
7741093 8187394 8187405 ISNI Q423048 P213 intl (authors)
241631 264833 264834 JPG Q2494649 P245 US Getty
240857 331791 331797 KRNLK Q495005 P5034 Korea
718263 726993 730007 LAC Q913250 P1670 Canada
16256543 10156417 16912675 LC Q13219454 P244 US, UK, MX, ZA, NZ
186566 205260 205260 LNB Q1133733 P1368 Latvia
14768 15876 15879 LNL Q2901488 !!! Lebanon
6299 8567 8569 MRBNR Q2901478 !!! Morocco
227699 227651 227707 N6I Q1672830 P1946 Ireland
1079274 1114415 1116232 NDL Q2553334 P349 Japan
1617979 1712963 1712980 NII Q10726338 P271, P1739, P2409 Japan
817783 872953 879014 NKC Q13550863 P691 Czech Rep
1036908 1111676 1114335 NLA Q623578 P409 Australia
608 3940 3940 NLB Q890364 P3988 Singapore
850304 1788698 1810067 NLI Q188915 P949 Israel
1219045 1482467 1494282 NLP Q856423 P1695 Poland
245488 147268 294546 NLR Q1048694 P947, P1973 Russia
508256 537830 539669 NSK Q631375 P1375 Croatia>Zagreb
67454 33727 67454 NSZL Q1063819 P951 Hungary
2650671 2700697 2703554 NTA Q1526131 P1006 Netherlands
3090182 1731205 3482894 NUKAT Q11789729 P1207 Poland>WarsawU
7389 17359 ORCID Q51044 P496 intl (researchers)
1228 1228 1228 PERSEUS Q639661 !!! intl (ancient places)
404594 423917 423920 PTBNP Q245966 P1005 Portugal
237240 253180 253914 RERO Q3456783 P3065 Switzerland
205630 215367 215798 SELIBR Q1798125 P906 Sweden
209 209 209 SRP !!? Syriac Reference Portal
2927879 3281884 3300486 SUDOC Q47757534 P269 France academic
75989 124465 124466 SWNL P1255? Switzerland
85956 85958 UIY Q627423 !!! Iceland (NULI)
8768 10452 10452 VLACC !!! Flemish Public Libraries
125635 126826 W2Z Q924551 P1439?, P1847? Norway
1286035 1570365 WKP Wikidata entities
6272821 7105991 Wikipedia Wikipedia links
528 1102 1140 XA Extended Authorities n/a OCLC manual
2069996 2026405 2036656 XR Extended Relationships n/a OCLC manual

--Vladimir Alexiev (talk) 10:13, 24 August 2018 (UTC)

VIAF Help Needed[edit]

T.seppelt (talk) 21:00, 18 February 2016 (UTC) Vladimir Alexiev (talk) 11:59, 13 March 2017 (UTC) GerardM (talk) 15:58, 26 March 2017 (UTC) Jonathan Groß (talk) 17:52, 26 March 2017 (UTC) Andy Mabbett (Pigsonthewing); Talk to Andy; Andy's edits Jneubert (talk) 13:47, 29 April 2017 (UTC) Framawiki (please notify !) (talk)

Pictogram voting comment.svg Notified participants of WikiProject Authority control Ash Crow
Dereckson
Harmonia Amanda
Hsarrazin
Jura
Чаховіч Уладзіслаў
Sascha
Joxemai
Place Clichy
Branthecan
Azertus
ToJack
Jon Harald Søby
PKM
Pmt
Sight Contamination
MaksOttoVonStirlitz
BeatrixBelibaste
ajayi adeniyi
Moebeus
Dcflyer
Pictogram voting comment.svg Notified participants of WikiProject Names User:Zolo
Jane023 (talk) 08:50, 30 May 2013 (UTC)
User:Vincent Steenberg
User:Kippelboy
User:Shonagon
Marsupium (talk) 13:46, 18 October 2013 (UTC)
GautierPoupeau (talk) 16:55, 9 January 2014 (UTC)
Multichill (talk) 19:13, 8 July 2014 (UTC)
Susannaanas (talk) 11:32, 12 August 2014 (UTC) I want to synchronize the handling of maps with this initiative
Mushroom (talk) 00:10, 24 August 2014 (UTC)
Jheald (talk) 17:09, 9 September 2014 (UTC)
Spinster (talk) 15:16, 12 September 2014 (UTC)
PKM (talk) 21:16, 8 October 2014 (UTC)
Vladimir Alexiev (talk) 17:12, 7 January 2015‎ (UTC)
Sic19 (talk) 21:12, 19 February 2016 (UTC)
Wittylama (talk) 13:13, 22 February 2017 (UTC)
Armineaghayan (talk) 08:40, 10 March 2017 (UTC)
Hannolans (talk) 18:36, 16 April 2017 (UTC)
User:Martingggg
Zeroth (talk) 02:21, 4 June 2018 (UTC)
User:7samurais
User:mrtngrsbch
Pictogram voting comment.svg Notified participants of WikiProject Visual arts Beat Estermann
Vladimir Alexiev
Ilya
Fralambert
Sadads
Astinson
Strakhov
Zeromonk
Spinster
Wittylama
Daniel Mietchen
Susannaanas
Sic19
Jason.nlw
Carlojoseph14
YULdigitalpreservation
MB-one
Ouvrard
MartinPoulter
Missvain
VIGNERON
Ainali
Birk Weiberg
Pmt
Mauricio V. Genta
Smallison
ProtoplasmaKid
2le2im-bdc
Rodrigo Tetsuo Argenton
Ivanhercaz
VisbyStar

Pictogram voting comment.svg Notified participants of WikiProject Cultural heritage


  • If you sort by col "property" and look for "!!!", this represents VIAF sources that don't have corresponding WD properties (which should be marked as VIAF component (Q26921380). Wikidata:Property_proposal/Authority_control#missing_VIAF_components suggests to create them.
  • Please help me verify props in the table above marked "?" or cases where multiple props are listed (although VIAF has both persons and works, maybe from a certain source it takes only persons)
  • Do you think we should expand the Qnnn and Pnnn above using the respective macros, so they show up with their names?
  • Do you think we should record col "code" in WD in some way? A few of these are available as third-party formatter URL (P3303), but most aren't. It took me a few hours to correlate all VIAF-component sources against WD, and it wasn't fun. Guess we can add it as qualifier? Eg
SUDOC authorities ID (P269) has quality (P1552) VIAF component (Q26921380)
code (P3295) "SUDOC"
  • And most importantly, help me with "id proliferation" as described below. Please comment on that section!

--Vladimir Alexiev (talk) 08:25, 23 August 2018 (UTC)

ID Proliferation with VIAF[edit]

See the low WD coreference percentages in #VIAF Volumetrics. I believe the situation is not much different for all VIAF-contributing sources.

What can we do to increase them? There is a simple way: proliferate IDs between all of these authorities.

--Vladimir Alexiev (talk) 16:15, 22 August 2018 (UTC)

@Vladimir Alexiev: I get a "query is malformed" error on your ULAN without VIAF... - PKM (talk) 21:01, 22 August 2018 (UTC)
@PKM: fixed. And somehow the number now is lower?? --Vladimir Alexiev (talk) 08:03, 23 August 2018 (UTC)
@Vladimir Alexiev: Thanks. Maybe someone jumped in to fix some? It wasn’t me (yet). - PKM (talk) 19:15, 23 August 2018 (UTC)

Introducing mapping relation type (P4390) for full SKOS mappings[edit]

For knowledge organization systems which do not deal with clearly defined entities (such as human (Q5)), you often find in-exact relations between the external entity and matching Wikidata items. E.g., the STW concept Yugoslavia (until 1990) is not an exact match to Yugoslavia (Q36704), which is described as "1918–1992 country in Southeastern and Central Europe". For STW's Executive selection, Assessment center (Q265558) is closely related, but categorically different (process vs. instrument). While in the latter case it might be useful to create an exactly matching item, in the former it clearly would be not.

Within the domain of traditional knowledge organization system, different mapping relations (such as "close match") have been used to cover such situations. The "external id" properties of Wikidata lacked such expressiveness, till the mapping relation type (P4390) property was introduced. The property, to be used exclusively as a qualifier on external-id properties, allows to more precisely define relations as exact match (Q39893449), close match (Q39893184), broad match (Q39894595), narrow match (Q39893967) or related match (Q39894604). These relation types reflect the according SKOS mapping properties.

Since its introduction in October last year, the property has seen some uptake, particularly in the biomedial field. The ongoing mapping of STW Thesaurus for Economics (Q26903352) to Wikidata is based on qualified relations (state of the mapping, shown as SKOS). The effort aims at creating a finally complete mapping of the STW descriptors.

Workflow for the mapping process[edit]

Our - still experimental - workflow for the STW mapping is as follows:

  1. Use Mix-n-match catalogs for each sub-thesaurus (#507, #1259/#1260) to assign STW descriptor IDs one-by-one to Wikidata items.
    This sounds simple, yet often reveals qualitiy issues. Some of these, such as obvious duplicates in Wikidata, can be resolved immediately. Sometimes an ugly mess shows up, which can only be reported to be solved by the community later on. Of course, quality flaws in STW might be revealed also. During this step, it may also be advisable to take notes about other items, which are not the closest ones and not linked via external-id entry in Wikidata, but may be worth linking from the side of the external vocabulary.
    • For non-exact relationships, immediately open the newly linked item and manually qualify STW Thesaurus for Economics ID (P3911).
    • For STW descriptors which are lacking a counterpart in Wikidata and would make sense there, add an item semi-automatically, with exact match (Q39893449) to the STW ID and all avaialable information from the thesaurus (more on that later).
  2. Assign exact match (Q39893449) to the remaining unqualified STW Thesaurus for Economics ID (P3911) via Quickstatements, with the input produced by a script executing a SPARQL query. This step can be executed multiple times during the mapping process for one sub-thesaurus, in order to keep the list of unqualified entries short.

The sequence within the Mix-n-match input file turned out to be crucial for a smooth one-by-one workflow. We sorted the generated M-n-m input by the minimal notation of attached subject categories for the descriptors, and within that alphabetically.

Quality control for mappings using mapping relation types[edit]

Maintenance and qualitiy control on a mapping have to take into account that multiple external-id values for a Wikidata item, or one external-id linked to multiple Wikidata items are possible and may perfectly make sense with in-exact mapping relations (e.g., STWs Appenzell is a "broad match" to Appenzell Ausserrhoden (Q12079) and Appenzell Innerrhoden (Q12094)). This is not reflected in the "single value" and "distinct values" constraints. Therefore, we defined a number of QS reports for the STW mapping to catch anomalies specifically in qualified mappings.

Feedback welcome[edit]

We are interested to exchange experiences with others who are mapping KOS to Wikidata, possibly with different workflows or other tools used. The reports and scripts linked above are meant to be customizable, we'd be happy to receive suggestions for improvment or github pull requests. -- Jneubert (talk) 15:09, 29 August 2018 (UTC)

JakobVoss (talk) Vladimir Alexiev (talk) 09:27, 5 July 2016 (UTC) TomT0m author  TomT0m / talk page 20:15, 21 September 2016 (UTC) Jneubert (talk) 09:22, 29 January 2018 (UTC)


Pictogram voting comment.svg Notified participants of WikiProject KOS

Item creation from a thesaurus concept via Quickstatements[edit]

During the above mentioned mapping process from STW Thesaurus for Economics (Q26903352), we sometimes want to create new items in Wikidata. With the "New item" button in Mix-n-match, only rudimentary information can be transferred to the new item. Therefore we generated a list of all not-yet-mapped STW descriptors, formatted for Quickstatements input. It includes labels and aliases (skos:preLabel/skos:altLabel) in all available languages, as well an instance-of economical concept (Q29028649) statement, sourced from the STW descriptor, optionally a link to the according GND concept, derived from the STW/GND mapping, rarely a description (skos:scopeNote), but always of course a STW Thesaurus for Economics ID (P3911) link.

The workflow is - during working through a mix-n-match list - for any missing concept simply to copy & paste the complete set of QS statements into the QS input window, removing aliases which are not appropriate for Wikidata (such as "oil platform" for "offshore industry"), and running the statements. The list is sorted exactly like the mix-n-match list and recreated every hour, so the same items are on top of both lists, and every case solved by either linking or creating an item disappears automatically from the list. In our experiences, this works quite smoothly.

If others want to adapt such a workflow for other vocabularies - thanks to SKOS and LOD standards that shouldn't be too difficult -, here is the script for generating the list, and the query called with it. -- Feedback, as always, welcome. Jneubert (talk) 12:11, 5 September 2018 (UTC)

History[edit]

Please add here references, blogs etc on the topic.

https://twitter.com/hashtag/coreferencing: tweet using tag #coreferencing. Tweets on involving Getty, British Museum thesauri, some fancy shots...

Participants[edit]

Beat Estermann
Vladimir Alexiev
Ilya
Fralambert
Sadads
Astinson
Strakhov
Zeromonk
Spinster
Wittylama
Daniel Mietchen
Susannaanas
Sic19
Jason.nlw
Carlojoseph14
YULdigitalpreservation
MB-one
Ouvrard
MartinPoulter
Missvain
VIGNERON
Ainali
Birk Weiberg
Pmt
Mauricio V. Genta
Smallison
ProtoplasmaKid
2le2im-bdc
Rodrigo Tetsuo Argenton
Ivanhercaz
VisbyStar

Pictogram voting comment.svg Notified participants of WikiProject Cultural heritage Please become members of this project!


[+] Add yourself to the list

The participants listed below can be notified using the following template in discussions:

{{Ping project|Authority control}}

Related projects[edit]