User:TweetsFactsAndQueries/A Guide To WDQS

From Wikidata
Jump to: navigation, search

WDQS, the Wikidata Query Service, is an awesome tool to answer many questions you might have, and this guide will teach you how to use it.

Before we start[edit]

This guide looks very long, perhaps intimidatingly so. Please don’t let that scare you away! SPARQL is complicated, but the simple basics will already get you a long way – if you want, you can stop reading after #Our first query, and you’ll already know enough to write many interesting queries. After that, the sections just add information about more topics that you can use to write different queries. Each of them will empower you to write even more awesome queries, but none of them are necessary – you can stop reading at any point and hopefully still walk away with a lot of useful knowledge!

Also, if you’ve never heard of Wikidata, SPARQL, or WDQS before, here’s a short explanation of those terms:

SPARQL basics[edit]

A simple SPARQL query looks like this:

SELECT ?a ?b ?c
WHERE
{
  x y ?a.
  m n ?b.
  ?b f ?c.
}

The SELECT clause lists variables that you want returned (variables start with a question mark), and the WHERE clause contains restrictions on them, mostly in the form of triples. All information in Wikidata (and similar knowledge databases) is stored in the form of triples; when you run the query, the query service tries to fill in the variables with actual values so that the resulting triples appear in the knowledge database, and returns one result for each combination of variables it finds.

A triple can be read like a sentence (which is why it ends with a period), with a subject, a predicate, and an object:

SELECT ?fruit
WHERE
{
  ?fruit hasColor yellow.
  ?fruit tastes sour.
}

The results for this query could include, for example, “lemon”. In Wikidata, most properties are “has”-kind properties, so the query might instead read:

SELECT ?fruit
WHERE
{
  ?fruit color yellow.
  ?fruit taste sour.
}

which reads like “?fruit has color ‘yellow’” (not?fruit is the color of ‘yellow’” – keep this in mind for property pairs like “parent”/“child”!).

However, that’s not a good example for WDQS. Taste is subjective, so Wikidata doesn’t have a property for it. Instead, let’s think about parent/child relationships, which are mostly unambiguous.

Our first query[edit]

Suppose we want to list all children of the baroque composer Johann Sebastian Bach. Using pseudo-elements like in the queries above, how would you write that query?

Hopefully you got something like this:

SELECT ?child
WHERE
{
  # either this...
  ?child parent Bach.
  # or this...
  ?child father Bach.
  # or this.
  Bach child ?child.
  # (note: everything after a ‘#’ is a comment and ignored by WDQS.)
}

The first two triples say that the ?child must have the parent/father Bach; the third says that Bach must have the child ?child. Let’s go with the second one for now.

So what remains to be done in order to turn this into a proper WDQS query? On Wikidata, items and properties are not identified by human-readable names like “father” (property) or “Bach” (item). (For good reason: “Johann Sebastian Bach” is also the name of a German painter, and “Bach” might also refer to the surname, the French commune, the Mercury crater, etc.) Instead, Wikidata items and properties are assigned an identifier. To find the identifier for an item, we search for the item and copy the Q-number of the result that sounds like it’s the item we’re looking for (based on the description, for example). To find the identifier for a property, we do the same, but search for “P:search term” instead of just “search term”, which limits the search to properties. This tells us that the famous composer Johann Sebastian Bach is Q1339, and the property to designate an item’s father is P:P22.

And last but not least, we need to include prefixes. For simple WDQS triples, items should be prefixed with wd:, and properties with wdt:. (But this only applies to fixed values – variables don’t get a prefix!)

Putting this together, we arrive at our first proper WDQS query:

SELECT ?child
WHERE
{
# ?child  father   Bach
  ?child wdt:P22 wd:Q1339.
}

Try it!

Click that “Try it” link, then “Run” the query on the WDQS page. What do you get?

child
wd:Q57225
wd:Q76428

Well that’s disappointing. You just see the identifiers. You can click on them to see their Wikidata page (including a human-readable label), but isn’t there a better way to see the results?

Well, as it happens, there is! (Aren’t rhetorical questions great?) If you include the magic text

SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }

somewhere within the WHERE clause, you get additional variables: For every variable ?foo in your query, you now also have a variable ?fooLabel, which contains the label of the item behind ?foo. If you add this to the SELECT clause, you get the item as well as its label:

SELECT ?child ?childLabel
WHERE
{
# ?child  father   Bach
  ?child wdt:P22 wd:Q1339.
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}

Try it!

Try running that query – you should see not only the item numbers, but also the names of the various children.

child childLabel
wd:Q57225 Johann Christoph Friedrich Bach
wd:Q76428 Carl Philipp Emanuel Bach

Autocompletion[edit]

That SERVICE snippet looks tough to remember though, right? And going through the search function all the time while you’re writing the query is also tedious. Fortunately, WDQS offers a great solution to this: autocompletion. In the query.wikidata.org query editor, you can press Ctrl+Space at any point in the query and get suggestions for code that might be appropriate; select the right suggestion with the up/down arrow keys, and press Enter to select it.

For example, instead of writing out SERVICE wikibase:label { bd:serviceParam wikibase:language "en". } every time, you can just type SERV, hit Ctrl+Space, and the first suggestion will be that complete label service incantation, ready for use! Just hit Enter to accept it. (The formatting will be a bit different, but that doesn’t matter.)

And autocompletion can also search for you. If you type one of the Wikidata prefixes, like wd: or wdt:, and then just write text afterwards, Ctrl+Space will search for that text on Wikidata and suggest results. wd: searches for items, wdt: for properties. For example, instead of looking up the items for Johann Sebastian Bach (Q1339) and father (P22), you can just type wd:Bach and wdt:fath and then just select the right entry from the autocompletion. (This even works with spaces in the text, e. g. wd:Johann Sebastian Bach.)

Advanced triple patterns[edit]

So now we’ve seen all children of Johann Sebastian Bach – more specifically: all items with the father Johann Sebastian Bach. But Bach had two wives, and so those items have two different mothers: what if we only want to see the children of Johann Sebastian Bach with his first wife, Maria Barbara Bach (Q57487)? Try writing that query, based on the one above.

Done that? Okay, then onto the solution! The simplest way to do this is to add a second triple with that restriction:

SELECT ?child ?childLabel
WHERE
{
  ?child wdt:P22 wd:Q1339.
  ?child wdt:P25 wd:Q57487.
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}

Try it!

In English, this reads:

Child has father Johann Sebastian Bach. Child has mother Maria Barbara Bach.

That sounds a bit awkward, doesn’t it? In natural language, we’d abbreviate this:

Child has father Johann Sebastian Bach and mother Maria Barbara Bach.

In fact, it’s possible to express the same abbreviation in SPARQL as well: if you end a triple with a semicolon (;) instead of a period, you can add another predicate-object pair. This allows us to abbreviate the above query to:

SELECT ?child ?childLabel
WHERE
{
  ?child wdt:P22 wd:Q1339;
         wdt:P25 wd:Q57487.
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}

Try it!

which has the same results, but less repetition in the query.

Now suppose that, out of those results, we’re interested only in those children who also were also composers and pianists. The relevant properties and items are occupation (P106), composer (Q36834) and pianist (Q486748). Try updating the above query to add these restrictions!

Here’s my solution:

SELECT ?child ?childLabel
WHERE
{
  ?child wdt:P22 wd:Q1339;
         wdt:P25 wd:Q57487;
         wdt:P106 wd:Q36834;
         wdt:P106 wd:Q486748.
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}

Try it!

This uses the ; abbreviation two more times to add the two required occupations. But as you might notice, there’s still some repetition. This is as if we said:

Child has occupation composer and occupation pianist.

which we would usually abbreviate as:

Child has occupation composer and pianist.

And SPARQL has some syntax for that as well: just like a ; allows you to append a predicate-object pair to a triple (reusing the subject), a , allows you to append another object to a triple (reusing both subject and predicate). With this, the query can be abbreviated to:

SELECT ?child ?childLabel
WHERE
{
  ?child wdt:P22 wd:Q1339;
         wdt:P25 wd:Q57487;
         wdt:P106 wd:Q36834,
                  wd:Q486748.
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}

Try it!

Note: indentation and other whitespace doesn’t actually matter – I’ve just indented the query to make it more readable. You can also write this as:

SELECT ?child ?childLabel
WHERE
{
  ?child wdt:P22 wd:Q1339;
         wdt:P25 wd:Q57487;
         wdt:P106 wd:Q36834, wd:Q486748.
  # both occupations in one line
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}

Try it!

or, rather less readable:

SELECT ?child ?childLabel
WHERE
{
  ?child wdt:P22 wd:Q1339;
  wdt:P25 wd:Q57487;
  wdt:P106 wd:Q36834,
  wd:Q486748.
  # no indentation; makes it hard to distinguish between ; and ,
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}

Try it!

Luckily, the WDQS editor indents lines for you automatically, so you usually don’t have to worry about this.

Alright, let’s summarize here. We’ve seen that queries are structured like text. Each triple about a subject is terminated by a period. Multiple predicates about the same subject are separated by semicolons, and multiple objects for the same subject and predicate can be listed separated by commas.

SELECT ?s1 ?s2 ?s3
WHERE
{
  ?s1 p1 o1;
      p2 o2;
      p3 o31, o32, o33.
  ?s2 p4 o41, o42.
  ?s3 p5 o5;
      p6 o6.
}

Now I want to introduce one more abbreviation that SPARQL offers. So if you’ll humor me for one more hypothetical scenario…

Suppose we’re not actually interested in Bach’s children. (Who knows, perhaps that’s actually true for you!) But we are interested in his grandchildren. (Hypothetically.) There’s one complication here: a grandchild may be related to Bach via the mother or the father. That’s two different properties, which is inconvenient. Instead, let’s flip the relation around: Wikidata also has a “child” property, P:P40, which points from parent to child and is gender-independent. With this information, can you write a query that returns Bach’s grandchildren?

Here’s my solution:

SELECT ?grandChild ?grandChildLabel
WHERE
{
  wd:Q1339 wdt:P40 ?child.
  ?child wdt:P40 ?grandChild.
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}

Try it!

In natural language, this reads:

Bach has a child ?child. ?child has a child ?grandChild.

Once more, I propose that we abbreviate this English sentence, and then I want to show you how SPARQL supports a similar abbreviation. Observe how we actually don’t care about the child: we don’t use the variable except to talk about the grandchild. We could therefore abbreviate the sentence to:

Bach has as child someone who has a child ?grandChild.

Instead of saying who Bach’s child is, we just say “someone”: we don’t care who it is. But we can refer back to them because we’ve said “someone who”: this starts a relative clause, and within that relative clause we can say things about “someone” (e. g., that he or she “has a child ?grandChild”). In a way, “someone” is a variable, but a special one that’s only valid within this relative clause, and one that we don’t explicitly refer to (we say “someone who is this and does that”, not “someone who is this and someone who does that” – that’s two different “someone”s).

In SPARQL, this can be written as:

SELECT ?grandChild ?grandChildLabel
WHERE
{
  wd:Q1339 wdt:P40 [ wdt:P40 ?grandChild ].
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}

Try it!

You can use a pair of brackets ([]) in place of a variable, which acts as an anonymous variable. Inside the brackets, you can specify predicate-object pairs, just like after a ; after a normal triple; the implicit subject is in this case the anonymous variable that the brackets represent. (Note: also just like after a ;, you can add more predicate-object pairs with more semicolons, or more objects for the same predicate with commas.)

And that’s it for triple patterns! There’s more to SPARQL, but as we’re about to leave the parts of it that are strongly analogous to natural language, I’d like to summarize that relationship once more:

natural language example SPARQL example
sentence Juliet loves Romeo. period juliet loves romeo.
conjunction (clause) Romeo loves Juliet and kills himself. semicolon romeo loves juliet; kills romeo.
conjunction (noun) Romeo kills Tybalt and himself. comma romeo kills tybalt, romeo.
relative clause Juliet loves someone who kills Tybalt. brackets juliet loves [ kills tybalt ].

Instances and classes[edit]

Earlier, I said that most Wikidata properties are “has” relations: has child, has father, has occupation. But sometimes (in fact, frequently), you also need to talk about what something is. But there are in fact two kinds of relations there:

  • Gone with the Wind is a film.
  • A film is a work of art.

Gone with the Wind is one particular film. It has a particular director (Victor Fleming), a specific duration (238 minutes), a list of cast members (Clark Gable, Vivien Leigh, …), and so on.

Film is a general concept. Films can have directors, durations, and cast members, but the concept “film” as such does not have any particular director, duration, or cast members. And although a film is a work of art, and a work of art usually has a creator, the concept of “film” itself does not have a creator – only particular instances of this concept do.

This difference is why there are two properties for “is” in Wikidata: instance of (P31) and subclass of (P279). Gone with the Wind is a particular instance of the class “film”; the class “film” is a subclass (more specific class; specialization) of the more general class “work of art”.

So what does this mean for us when we’re writing SPARQL queries? When we want to search for “all works of art”, it’s not enough search for all items that are directly instances of “work of art”:

SELECT ?work ?workLabel
WHERE
{
  ?work wdt:P31 wd:Q838948. # instance of work of art
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}

Try it!

As I’m writing this, that query only returns 2815 results – obviously, there are more works of art than that! The problem is that this misses items like Gone with the Wind, which is only an instance of “film”, not of “work of art”. “film” is a subclass of “work of art”, but we need to tell SPARQL to take that into account when searching.

One possible solution to this is the [] syntax we talked about: Gone with the Wind is an instance of some subclass of “work of art”. (For exercise, try writing that query!) But that still has problems:

  1. We’re no longer including items that are directly instances of work of art.
  2. We’re still missing items that are instances of some subclass of some other subclass of “work of art” – for example, Snow White and the Seven Dwarfs is an animated film, which is a film, which is a work of art. In this case, we need to follow two “subclass of” statements – but it might also be three, four, five, any number really.

The solution: ?item wdt:P31/wdt:P279* ?class. This means that there’s one “instance of” and then any number of “subclass of” statements between the item and the class.

SELECT ?work ?workLabel
WHERE
{
  ?work wdt:P31/wdt:P279* wd:Q838948. # instance of any subclass of work of art
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}

Try it!

(I don’t recommend running that query. WDQS can handle it (just barely), but your browser might crash when trying to display the results because there’s so many of them.)

Now you know how to search for all works of art, or all buildings, or all human settlements: the magic incantation wdt:P31/wdt:P279*, along with the appropriate class. This uses some more SPARQL features that I haven’t explained yet, but quite honestly, this is almost the only relevant use of those features, so you don’t need to understand how it works in order to use WDQS effectively. If you want to know, I’ll explain it in a bit, but you can also just skip the next section and memorize or copy+paste wdt:P31/wdt:P279* from here when you need it.

Property paths[edit]

Property paths are a way to very tersely write down a path of properties between two items. The simplest path is just a single property, which forms an ordinary triple:

?item wdt:P31 ?class.

You can add path elements with a forward slash (/).

?item wdt:P31/wdt:P279/wdt:P279 ?class.

This is equivalent to either of the following:

?item wdt:P31 ?temp1.
?temp1 wdt:P279 ?temp2.
?temp2 wdt:P279 ?class.
?item wdt:P31 [ wdt:P279 [ wdt:P279 ?class ] ].

Exercise: rewrite the “grandchildren of Bach” query from earlier to use this syntax.

An asterisk (*) after a path element means “zero or more of this element”.

?item wdt:P31/wdt:P279* ?class.
# means:
?item wdt:P31 ?class
# or
?item wdt:P31/wdt:P279 ?class
# or
?item wdt:P31/wdt:P279/wdt:P279 ?class
# or
?item wdt:P31/wdt:P279/wdt:P279/wdt:P279 ?class
# or ...

If there are no other elements in the path, ?a something* ?b means that ?b might also just be ?a directly, with no path elements between them at all.

A plus (+) is similar to an asterisk, but means “one or more of this element”. The following query finds all descendants of Bach:

SELECT ?descendant ?descendantLabel
WHERE
{
  wd:Q1339 wdt:P40+ ?descendant.
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}

Try it!

If we used an asterisk instead of a plus here, the query results would include Bach himself.

A question mark (?) is similar to an asterisk or a plus, but means “zero or one of this element”.

You can separate path elements with a vertical bar (|) instead of a forward slash; this means “either-or”: the path might use either of those properties. (But not both – an either-or path segment always matches a path of length one.)

You can also group path elements with parentheses (()), and freely combine all these syntax elements (/|*+?). This means that another way to find all descendants of Bach is:

SELECT ?descendant ?descendantLabel
WHERE
{
  ?descendant (wdt:P22|wdt:P25)+ wd:Q1339.
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}

Try it!

Instead of using the “child” property to go from Bach to his descendants, we use the “father” and “mother” properties to go from the descendants to Bach. The path might include two mothers and one father, or four fathers, or father-mother-mother-father, or any other combination. (Though, of course, Bach can’t be the mother of someone, so the last element will always be father.)

Qualifiers[edit]

(Good news first: this section introduces no additional SPARQL syntax – yay! Take a quick breath and relax, this should be a piece of cake. Right?)

So far, we’ve only talked about simple statements: subject, property, object. But Wikidata statements are more than that: they can also have qualifiers and references. For example, the Mona Lisa (Q12418) has three material used (P186) statements:

  1. oil paint (Q296955), the main material;
  2. poplar wood (Q291034), with the qualifier applies to part (P518)  painting surface (Q861259) – this is the material that the Mona Lisa was painted on; and
  3. wood (Q287), with the qualifiers applies to part (P518)  stretcher bars (Q1737943) and start time (P580) 1951 – this is a part that was added to the painting later.

Suppose we want to find all paintings with their painting surface, that is, those material used (P186) statements with a qualifier applies to part (P518)  painting surface (Q861259). How do we do that? That’s more information than can be represented in a single triple.

The answer is: more triples! (Rule of thumb: Wikidata’s solution for almost everything is “more items”, and the corresponding WDQS rule is “more triples”. References, numeric precision, values with units, geocoordinates, etc., all of which we’re skipping here, also work like this.) So far, we’ve used the wdt: prefix for our statement triples, which points directly to the object of the statement. But there’s also another prefix: p:, which points not to the object, but to a statement node. This node then is the subject of other triples: the prefix ps: (for property statement) points to the statement object, the prefix pq: (property qualifier) to qualifiers, and prov:wasDerivedFrom points to reference nodes (which we’ll ignore for now).

That was a lot of abstract text. Here’s a concrete example for the Mona Lisa:

wd:Q12418 p:P186 ?statement1.    # Mona Lisa: material used: ?statement1
?statement1 ps:P186 wd:Q296966.  # value: oil paint

wd:Q12418 p:P186 ?statement2.    # Mona Lisa: material used: ?statement2
?statement2 ps:P186 wd:Q291034.  # value: poplar wood
?statement2 pq:P518 wd:Q861259.  # qualifier: applies to part: painting surface

wd:Q12418 p:P186 ?statement3.    # Mona Lisa: material used: ?statement3
?statement3 ps:P186 wd:Q287.     # value: wood
?statement3 pq:P518 wd:Q1737943. # qualifier: applies to part: stretcher bar
?statement3 pq:P580 1951.        # qualifier: start time: 1951 (pseudo-syntax)

We can abbreviate this a lot with the [] syntax, replacing the ?statement variables:

wd:Q12418 p:P186 [ ps:P186 wd:Q296966 ].

wd:Q12418 p:P186 [
            ps:P186 wd:Q291034;
            pq:P518 wd:Q861259
          ].

wd:Q12418 p:P186 [
            ps:P186 wd:Q287;
            pq:P518 wd:Q1737943;
            pq:P580 1951
          ].

Can you use this knowledge to write a query for all paintings with their painting surface?

Here’s my solution:

SELECT ?painting ?paintingLabel ?material ?materialLabel
WHERE
{
  ?painting wdt:P31/wdt:P279* wd:Q3305213;
            p:P186 [ ps:P186 ?material; pq:P518 wd:Q861259 ].
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}

Try it!

First, we limit ?painting to all instances of painting (Q3305213) or a subclass thereof. Then, we extract the material from the p:P186 statement node, limiting the statements to those that have an applies to part (P518)  painting surface (Q861259) qualifier.

ORDER and LIMIT[edit]

We return to our regular scheduled program of more SPARQL features.

So far, we’ve only had queries where we were interested in all results. But it’s quite common to care only about a few results: those that are most extreme in some way – oldest, youngest, earliest, latest, highest population, lowest melting point, most children, most materials used, and so on. The common factor here is that the results are ranked in some way, and then we care about the first few results (those with the best rank).

This is controlled by two clauses, which are appended to the WHERE {} block (after the braces, not inside!): ORDER BY and LIMIT.

ORDER BY something sorts the results by something. something can be any expression – for now, the only kind of expression we know are simple variables (?something), but we’ll see some other kinds later. This expression can also be wrapped in either ASC() or DESC() to specify the sorting order (ascending or descending). (If you don’t specify either, the default is ascending sort, so ASC(something) is equivalent to just something.)

LIMIT count cuts off the result list at count results, where count is any natural number. For example, LIMIT 10 limits the query to ten results. LIMIT 1 only returns a single result.

(You can also use LIMIT without ORDER BY. In this case, the results aren’t sorted, so you don’t have any guarantee which results you’ll get. Which is fine if you happen to know that there’s only a certain number of results, or you’re just interested in some result, but don’t care about which one. In either case, adding the LIMIT can significantly speed up the query, since WDQS can stop searching for results as soon as it’s found enough to fill the limit.)

Exercise time! Try to write a query that returns the ten most populous countries. (A country is a sovereign state (Q3624078), and the property for population is P:P1082.) You can start by searching for countries with their population, and then add the ORDER BY and LIMIT clauses.

Here’s my solution:

SELECT ?country ?countryLabel ?population
WHERE
{
  ?country wdt:P31/wdt:P279* wd:Q3624078;
           wdt:P1082 ?population.
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}
ORDER BY DESC(?population)
LIMIT 10

Try it!

Note that if we want the most populous countries, we have to order by descending population, so that the first results will be the ones with the highest values.

Exercise[edit]

We’ve covered a lot of ground so far – I think it’s time for some exercises. (You can skip this section if you’re in a hurry.)

Arthur Conan Doyle books[edit]

Write a query that returns all books by Sir Arthur Conan Doyle.

Chemical elements[edit]

Write a query that returns all chemical elements with their element symbol and atomic number, in order of their atomic number.

Rivers that flow into the Mississippi[edit]

Write a query that returns all rivers that flow directly into the Mississippi River. (The main challenge is finding the correct property…)

Rivers that flow into the Mississippi II[edit]

Write a query that returns all rivers that flow into the Mississippi River, directly or indirectly.

OPTIONAL[edit]

In the exercises above, we had a query for all books by Sir Arthur Conan Doyle:

SELECT ?book ?bookLabel
WHERE
{
  ?book wdt:P50 wd:Q35610.
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}

Try it!

But that’s a bit boring. There’s so much potential data about books, and we only show the label? Let’s try to craft a query that also includes the title (P1476), illustrator (P110), publisher (P123) and publication date (P577).

A first attempt might look like this:

SELECT ?book ?title ?illustratorLabel ?publisherLabel ?published
WHERE
{
  ?book wdt:P50 wd:Q35610;
        wdt:P1476 ?title;
        wdt:P110 ?illustrator;
        wdt:P123 ?publisher;
        wdt:P577 ?published.
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}

Try it!

Run that query. As I’m writing this, it only returns two results – a bit meager! Why is that? We found over a hundred books earlier!

The reason is that to match this query, a potential result (a book) must match all the triples we listed: it must have a title, and an illustrator, and a publisher, and a publication date. If it has some of those properties, but not all of them, it won’t match. And that’s not what we want in this case: we primarily want a list of all the books – if additional data is available, we’d like to include it, but we don’t want that to limit our list of results.

The solution is to tell WDQS that those triples are optional:

SELECT ?book ?title ?illustratorLabel ?publisherLabel ?published
WHERE
{
  ?book wdt:P50 wd:Q35610.
  OPTIONAL { ?book wdt:P1476 ?title. }
  OPTIONAL { ?book wdt:P110 ?illustrator. }
  OPTIONAL { ?book wdt:P123 ?publisher. }
  OPTIONAL { ?book wdt:P577 ?published. }
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}

Try it!

This gives us the additional variables (?title, ?publisher etc.) if the appropriate statement exists, but if the statement doesn’t exist, the result isn’t discarded – the variable simply isn’t set.

Note: it’s very important to use separate OPTIONAL clauses here. If you put all the triples into a single clause, like here –

SELECT ?book ?title ?illustratorLabel ?publisherLabel ?published
WHERE
{
  ?book wdt:P50 wd:Q35610.
  OPTIONAL {
    ?book wdt:P1476 ?title;
          wdt:P110 ?illustrator;
          wdt:P123 ?publisher;
          wdt:P577 ?published.
  }
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}

Try it!

– you’ll notice that most of the results don’t include any extra information. This is because an optional clause with multiple triples only matches when all those triples can be satisfied. That is: if a book has a title, an illustrator, a publisher, and a publication date, then the optional clause matches, and those values are assigned to the appropriate variables. But if a book has, for example, a title but no illustrator, the entire optional clause doesn’t match, and although the result isn’t discarded, all four variables remain empty.

Expressions, FILTER and BIND[edit]

This section might seem a bit less organized than the other ones, because it covers a fairly wide and diverse topic. The basic concept is that we’d now like to do something with the values that, so far, we’ve just selected and returned indiscriminately. And expressions are the way to express these operations on values. There are many kinds of expressions, and a lot of things you can do with them – but first, let’s start with the basics: data types.

Data types[edit]

Each value in SPARQL has a type, which tells you what kind of value it is and what you can do with it. The most important types are:

  • item, like wd:Q42 for Douglas Adams (Q42).
  • boolean, with the two possible values true and false. Boolean values aren’t stored in statements, but many expressions return a boolean value, e. g. 2 < 3 (true) or "a" = "b" (false).
  • string, a piece of text. String literals are written in double quotes.
  • monolingual text, a string with a language tag attached. In a literal, you can add the language tag after the string with an @ sign, e. g. "Douglas Adams"@en.
  • numbers, either integers (1) or decimals (1.23).
  • dates. Date literals can be written by adding ^^xsd:dateTime (case sensitive – ^^xsd:datetime won’t work!) to an ISO 8601 date string: "2012-10-29"^^xsd:dateTime. (Wikidata does not yet support timestamps with hours, minutes, seconds, etc.)

Operators[edit]

The familiar mathematical operators are available: +, -, *, / to add, subtract, multiply or divide numbers, <, >, =, <=, >= to compare them. The inequality test ≠ is written !=. Comparison is also defined for other types; for example, "abc" < "abd" is true (lexical comparison), as is "2016-01-01"^^xsd:dateTime > "2015-12-31"^^xsd:dateTime and wd:Q4653 != wd:Q283111. And boolean conditions can be combined with && (logical and: a && b is true if both a and b are true) and || (logical or: a || b is true if either (or both) of a and b is true).

FILTER[edit]

FILTER(condition). is a clause you can insert into your SPARQL query to, well, filter the results. Inside the parentheses, you can put any expression of boolean type, and only those results where the expression returns true are used.

For example, to get a list of all humans born in 2015, we first get all humans with their date of birth –

SELECT ?person ?personLabel ?dob
WHERE
{
  ?person wdt:P31 wd:Q5;
          wdt:P569 ?dob.
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". } 
}

– and then filter that to only return the results where the year of the date of birth is 2015. There are two ways to do that: extract the year of the date with the YEAR function, and test that it’s 2015 –

FILTER(YEAR(?dob) = 2015).

– or check that the date is between Jan. 1st (inclusive), 2015 and Jan. 1st, 2016 (exclusive):

FILTER("2015-01-01"^^xsd:dateTime <= ?dob && ?dob < "2016-01-01"^^xsd:dateTime).

I’d say that the first one is more straightforward, but it turns out the second one is much faster, so let’s use that:

SELECT ?person ?personLabel ?dob
WHERE
{
  ?person wdt:P31 wd:Q5;
          wdt:P569 ?dob.
  FILTER("2015-01-01"^^xsd:dateTime <= ?dob && ?dob < "2016-01-01"^^xsd:dateTime).
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". } 
}

Try it!

Another possible use of FILTER is related to labels. The label service is very useful if you just want to display the label of a variable. But if you want to do stuff with the label – for example: check if it starts with “Mr. ” – you’ll find that it doesn’t work:

SELECT ?human ?humanLabel
WHERE
{
  ?human wdt:P31 wd:Q15632617.
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
  FILTER(STRSTARTS(?humanLabel, "Mr. ")).
}

Try it!

This query finds all instances of fictional human (Q15632617) and tests if their label starts with "Mr. " (STRSTARTS is short for “string starts [with]”; there’s also STRENDS and CONTAINS). The reason why this doesn’t work is that the label service adds its variables very late during query evaluation; at the point where we try to filter on ?humanLabel, the label service hasn’t created that variable yet.

Fortunately, the label service isn’t the only way to get an item’s label. Labels are also stored as regular triples, using the predicate rdfs:label. Of course, this means all labels, not just English ones; if we only want English labels, we’ll have to filter on the language of the label:

FILTER(LANG(?label) = "en").

The LANG function returns the language of a monolingual string, and here we only select those labels that are in English. The full query is:

SELECT ?human ?label
WHERE
{
  ?human wdt:P31 wd:Q15632617;
         rdfs:label ?label.
  FILTER(LANG(?label) = "en").
  FILTER(STRSTARTS(?label, "Mr. ")).
}

Try it!

We get the label with the ?human rdfs:label ?label triple, restrict it to English labels, and then check if it starts with “Mr. ”.

BIND, BOUND, IF[edit]

These three features are often used in conjunction, so I’ll first explain all three of them and then show you some examples.

A BIND(expression AS ?variable). clause can be used to assign the result of an expression to a variable (usually a new variable, but you can also overwrite existing ones).

BOUND(?variable) tests if a variable has been bound to a value (returns true or false). It’s mostly useful on variables that are introduced in an OPTIONAL clause.

IF(condition,thenExpression,elseExpression) evaluates to thenExpression if condition evaluates to true, and to elseExpression if condition evaluates to false. That is, IF(true, "yes", "no") evaluates to "yes", and IF(false, "great", "terrible") evaluates to "terrible".

BIND can be used to bind the results of some calculation to a new variable. This can be an intermediate result of a larger calculation or just directly a result of the query. For example, to get the age of victims of capital punishment:

SELECT ?person ?personLabel ?age
WHERE
{
  ?person wdt:P31 wd:Q5;
          wdt:P569 ?born;
          wdt:P570 ?died;
          wdt:P1196 wd:Q8454.
  BIND(?died - ?born AS ?ageInDays).
  BIND(?ageInDays/365.2425 AS ?ageInYears).
  BIND(FLOOR(?ageInYears) AS ?age).
  # or, as one expression:
  #BIND(FLOOR((?died - ?born)/365.2425) AS ?age).
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}

Try it!

BIND can also be used to simply bind constant values to variables in order to increase readability. For example, a query that finds all female priests:

SELECT ?woman ?womanLabel
WHERE
{
  ?woman wdt:P31 wd:Q5;
         wdt:P21 wd:Q6581072;
         wdt:P106 wd:Q42603.
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}

Try it!

can be rewritten like this:

SELECT ?woman ?womanLabel
WHERE
{
  BIND(wdt:P31 AS ?instanceOf).
  BIND(wd:Q5 AS ?human).
  BIND(wdt:P21 AS ?sexOrGender).
  BIND(wd:Q6581072 AS ?female).
  BIND(wdt:P106 AS ?occupation).
  BIND(wd:Q42603 AS ?priest).
  ?woman ?instanceOf ?human;
         ?sexOrGender ?female;
         ?occupation ?priest.
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}

Try it!

The meaningful part of the query, from ?woman to ?priest., is now probably more readable. However, the large BIND block right in front of it is pretty distracting, so this technique should be used sparingly. (In the WDQS user interface, you can also hover your mouse over any term like wd:Q123 or wdt:P123 and see the label and description for the entity, so ?female is only more readable than wd:Q6581072 if you ignore that feature.)

IF expressions are often used with BOUND as the expression. For example, suppose you have a query that shows some humans, and instead of just showing their label, you’d like to display their pseudonym (P742) if they have one, and only use the label if a pseudonym doesn’t exist. For this, you select the pseudonym in an OPTIONAL clause (it has to be optional – you don’t want to throw out results that don’t have a pseudonym), and then use BIND(IF(BOUND(… to select either the pseudonym or the label.

SELECT ?writer ?label
WHERE
{
  # French writer born in the second half of the 18th century
  ?writer wdt:P31 wd:Q5;
          wdt:P27 wd:Q142;
          wdt:P106 wd:Q36180;
          wdt:P569 ?dob.
  FILTER("1751-01-01"^^xsd:dateTime <= ?dob && ?dob < "1801-01-01"^^xsd:dateTime).
  # get the English label
  ?writer rdfs:label ?writerLabel.
  FILTER(LANG(?writerLabel) = "en").
  # get the pseudonym, if it exists
  OPTIONAL { ?writer wdt:P742 ?pseudonym. }
  # bind the pseudonym, or if it doesn’t exist the English label, as ?label
  BIND(IF(BOUND(?pseudonym),?pseudonym,?writerLabel) AS ?label).
}

Try it!

Other properties that may be used in this way include nickname (P1449), posthumous name (P1786), and taxon common name (P1843) – anything where some sort of “fallback” makes sense.

You can also combine BOUND with FILTER to ensure that at least one of several OPTIONAL blocks has been fulfilled. For example, let’s get all astronauts that went to the moon, as well as the members of Apollo 13 (Q182252) (close enough, right?). That restriction can’t be expressed as a single property path, so we need one OPTIONAL clause for “member of some moon mission” and another one for “member of Apollo 13”. But we only want to select those results where at least one of those conditions is true.

SELECT ?astronaut ?astronautLabel
WHERE
{
  ?astronaut wdt:P31 wd:Q5;
             wdt:P106 wd:Q11631.
  OPTIONAL {
    ?astronaut wdt:P450 ?mission.
    ?mission wdt:P31 wd:Q495307.
  }
  OPTIONAL {
    ?astronaut wdt:P450 wd:Q182252.
    BIND(wd:Q182252 AS ?mission).
  }
  FILTER(BOUND(?mission)).
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}

Try it!

COALESCE[edit]

The COALESCE function can be used as an abbreviation of the BIND(IF(BOUND(?x), ?x, ?y) AS ?z). pattern for fallbacks mentioned above: it takes a number of expressions and returns the first one that evaluates without error. For example, the above “pseudonym” fallback

BIND(IF(BOUND(?pseudonym),?pseudonym,?writerLabel) AS ?label).

can be written more concisely as

BIND(COALESCE(?pseudonym, ?writerLabel) AS ?label).

and it’s also easy to add another fallback label in case the ?writerLabel isn’t defined either:

BIND(COALESCE(?pseudonym, ?writerLabel, "<no label>") AS ?label).

Grouping[edit]

So far, all the queries we’ve seen were queries that found all items satisfying some conditions; in some cases, we also included extra statements on the item (paintings with materials, Arthur Conan Doyle books with title and illustrator).

But it’s very common that we don’t want a long list of all results. Instead, we might ask questions like this:

  • How many paintings were painted on canvas / poplar wood / etc.?
  • What is the highest population of each country’s cities?
  • What is the total number of guns produced by each manufacturer?
  • Who publishes, on average, the longest books?

City populations[edit]

Let’s look at the second question for now. It’s fairly simple to write a query that lists all cities along with their population and country, ordered by country:

SELECT ?country ?city ?population
WHERE
{
  ?city wdt:P31/wdt:P279* wd:Q515;
        wdt:P17 ?country;
        wdt:P1082 ?population.
}
ORDER BY ?country

Try it!

(Note: that query returns a lot of results, which might cause trouble for your browser. You might want to add a LIMIT clause.)

Since we’re ordering the results by country, all cities belonging to a country form one contiguous block in the results. To find the highest population within that block, we want to consider the block as a group, and aggregate all the individual population values into one value: the maximum. This is done with a GROUP BY clause below the WHERE block, and an aggregate function (MAX) in the SELECT clause.

SELECT ?country (MAX(?population) AS ?maxPopulation)
WHERE
{
  ?city wdt:P31/wdt:P279* wd:Q515;
        wdt:P17 ?country;
        wdt:P1082 ?population.
}
GROUP BY ?country

Try it!

We’ve replaced the ORDER BY with a GROUP BY. The effect of this is that all results with the same ?country are now grouped together into a single result. This means that we have to change the SELECT clause as well. If we kept the old clause SELECT ?country ?city ?population, which ?city and ?population would be returned? Remember, there are many results in this one result; they all have the same ?country, so we can select that, but since they can all have a different ?city and ?population, we have to tell WDQS which of those values to select. That’s the job of the aggregate function. In this case, we’ve used MAX: out of all the ?population values, we select the maximum one for the group result. (We also have to give that value a new name with the AS construct, but that’s just a minor detail.)

This is the general pattern for writing group queries: write a normal query that returns the data you want (not grouped, with many results per “group”), then add a GROUP BY clause and add an aggregate function all the non-grouped variables in the SELECT clause.

Painting materials[edit]

Let’s try it out with another question: How many paintings were painted on each material? First, write a query that just returns all paintings along with their painting material. (Take care to only use those material used (P186) statements with an applies to part (P518)  painting surface (Q861259) qualifier.)

SELECT ?material ?painting
WHERE
{
  ?painting wdt:P31/wdt:P279* wd:Q3305213;
            p:P186 [ ps:P186 ?material; pq:P518 wd:Q861259 ].
}

Try it!

Next, add a GROUP BY clause on the ?material, and then an aggregate function on the other selected variable (?painting). In this case, we are interested in the number of paintings; the aggregate function for that is COUNT.

SELECT ?material (COUNT(?painting) AS ?count)
WHERE
{
  ?painting wdt:P31/wdt:P279* wd:Q3305213;
            p:P186 [ ps:P186 ?material; pq:P518 wd:Q861259 ].
}
GROUP BY ?material

Try it!

One problem with this is that we don’t have the label for the materials, so the results are a bit inconvenient to interpret. If we just add the label variable, we’ll get an error:

SELECT ?material ?materialLabel (COUNT(?painting) AS ?count)
WHERE
{
  ?painting wdt:P31/wdt:P279* wd:Q3305213;
            p:P186 [ ps:P186 ?material; pq:P518 wd:Q861259 ].
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}
GROUP BY ?material

Try it!

Bad aggregate

“Bad aggregate” is an error message you’ll probably see a lot when working with group queries; it means that one of the selected variables needs an aggregate function but doesn’t have one, or it has an aggregate function but isn’t supposed to have one. In this case, WDQS thinks that there might be multiple ?materialLabels per ?material (even though we know that can’t happen), and so it complains that you’re not specifying an aggregate function for that variable.

One solution is to group over multiple variables. If you list multiple variables in the GROUP BY clause, there’s one result for each combination of those variables, and you can select all those variables without aggregate function. In this case, we’ll group over both ?material and ?materialLabel.

SELECT ?material ?materialLabel (COUNT(?painting) AS ?count)
WHERE
{
  ?painting wdt:P31/wdt:P279* wd:Q3305213;
            p:P186 [ ps:P186 ?material; pq:P518 wd:Q861259 ].
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}
GROUP BY ?material ?materialLabel

Try it!

We’re almost done with the query – just one more improvement: we’d like to see the most-used materials first. Fortunately, we’re allowed to use the new, aggregated variables from the SELECT clause (here, ?count) in an ORDER BY clause, so this is very simple:

SELECT ?material ?materialLabel (COUNT(?painting) AS ?count)
WHERE
{
  ?painting wdt:P31/wdt:P279* wd:Q3305213;
            p:P186 [ ps:P186 ?material; pq:P518 wd:Q861259 ].
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}
GROUP BY ?material ?materialLabel
ORDER BY DESC(?count)

Try it!

As an exercise, let’s do the other queries too.

Guns by manufacturer[edit]

What is the total number of guns produced by each manufacturer?

Publishers by number of pages[edit]

What is the average (function: AVG) number of pages of books by each publisher?

HAVING[edit]

A small addendum to that last query – if you look at the results, you might notice that the top result has an outrageously large average, over ten times that of the second place. A bit of investigation reveals that this is because that publisher (UTET (Q4002388)) only published a single book with a number of pages (P1104) statement, Grande dizionario della lingua italiana (Q3775610), which skews the results a bit. To remove outliers like that, we could try to select only publishers that published at least two books with number of pages (P1104) statements on Wikidata.

How do we do that? Normally, we restrict results with a FILTER clause, but in this case we want to restrict based on the group (the number of books), not any individual result. This is done with a HAVING clause, which can be placed right after a GROUP BY clause and takes an expression just like FILTER does:

SELECT ?publisher ?publisherLabel (AVG(?pages) AS ?avgPages)
WHERE
{
  ?book wdt:P123 ?publisher;
        wdt:P1104 ?pages.
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}
GROUP BY ?publisher ?publisherLabel
HAVING(COUNT(?book) > 1)
ORDER BY DESC(?avgPages)

Try it!

Aggregate functions summary[edit]

Here’s a short summary of the available aggregate functions:

  • COUNT: the number of elements. You can also write COUNT(*) to simply count all results.
  • SUM, AVG: the sum or average of all elements, respectively. If the elements aren’t numbers, you’ll get weird results.
  • MIN, MAX: the minimum or maximum value of all elements, respectively. This works for all value types; numbers are sorted numerically, strings and other types lexically.
  • SAMPLE: any element. This is occasionally useful if you know there’s only one result, or if you don’t care which one is returned.
  • GROUP_CONCAT: concatenates all elements. Rarely useful, but if you’re curious, you can look it up in the SPARQL specification.

Additionally, you can add a DISTINCT modifier for any of these functions to eliminate duplicate results. For example, if there are two results but they both have the same value in ?var, then COUNT(?var) will return 2 but COUNT(DISTINCT ?var) will only return 1. You often have to use DISTINCT when your query can return the same item multiple times – this can happen if, for example, you use ?item wdt:P31/wdt:P279* ?class, and there are multiple paths from ?item to ?class: you will get a new result for each of those paths, even though all the values in the result are identical. (If you’re not grouping, you can also eliminate those duplicate results by starting the query with SELECT DISTINCT instead of just SELECT.)

And beyond…[edit]

This guide ends here. SPARQL doesn’t: there’s still a lot that I haven’t shown you – I never promised this was going to be a complete guide! If you got this far, you already know a lot about WDQS and should be able to write some very powerful queries. But if you want to learn even more, here are some things you can look at:

  • Subqueries. You add another entire query in curly brackets ({ SELECT ... WHERE { ... } LIMIT 10 }), and the results are visible in the outer query. (If you’re familiar with SQL, you’ll have to rethink the concept a bit – SPARQL subqueries are purely “bottom-up” and can’t use values from the outer query, like SQL “correlated subqueries” can.)
  • MINUS lets you select results that don’t fit some graph pattern. FILTER NOT EXISTS is mostly equivalent (see the SPARQL spec for an example where they differ), but – at least on WDQS – usually slower by quite a bit.
  • VALUES is a simple way to introduce multiple possible values for a variable.

Your main reference for these and other topics is the SPARQL specification.

And of course, there are some parts of Wikidata still missing as well, such as references, numeric precision (100±2.5), values with units (two kilograms), geocoordinates, sitelinks, statements on properties, and more. You can see how those are modeled as triples under mw:Wikibase/Indexing/RDF Dump Format.