Wikidata:Requests for permissions/Bot/CovidDatahubBot

From Wikidata
Jump to navigation Jump to search


The following discussion is closed. Please do not modify it. Subsequent comments should be made in a new section. A summary of the conclusions reached follows.

 Approved--Ymblanter (talk) 20:07, 29 December 2020 (UTC)[reply]

CovidDatahubBot[edit]

CovidDatahubBot (talk • contribs • new items • new lexemes • SUL • Block log • User rights log • User rights • xtools)
Operator: TiagoLubiana (talk • contribs • logs)

Task/s: Auto-update of death and case counts for outbreak items that are not regularly updated by the community.

Code:

Function details: --TiagoLubiana (talk) 21:28, 14 May 2020 (UTC)[reply]

Covid-19 country case, death and recovery counts bot[edit]

This bot will:

  • Update only items for country outbreaks that are neither known to be manually updated by dedicated editors, nor updated in the last 23 hours. This limitation is set to avoid conflicting with items that are maintained by different users across the globe.

Items will be updated using the following schema (based on E188):

Every statement will be referenced by

The code is available at this GitHub repo. Currently it is generating QuickStatements claims that are verified and manually added as a batch.

If approved, a backend using WikidataIntegrator would be implemented for daily updates.

The license of data to import is CC0 (https://datahub.io/core/covid-19)

Specifically, these pages have discussions that are key for the matter:

Discussion[edit]

  • @Jura1: The aggregator uses many different sources, each with a format, specification, and rate of the update, so having references to the original sources would require an enormous effort. I have tried to use the WHO Reports for this goal, but they are severely delayed in comparison to the JHU and the national sources. Countries that have items that are updated by the Wikidata community with local sources TiagoLubiana (talk) 23:13, 17 May 2020 (UTC)[reply]
@TiagoLubiana: If you are still interested in this task please register the bot.--Ymblanter (talk) 18:10, 12 June 2020 (UTC)[reply]
@Ymblanter:. Hello, thanks for commenting. What do you mean by registering the bot? Making the bot account itself and start running? TiagoLubiana (talk) 15:31, 13 June 2020 (UTC)[reply]
Yes, not to start running, but to make some test edits from the bot account.--Ymblanter (talk) 15:32, 13 June 2020 (UTC)[reply]
  • To elaborate, many jurisdictions are revising past days' case and death counts based on updated information (since tests can take some time to come back and investigation is often needed to determine whether the person is actually a resident of that jurisdiction). For example, this jurisdiction currently reports 437 cases and 10 deaths as of March 19, whereas on March 19 they reported only 196 cases and eight deaths. Unless the secondary source and this bot goes back and updates past days' figures to match the primary source, it's better to delete the old figures than to keep them around. – Minh Nguyễn 💬 05:05, 29 June 2020 (UTC)[reply]
    The question about historic data is understood, it's just not the Wikidata way to re-write history.
    Anyways, if you just want upload a table of current values for all countries, the place to do that is Commons. --- Jura 08:34, 29 June 2020 (UTC)[reply]
    @Jura1: I'm planning to write a separate bot that would update my state's counties' case counts, to fix the mess that is w:Template:COVID-19 pandemic data/California medical cases by county. To ensure accuracy, I'll have to choose one of the following approaches:
    • Each day, delete all statements other than the latest day's. This is what I'm planning to do.
    • Each day, update all the previous days' statements, discarding the previous values for those days, as the authorities retroactively revise those numbers. This is the approach I'm currently taking in Commons tabular data, with the assumption that anyone interested in historical snapshots can look at the revision history.
    • Each day, add a statement qualified by both point in time (P585) and publication date (P577) for each previous day. 2020 COVID-19 pandemic in Santa Clara County (Q92341065) would grow from roughly 18,000 number of cases (P1603) statements today to roughly 24,000 statements by the end of the month.
     – Minh Nguyễn 💬 07:14, 3 August 2020 (UTC)[reply]
    @Mxn: It's nice to see some activity on this desperately neglected page. Is there any way you could expand the functionality of the bot to take in the Johns Hopkins data for everywhere, not just California? Regarding your question, the revising of past data does make it tricky. Wikidata should ultimately have information about case counts across points in time, not just now, for use in things like w:Template:Interactive COVID-19 maps/Per capita confirmed cases (I'm not sure where Wugapodes is scooping data from for that). But that might be something to just forget about for now and fill back in later once the pandemic is over and numbers are settled. {{u|Sdkb}} talk 03:24, 4 August 2020 (UTC)[reply]
    The data are from the Johns Hopkins repo. JH asserts copyright over the data which in the US is inapplicable since you can't copyright databases (but you sometimes can in the EU). Not sure of the Wikidata policies, but this is what prevented the data being placed on Commons. The code for WugBot's map data is on GitHub and licensed under the MIT license. Wugapodes (talk) 03:36, 4 August 2020 (UTC)[reply]
    @Sdkb: I've been working with the Covid Atlas/Corona Data Scraper project, which maintains scrapers for local dashboards along with JHU. In my test of importing California county statements yesterday, I ran into several data processing issues, which I'm working with the CDS project to address. My integration code is in a GitHub repository alongside custom scrapers for Bay Area counties that I still need to port to CDS. Once I port the workflow to something more robust than QuickStatements, I'd be open to expanding it beyond California. Unfortunately, I can't promise a timeline for that, since the California counties keep throwing curveballs at us almost daily. – Minh Nguyễn 💬 07:10, 4 August 2020 (UTC)[reply]
  • Hi all! What is the status of this currently? We've had to remove the per capita data table (which sources from Wikidata) from en-Wikipedia due to the number of outdated items, but we're ready to bring it back once this is running. {{u|Sdkb}} talk 21:06, 28 June 2020 (UTC)[reply]
  • @Sdkb: Thanks! I really want to get this done, but it is not easy (currently) to just update statements using the WikidataIntegrator infrastructure. Doing it in my spare time. If someone wants to take over and just make this faster, it would be great. TiagoLubiana (talk) 01:32, 29 June 2020 (UTC)[reply]
@TiagoLubiana: Thanks for the update! And apologies if the above came across as rushing — I recognize from the discussion so far that it's a complex task that'll take some time. Cheers, {{u|Sdkb}} talk 01:38, 29 June 2020 (UTC)[reply]
  • For whatever you to do at Wikipedia, maybe it's easier to just store manually a data table on Commons for all countries. I suppose at most places people realized that counts become historic information fairly quickly and have to be understood accordingly. At some point in the future, what was considered the historic count will be more important than the actual count. --- Jura 20:41, 10 August 2020 (UTC)[reply]
The bot could autorun the code above + add the most recent statements (as best rank) to the items without messing with old statements. And then we deal with conflicts on a case by case basis.
If we have an okay for that, I'll dedicate some full days to this, and then it will be done or I'll know which kind of help would be needed. Would that work for you all? TiagoLubiana (talk) 21:03, 26 December 2020 (UTC)[reply]
Importing from Wikipedia pretty much defeats the entire purpose; we want it to be the other way around. I believe the maintainers of Template:COVID-19 pandemic death rates (Q102044164) are themselves importing from somewhere on Commons; that template really should not exist, as it forks the main COVID-19 data template for no good reason. As I've said before, getting the data from the sources is certainly a big challenge, so I don't fault any editor for not achieving it, but collectively, this was a vital task that we needed to accomplish back in February; at this point, there are gargantuan sunk costs, both in terms of wasted manual updating effort over the past eight months, and in terms of the fact that if we'd set up a system early Wikimedia projects and likely even government agencies would have centralized around it instead of the current mess. {{u|Sdkb}} talk 21:18, 26 December 2020 (UTC)[reply]

┌────────────────────────────────────────────────────────────────────────────────────────────────────┘ Hi @sdkb:! Having worked on a number of efforts that tried to unite data efforts to reduce manual updating, I doubt that anything we had done would have unilaterally avoided the current mess. So please don't feel anxious about that. Iterating towards synchrony is a good path; there is never a bad time for it; and the way we get to eventual harmonization is all doing this where we see steps forward -- such as this automation step. Sj (talk) 03:13, 27 December 2020 (UTC)[reply]

As to importing from Wikipedia: that seems like a fine intermediate step. The most current in-wikiverse data flow is currently directly from sources into WP: in particular, the "most well-maintained" infoboxes on a few large wikipedias. The next step is, without disrupting that flow (and convincing WP to import from WD) synchronizing that with WD, and demonstrating the naming conventions and other style guides that seem most appropriate for hosting the data here. Then in parallel people can use that data to synchronize w/ other duplicative places on WP, and w/ smaller-language wikipedias and other wikis. Then we can think about switching the primary data ingestion to updating WD first, and having even the main WP infoboxes draw from that WD source -- this requires a final step of discussion and norms-shifting and socialization.
Tiago: thanks for clarifying; @Jural, Wugapodes, Ymblanter: any reason not to move forward along this line? Sj (talk) 03:13, 27 December 2020 (UTC)[reply]
I do not have any particular opinion, but provided no further objections have been raised, I am going to aprrove the bot in several days.--Ymblanter (talk) 20:00, 27 December 2020 (UTC)[reply]
The above discussion is preserved as an archive. Please do not modify it. Subsequent comments should be made in a new section.

New section[edit]

    • Can we see correct sample edits? DO NOT OVERWRITE OR DELETE HISTORIC DATA in Wikidata. --- Jura 22:28, 29 December 2020 (UTC)[reply]
      @TiagoLubiana:--Ymblanter (talk) 15:31, 30 December 2020 (UTC)[reply]
      @Jura1: I get your point, and I generally agree. The canonical way would be to deprecate statements that are not true anymore. I'd argue, though, that for the template page (Template:COVID-19 pandemic death rates (Q102044164)) it is worth it to do re-writes, as the actual source of the information changes, and there is no point keeping the history there. I won't be doing overwrites for the pandemic-related items for each country, and in a few days, I'll provide your sample edits. --TiagoLubiana (talk) 16:08, 30 December 2020 (UTC)[reply]
      @Jura1: Sample edits done for 10 different countries. I did not add the "preferred" rank as this would quickly become outdated. What could be done is to set all previous "preferred" to "normal" and set the new one as preferred. This makes its re-use a bit easier. Specific countries for which ranking is being dealt with manually could be excluded. Does that make sense? CovidDatahubBot (talk) 13:02, 10 January 2021 (UTC)[reply]
      • Looks better. Is there a way to set a better source than enwiki [2]?
      • If it's mostly maintained by bot, I'd go ahead and set the rank by bot. Maybe avoid changing deprecated rank to normal (Deprecated would for numbers that for some reason are found to be inaccurate).
      • Also, for edits on template items, I don't really mind if it's overwrites data as I don't see template items as part of the main database. --- Jura 15:31, 10 January 2021 (UTC)[reply]
      • That is great, thank you, Jura. I guess I could set a better source (e.g. the one enwiki is pulling from). It is rather tricky, though, as which source to use. The WHO reports are a bit outdated (but might be a good option: https://covid19.who.int/table) and I'm pulling data from a template which is pulling data from here. I do get cold feet with copyright issues, though. Any thoughts? TiagoLubiana (talk) 13:01, 16 January 2021 (UTC)[reply]

Q87250860: destructive edits[edit]

Hi, Your bot makes destructive edits on the on the COVID-19 pandemic in Luxembourg's wikidata page (https://www.wikidata.org/wiki/Q87250860). Please remove Luxembourg from automatic updates. I'm updating the wikidata page of Luxembourg. thanks --Sultan Edijingo (talk) 19:58, 16 January 2021 (UTC)[reply]

Hello. The update via bot brings wrong numbers for COVID-19 pandemic in Taiwan's item (Q84081307) : indeed, January 15th data is imported as January 16th one. By the way, January 16th were already filled in manually before the bot modification, so it resulted in two contradictory data for the same day ; bot also doesn't fill the recoveries number. I update manually the item for Taiwan with deaths/confirmed/recovered cases every "west-european evening", once data is freezed on the CDC (national health authority) website, at 00:30 PM Taiwanese hour (equivalent to west-european mid-afternoon). Thanks. - Daxipedia - 達克斯百科 (talk) 15:19, 17 January 2021 (UTC)[reply]
  • Hello @Sultan Edijingo: and @Daxipedia: Thank you for letting me know. I'll add Taiwan to the whitelist. For others that might read this discussion, feel free to open pull requests at the GitHub repository and fix any issues. It is important to point out that Wikidata is an independent project, and we need to coordinate things here to (and not only with particular wikis). The Wikidata:WikiProject_COVID-19 is a good place to know other people involved. It would be great to automate the items for each country with the best sources available. Your work is amazing, and automating is one way to preserve its beyond the bus factor :) TiagoLubiana (talk) 19:41, 17 January 2021 (UTC)[reply]

@TiagoLubiana: Your bot also puts wrong data into COVID-19 pandemic in Hungary (Q87119811). I am afraid of that it should be shut down immediately, as incorrect dates and values are not acceptable. Bencemac (talk) 08:20, 18 January 2021 (UTC)[reply]

  • @Bencemac: The values are imported from Wikipedia. They are dated as in Wikipedia. I see your point, but divergent information, as long as it is sourced, is not wrong, is at most deprecated. As discussed above:
  • Please update in the GitHub repository for automatic update of your country.
  • I am okay with changing the source of the bot for a more reliable one. So any suggestions are welcome. Maybe the JHU Dashboard directly.

Are you responsible for the updates of COVID-19 pandemic in Hungary (Q87119811)? TiagoLubiana (talk) 11:45, 18 January 2021 (UTC)[reply]

@TiagoLubiana: The bot have added data from 2021-01-16 with point in time (P585)=2021-01-17 from Wikipedia and did it before. This is my problem. And yes, I am updating the page but I am open for stepping down, if it works well. Bencemac (talk) 14:30, 18 January 2021 (UTC)[reply]
@Bencemac: Oh, it was a timezone-related issue, I noticed that. Anyways, it is fixed now, and coming from the source mentioned below, so the date is exactly the one there. TiagoLubiana (talk) 18:30, 18 January 2021 (UTC)[reply]
(Thank you, by the way :) ) TiagoLubiana (talk) 22:15, 21 January 2021 (UTC)[reply]

Change to an external sources[edit]

As per multiple requests, I changed the source of the bot to an external one (https://github.com/CSSEGISandData/COVID-19). I am running some test edits. Of course, any item that is reported as manually updated is removed from the batch updates. Thank you for the notices, keep reporting problems with the bot, I'll solve them ASAP. TiagoLubiana (talk) 13:29, 18 January 2021 (UTC)[reply]

Report incomplete update[edit]

Hi @TiagoLubiana: In this update the date and reference for the number of cases were missing. --Julián L. Páez (talk) 05:24, 30 January 2021 (UTC)[reply]

Start updating ranks[edit]

As per requests (by @Kulawik.pl, Btcprox:), I updated the bot to start updating ranks too. It:

  • Sets all previous "preferred" ranks on deaths and cases to "normal"
  • It set the current, most updated entry as "preferred"

If there are any issues, just let me know!

Best, TiagoLubiana (talk) 13:04, 1 February 2021 (UTC)[reply]

Retrieved from "https://www.wikidata.org/w/index.php?title=Wikidata:Requests_for_permissions/Bot/CovidDatahubBot&oldid=1352670081"