Hi, I see you did a huge batch with 8000+. I usually try to have 2-3000 as a max for now because larger than so I usually suspect that there is a subclass that I have missed. In this case there seem to be 5000+ articles with "aspiration cytology" which is currently missing an item. Now that the batch has already finished we need this https://github.com/dpriskorn/ItemSubjector/issues/5 implemented to replace the subject.
@So9q I see a lot of interesting features in this version. Great job!! But there are a couple of issues.
I think the DEBUG mode is switched on by default. I see a lot of messages on the screen.
I want to try the JOBS on toolforge. But it's not very clear how I can run prepared jobs on Toolforge/PAWS.
I tried -l option. It doesn't work anymore. I think this option is replaced by -a option. But with -a option, I have to press Enter for every new scholarly article. I am not sure whether you are introducing it as a feature replacing the previous batch option.
Great! I see you are editing a lot. Counting all your edits for september and oktober until now you have 1M edits! If all of them are main subject then you have single handedly taken Wikidata from 14M to 15M total main subjects on the 37M articles. Wow!
According to this query we had 27M articles without any main subject before. I'm curious to see how many it is after our effort. At best, it is 1M less, so still 26M to go! :D
@So9q thanks. I have been monitoring it here: https://scholia.toolforge.org/statistics Today, the value is at 17,064,544 and on September 25, it was 15,831,145. So, I would say around 3 weeks.
@So9q I recall couple of them some weeks ago. The first one was 'Systemic therapy', which was ambiguous since it occurs in two different fields: psychology and cancer therapy.
One possible suggestion for improvement in ItemSubjector:
possible warnings when there is use of the property P1889 (different from), i.e., there are two items with the same label.
The second one was alcholism vs. alcholism treatment. I think that I added the former to an item which already had the latter. So, I am now careful with single word labels.
Are you participating in WikidataCon 2021? I will be sharing my personal experiences here: https://pretalx.com/wdcon21/talk/CUQXWE/ Will you be able to attend this session?
Thanks for creating that and for inviting me. I feel honored. I signed up for WikidataCon and I will do my very best to attend. Would you like me to prepare something?
Also this query now times out on Blazegraph. A similar query works on QLever on older data from a few months back. I'm planning to set up a QLever instance in Toolforge in the near future.
Also: Did you try the new feature I pushed on master yesterday? :D
@So9q I tested the newest feature. It will be quite useful for certain subjects. I may need to test it more.
I did not yet test the Issue 26 fix.
Thanks for your reply concerning WikidataCon. I feel that the participants may have questions about ItemSubjector. If possible, you could present some of its key features.
I have been working on Q35456 lately and will work on all of medicines for the forseeable future. Using the --sparql with --limit with the newest version makes the tool way better IMO because it first fetches (while I walk the dogs) and then I can approve/disapprove in one go in the end and start the job in k8s and leave it running until i finishes.
My biggest k8s job has been around 30k items so far running for 8 hours or more. Now I'm thinking 100k jobs and runtime of 24h might be easy to archive.
@So9q Based on what I remember, I think the number was somewhere between 15M-16M (Scholia Statistics). I may have to dig my tweets. I had shared some screen-captures in the beginning. Check this. It shows the difference: around 15M to 17M (October 2021). I think that a majority of them comes from itemSubjector.
Documenting essential medicinces (Q35456) will be useful. Wow, 100k jobs will be great.
