User:ProteinBoxBot/Gene and protein items

Introduction

The ProteinBoxBot maintains information about Genes, Diseases and Drugs in Wikidata. The entities in these three domains are maintained by different corresponding sub-processes of the main bot.

The objective of the gene sub-process (sub-bot?) is to add and update Wikidata with information about genes.

Current Scope

The set of entities maintained by this bot are determined based on their presence in the expert-curated NCBI Entrez Gene database.

At present, the bot is limited to genes from Homo Sapiens and Mus Musculus.

Items maintained by this bot

• Human Genes. List them all with a query for items with taxon human and some value for Entrez Gene ID: CLAIM[703:5] AND CLAIM[351]
• Mouse Genes. List them all with a query for items with taxon mouse and some value for Entrez Gene ID:  CLAIM[703:83310] AND CLAIM[351]
• All genes with an entrez gene id. CLAIM[351]
• Human Proteins. List them all with a query for items with taxon human and some value for Uniprot ID: CLAIM[703:5] AND CLAIM[352]

Gene properties currently maintained by this bot for these items

Property	Description	Datatype	Expected value (if not listed, see property definition)
Property:P279	subclass of	Item	Should always include gene (Q7187)
Property:P351	Entrez Gene ID	String	Should exist for EVERY item processed by this bot
Property:P703	found in taxon	Item	Currently should only include either human Q5 or mouse Q83310
Property:P594	ensembl Gene ID	String
Property:P704	ensembl Transcript ID	String
Property:P353	Gene symbol	String
Property:P354	HGNC symbol	String
Property:P593	homologene id	String
Property:P639	RefSeq RNA ID	String
Property:P1057	chromosome	Item
Property:P684	ortholog	item	should be a gene	property suggests using a species qualifier. this information could also be pulled from the targeted gene's taxon property, so perhaps skip the qualifier ?
Property:P644	genomic start	String		requires qualifier indicates genome build
Property:P645	genomic stop	String		''
Property:P671	Mouse Genome Informatics ID	String
Property:P688	encodes	Item

Gene properties PLANNED for this bot

Property	Description	Datatype	Expected value (if not listed, see property definition)	Note
Property:P692	Gene Atlas Image	image		candidate for improvement if a newer, similar data source could be found. But, lets put in the links so we can completely reproduce the current template. Note that the genes that were already in Wikipedia pretty much all have their geneatlas images linked. The reason this is here is that the bot does not maintain these links - this was a one time import.

The 'encodes' property links gene items to items specifically about the protein, RNA, or other 'product' of the gene. A single gene corresponds to a particular region of a genome that is related to some set of functions. These functions are carried about by the gene's products. Different products may perform vastly different functions. Hence we separate functional information from the gene item itself, and attach this information to the product items wherever possible. (See discussion.)

PROTEIN properties maintained for this bot

Property	Description	Datatype	Expected value (if not listed, see property definition)	note
P279	subclass of	Item	One of: Protein (Q8054), RNA (Q11053), non-coding RNA (Q427087), .
P352	UniProt ID	String	Should exist for EVERY item processed by this bot
P638	PDB ID	String
P637	RefSeq Protein ID	String
P702	encoded by	Item	Should exist for EVERY item processed by this bot
P705	Ensembl Protein ID	String
P591	EC number	String
P18	Protein Structure Image	Commons Media File		should this be to a commons category with multiple images?
P681	Cell Component	Item
P682	Biological Process	Item
P680	Molecular Function	Item

Data sources

The bot retrieves its content from the following trusted sources:

• mygene.info - is a Web service that integrates and provides a convenient API for information about human genes . It provides data from Entrez Gene and other authoritative sources
  • Data has also been loaded from
    1. NCBI Homo sapiens Annotation Release 106
    2. NCBI Mus musculus Annotation Release 104

Bot approval

Bot approval discussion July 2013: Wikidata:Requests_for_permissions/Bot/ProteinBoxBot

The following discussion is closed. Please do not modify it. Subsequent comments should be made in a new section. A summary of the conclusions reached follows.

 Approved--Ymblanter (talk) 15:20, 28 July 2013 (UTC)

---

ProteinBoxBot

ProteinBoxBot (talk • contribs • new items • new lexemes • SUL • Block log • User rights log • User rights • xtools)
Operator:Chinmay26 (talk) 03:14, 19 July 2013 (UTC) Chinmay26 (talk • contribs • logs)

Task/s: Populate Gene wikidata items with gene properties. Runs are here This bot is an update over existing Proteinboxbot

Function details: There are around 10,000+ Human protein templates which are maintained by pygenewiki(current bot). The Bot will create Gene Wikidata items and and populate them with gene/protein specific properties. Through molecular biology community discussions , now each wikipedia page will be sourced from four seperate wikidata items -- Human protein,Human Gene, Mouse Gene, Mouse protein . The entire design of the new items and their properties is here. I am using Pywikipedia-rewrite branch and the project is hosted on bitbucket --Chinmay26 (talk) 03:14, 19 July 2013 (UTC)

Two things:

• Does it make sense to use P89 (P89) rather than taxon ? I suppose the taxon will usually be a species, but that does not sound like a logical necessity, and the property does not serve the same purpose as in Brassica oleracea var. botrytis (Q7537). If you agree with that, and are in a hurry suppose we can create a new property without going through the whole proposal process.
• A much broader (and hackneyed sorry) issue but it appears clearly here: if human reelin is a subclass of protein, and enzyme is a subclass of protein, is there a way to know that "human reelin" is protein in a different sense than "enzyme" is. --Zolo (talk) 18:43, 19 July 2013 (UTC)

Thanks for the feedback! Your points in order:

• Can you clarify why you think P89 (P89) is serving a different purpose in Brassica oleracea var. botrytis (Q7537) and RELN (Q414043)? As near as I can tell, in both cases P89 (P89) links to items that in turn have properties for taxon name (P225), P74 (P74), taxon rank (P105), etc.
• Great point. On second look, I think Q13569356 should be an instance of (P31) of protein (Q8054), not a subclass of (P279). Would that be more correct?

Thanks! Cheers, Andrew Su (talk) 20:04, 19 July 2013 (UTC)

The Wikidata items this bot is concerned with represent classes of things, not a particular, concrete object, so subclass of (P279) and not instance of (P31) is the correct membership property to use here. The scenario here is analogous to the quark example in Help:Basic_membership_properties -- Wikidata items about genes and proteins speak about their subjects as a class of things (in which case P279 is appropriate), not, say, a particular molecule of reelin in a specific mouse's hippocampus (in which case P31 would be appropriate). Classifying genes and proteins with P279 is also consistent with the convention used in OBO ontologies, which use rdfs:subClassOf (the basis of P279) to represent genes and proteins. (See e.g. http://owl.cs.manchester.ac.uk/goal/ontologies/MouseGOAL.tar.gz, discussed in http://www.ncbi.nlm.nih.gov/pubmed/22541594.)

To Zolo's specific concern, I would note that enzyme (Q8047) is not a subclass of protein, because not all enzymes are proteins. Enzymes can be RNA molecules, too.

But that just moves the goalpost: it's still worthwhile to consider how we can indicate that a specific kind of protein is, say, a biomolecule in a different sense than an enzyme is a biomolecule. This issue is a matter of semantic distinction independent of whether we call these things instances or classes. We can distinguish enzymes from proteins by keeping them in separate branches of the term hierarchy. For example, proteins and enzymes can both be considered biomolecules (or whatever we decide), but proteins biomolecules that are defined by being sequences of amino acids, whereas enzymes are biomolecules that are defined strictly by having a catalytic activity. This distinction is discussed in more detail at Property_talk:P591#Distinguishing_enzymes_and_gene_products.

WT:MBTF would probably be a better venue for this discussion, if folks feel like exploring it in depth. I don't think it should hold up the nomination of ProteinBoxBot. Emw (talk) 23:15, 19 July 2013 (UTC)

I took enzyme because it is the first thing I could think of, but that would probably apply to things like DNA-binding protein (Q2252764) as well, but I agree that reelin seems more like a subclass than an instance of protein and that if the bot task is urgent it can be approved before the issue is elucidated. --Zolo (talk) 06:28, 20 July 2013 (UTC)

Keeping the "species" claims as such is probably fine. I was concerned that a new "taxon" (with values that are items) would be better if anyone ever wants to work with ProteinBoxBot to classify microbial genes and proteins, but it seems that the infoboxes for viral proteins like env point to resources like http://www.ncbi.nlm.nih.gov/Taxonomy/Browser/wwwtax.cgi?id=11676&rn=1, which, following the link for the organism to http://www.ncbi.nlm.nih.gov/Taxonomy/Browser/wwwtax.cgi?mode=Info&id=11676&lvl=3&lin=f&keep=1&srchmode=1&unlock, assigns the microbial organism a taxonomic rank of "species". Entrez Gene also assigns records to a species taxon, e.g. http://www.ncbi.nlm.nih.gov/gene/155971.

So in summary, keeping the "species" claims for genes and proteins in place is in line with reliable sources. Emw (talk) 23:44, 19 July 2013 (UTC)

In Brassica oleracea var. botrytis (Q7537), "species" is essentially a subproperty of subclass of (P279): cauliflower is a type of Brassica oleracea. In Reelin, "species" means something different, like "is to be found in humans". Now, you may argue that the P89 (P89) of cauliflower is badly redundant with parent taxon (P171) and I would agree, but that is a different issue: currently the main use of the property is the one that is made in Brassica oleracea var. botrytis (Q7537), not in reelin. --Zolo (talk) 06:28, 20 July 2013 (UTC)

I see. Thanks for clarifying -- I agree this is a problem that needs to be solved. In brief, claims involving P89 (P89) in items about taxa treat "species" as a subproperty of subclass of (P279) as you say, but claims involving P89 (P89) in items about genes and proteins treat "species" as a subproperty of part of (P361). This seems like it would cause problems with semantic reasoners that work with items about taxa and items about genes/proteins, which is entirely foreseeable.

Per your comment, I think a new property "found in taxon" might solve this problem. The property would explicitly note that it is a (distant) subproperty of P361 and should not take on the semantics of P279. Thoughts? Emw (talk) 12:31, 20 July 2013 (UTC)

Whew, glad we have a couple of data modeling heavyweights here to sort this out. I'm out of my league here, so suffice it to say that we'll follow whatever consensus emerges here. (And to echo Emw's previous comment, perhaps it's worth moving the discussion to WT:MBTF.) Cheers, Andrew Su 00:08, 21 July 2013 (UTC)

Comments from Emw: I've looked through ProteinBoxBot's test edits and compared them with the data model worked out at WT:MBTF. Below are some things I think would help to do before scaling up this bot's activity:

• For a few proteins, have ProteinBoxBot (PBB) create the additional 3 items expected per Gene Wiki article -- human gene, mouse gene, mouse protein -- and fill them in with the expected properties. The test edits only show how the bot works for human proteins.
• On protein items, have PBB link to EC enzyme classification (P660) instead of EC enzyme number (P591). Per Property_talk:P591#Distinguishing_enzymes_and_gene_products, "EC number" (P591) should be used on enzyme items, and "EC classification" (P660) should be used on protein items. This will require PBB to have some way to 1) link the "EC number" in the Wikipedia infobox to the enzyme's "EC accepted name" (e.g. EC number "3.4.21" -> EC accepted name "serine endopeptidase"), then 2) get the Wikidata ID of the enzyme item for the EC accepted name from step 1 (e.g. EC accepted name "serine endopeptidase" -> Wikidata ID Q420032).

For step 1, you should be able to map EC number to EC accepted name by using the lists in the table at http://www.chem.qmul.ac.uk/iubmb/enzyme/#recommend. For step 2, you could look up the ID of the item with that EC accepted name using the Wikidata API's wbsearchentities module, e.g. http://www.wikidata.org/w/api.php?action=wbsearchentities&search=serine%20endopeptidase&language=en&format=jsonfm. If no Wikidata item has the EC accepted name as a label or alias, then that enzyme item would need to be created.

• Have PBB make labels and descriptions of gene and protein items match the pattern described at WT:MBTF#Gene_and_protein_labels_and_descriptions. I've modified the reelin items as examples: human protein reelin (Q13569356), human gene RELN (Q414043), mouse gene Reln (Q13567973), mouse protein Reelin (Q14155909). For human proteins (the only gene/protein items with site links) this would mean adding/changing their description, since their label already matches the convention.

Also, to my understanding it's convention to make test edits using the bot account, not the bot operator account. Let me know if I can clarify anything! Emw (talk) 13:35, 21 July 2013 (UTC)

Thanks Zolo and Emw for the feedback.I am currently working on creating new human gene,mouse gene, mouse protein items for the first few "human protein templates". I will be following the model of "Reelin item" and add the necessary description field, EC classification etc as well. Regarding the "subclass of" property, should we go ahead with the current model?. Chinmay26 (talk) 16:40, 21 July 2013 (UTC)

Hi Chinmay, from my perspective, using the current model (all genes noted as subclass of 'gene', all proteins subclass of 'protein') makes sense. Emw (talk) 12:07, 22 July 2013 (UTC)

I have created found in taxon (P703). I think it is ok to use subclass. If we need to adjust things later on, it should still be differentiate proteins from group of proteins based on the properties used in the items. --Zolo (talk) 20:17, 22 July 2013 (UTC)

What is the actual situation with the bot? Have the objections been answered? Are we moving to approval--Ymblanter (talk) 15:25, 26 July 2013 (UTC)

Currently, the bot runs are for Human Protein items only. As Emw and Zolo suggested, i will extend the bot functionality and include mouse protein, mouse gene, human gene items. I am working on it(will be over in 2 more days) and i will show test runs here under the bot account. Chinmay26 (talk) 12:25, 27 July 2013 (UTC)

Ok, the bot will be approved in 24h provided there have been no objections raised.--Ymblanter (talk) 13:56, 27 July 2013 (UTC)

---

The above discussion is preserved as an archive. Please do not modify it. Subsequent comments should be made in a new section.

Re-approval discussion June 2015: Wikidata:Requests_for_permissions/Bot/ProteinBoxBot_2

The following discussion is closed. Please do not modify it. Subsequent comments should be made in a new section. A summary of the conclusions reached follows.

 Approved--Ymblanter (talk) 16:01, 29 July 2015 (UTC)

---

ProteinBoxBot

ProteinBoxBot (talk • contribs • new items • new lexemes • SUL • Block log • User rights log • User rights • xtools)
Operator:Andra Waagmeester (talk)

Task/s: Add content on genes and proteins to WikiData and subsequently enrich these items with relevant additions from authoritative resources.

Function details:

This is a (re)approval request for the ProteinBoxBot (original request) as stipulated on the Wikidata:Administrators noticeboard.The ProteinBoxBot is core in our efforts to enrich wikidata with Genes, Proteins, Diseases, Drugs and the relationships between them. We currently have code in place to enrich wikidata with content from entrez gene and Disease ontology. With the previous bot credentials this code has added human and mouse genes as well as the diseases in the disease ontology. The items have also been updated regularly. We now understand that part of these tasks have been performed out of the scope of the initial bot approval. We would like to request a reapproval of the ProteinBoxBot so we can continue enriching wikidata with genes and proteins. Next to the initial context of genes and proteins we would also like to request approval for adding diseases. Our work has also recently been presented at two occasions 1 2

The bot funtions and tasks are developed in sprints

We are currently developing the bot code to add drugs and the links between drugs, genes and diseases. We will request approval for these tasks in due time. Andrawaag (talk) 19:38, 2 June 2015 (UTC)

I will approve the bot in a couple of days provided there have been no objections.--Ymblanter (talk) 18:02, 5 June 2015 (UTC)

@Jura1:@Multichill: Any comments on this request for permissions? Is this in line with what was suggested on the Administrators' noticeboard? Andrawaag (talk) 20:54, 5 June 2015 (UTC)

@Andrawaag:

1. how can you ensure that the bot is not creating duplicates? In the last sprints, following items about the same disease were created: Q18554270, Q19587455, Q19609917, Q19833035. Have you developped a method to prevent such duplicates?
2. Besides adding identifiers of external databases, do you plan to add other information?
3. Is it possible to have a look at the source code of your bot? --Pasleim (talk) 22:04, 6 June 2015 (UTC)

@Pasleim:

1. Until now the bot has been relying on wdq to prevent creating duplicates. I am aware of the delay of content between wdq and wikidata. As you rightly note, some sip through. This is typically the case when there is an anomaly in the connection between the bot and the api. I usually deal with these duplicates, but I apparently missed some. Normally there is typically a week between bot runs to allow updating of wdq. I am considering adding local logging to deal with this issue. However, I don't want to rely on a local log file only, since that would disregard additions to wikidata outside the scope of our bot. A hybrid between the wdq and a local log file of our bot additions, is something I am going to implement very soon.

2. Next to identifiers of external databases, we also add titles and aliases, as well as some provenance of when and form where the statements originates. The properties are listed on the page describing a specific bot task: e.g. https://www.wikidata.org/wiki/User:ProteinBoxBot/Disease_items#Properties

3. The source code of our bot is maintained on bitbucket and can be found at https://bitbucket.org/sulab/wikidatabots/src. We are currently in the process of restructuring this to separate accepted bot tasks from those that need requesting permission.Andrawaag (talk) 20:41, 8 June 2015 (UTC)

@Pasleim:

Regarding point 1: We will introduce further measures in order to avoid any duplicate entries, this also means that we might abandon usage of wdq in the bot code, so the fact that wdq only uses data dumps from production wikidata will not be an issue. Furthermore, the bot will generate and maintain a local list, which is mapping IDs and/or accession numbers from authoritative resources to Wikidata item IDs. I think the combination of these two approaches will prevent creation of duplicate entries. Sebotic (talk) 20:40, 8 June 2015 (UTC)

Ok, I  support the bot tasks Gene and protein items, Disease items and Drug items. For the other tasks, please either provide more information in the next few days, or start a new approval request as soons as a more detailed plan is available. --Pasleim (talk) 19:32, 12 June 2015 (UTC)

•  Oppose please do a testrun with of about 100 edits first. The bot's most recent edits are still those of the previous accident.--- Jura 20:05, 12 June 2015 (UTC)

We aer refurbishing our bot code, so we will indeed run 100 test edits. I assume 100 test edits are okay? We completely stopped running the bot upon the request to not do anything until we got a reapproval. Andrawaag (talk) 09:27, 26 June 2015 (UTC)

@Jura1:@Pasleim:@Ymblanter: We have taken the time to fully reimplement our bot code. The current bot contains two layers. One being a core layer that takes responsibility for the communication with the api of WIkidata. The second layer is a task specific layer, which takes care of extracting knowledge from the relevant authoritative source (e.g. Entrez Gene in this specifc task approval case). Each approved task will share the Core layer.

I performed a little over 100 test edits, with the new bot and we are looking forward to run a full run on updating wikidata with gene information Andrawaag (talk) 09:45, 13 July 2015 (UTC)

@Jura1: does the pbb bot have your support following @andra 's last run? --I9606 (talk) 22:41, 17 July 2015 (UTC)

Sorry to jump in here at the last minute but could your bot include a url in the reference for the source of the info it adds (the particular web page - not just the web site) - even if it the same as the Entrez Gene ID (P351) statement? Joe Filceolaire (talk) 08:18, 23 July 2015 (UTC)

To be clear, you are suggesting adding Property:P854 (reference URL) in addition to the 'stated in' and 'imported from' reference properties that are already being used, correct? This seems doable - at least for the entrez gene ids - but I am curious to know if there is a specific use case you have in mind here? --I9606 (talk) 18:17, 23 July 2015 (UTC)

@Filceolaire: as I9606 already said, we can easily add an additional reference. Personally I would be a bit cautious here. URLs do change over time and with such a change on Entrez Genes, it is easier to fix it on the Entrez Gene ID property, which will be in effect on all Entrez Genes, without the need to rerun a bot cycle to change all the referenceURLs for all genes. Having said that, adding a referenceURL to the list of reference is really easy, so if your use case requires this, we can easily add it. Andrawaag (talk) 19:04, 23 July 2015 (UTC)

@Multichill: Does you prohibition to edit wikidata items with this bot still stands? We have dealt with the issues and would really like to restart our efforts. Andrawaag (talk) 16:30, 27 July 2015 (UTC)

 support The bot has been fixed and substantially improved. There are active developers waiting for this approval and this is slowing their work down. Please unblock the bot and iterate with the bot developers about minor changes such as the suggested addition of the URL above. (Note that the information that would be in that URL is already present in the data being added by the bot.) --I9606 (talk) 16:35, 27 July 2015 (UTC)

The bot is not currently blocked. I am not happy with the fact that there are no reactions but I am hesitant to approve the task provided that a lot of opposition has been voiced, and after the modifications nobody changed their opinion. if there is support for this task I will be happy to approve the bot.--Ymblanter (talk) 19:14, 27 July 2015 (UTC)

@Ymblanter: I think that "a lot of opposition" is an overstatement. There is only one 'oppose' listed above from @Jura1: who has not responded to repeated pings here and on his talk page. @Andrawaag: has patiently responded to all requests here including a substantial restructuring of the code to reduce the chances of similar errors showing up again. This is a good bot, adding content of high value from a respectful and dedicated developer. It is very frustrating to see progress here hampered not by logical argument by people that care, but by a lack of response from people entrusted with authority. --I9606 (talk) 20:42, 27 July 2015 (UTC)

 support I am also part of the team developing this bot, but just to underscore the point that there are many people who care about Wikidata thinking about this bot and this task, not a single rogue developer. Andrew Su (talk) 20:53, 27 July 2015 (UTC)

 support sorry for not responding to the pings earlier. Have been a bit busy and traveling. As far as I can see all issues have been addressed. I'm in favor of starting the bot and keeping an eye on feedback. One thing to note is that while looking at your source code I noticed you seem to be reinventing the wheel. I would recommend that at some point you switch to Pywikibot as the underlying library. It contains all the low and high level stuff you need to be able to communicate with Wikibase/Wikidata. Don't let that block you for now, but you should look into that for the future. We have several people around here who would probably be more than willing to help you convert your code. Thanks for your work. Multichill (talk) 16:36, 28 July 2015 (UTC)

---

The above discussion is preserved as an archive. Please do not modify it. Subsequent comments should be made in a new section.

Implementation

The bot code is open source and available for inspection. It is implemented in Python and is intended to be deployed as a cron job. The current bot is able to update Wikidata with all content (labels, synonyms, and external references) in less then 12 hours.