The goal is to take the best existing suggestions that are in the literature (Nosek et al., 2012; Priem, et al., 2010; Giner-Sorolla, 2012; Frey, 2005; Skinner, 1976; Deci 1971; Legris, et al., 2003; Thaler et al., 2008) and design one system that is coherent, easy to use, and conducive to good science (because it makes sense psychologically). If this is your first article in the series, read more about the motivation for change or the motivating principles behind this system at the links.
In one sentence, we are looking to design a Facebook for scientists or a Reddit for research; a profile, a feed of stories, a sophisticated like/ comment section, and a new set of impact metrics which makes use of the available information, allowing us to realign individual and group motives.
The basic unit of the system is the academic profile; here researchers post papers, drafts of papers, datasets, syntax files or other content which can be viewed, liked, and commented on by the scientific community at large. This and content from others (collaborators & professional groups) can be viewed in the profile feed, keeping scientists up to date on all the latest advancements in their field(s).
The fundamental reinforcements are the notifications which we receive when others interact with our content and the quality content provided in the newsfeed. These are the exact same things that make Facebook, Twitter and Instagram so phenomenally successful yet they have not been integrated into any science platform that I know of. The feed is crucial in getting scientists the best content (thus, increasing utility) and the notifications makes us feel good (again, increasing utility). The key of the whole thing is low effort with high utility.
The comment section is the place to discuss a paper and how it relates to the other literature, this is where the debate about the paper’s conclusions and assumptions takes place. This is also where the people who checked the stats, or re-ran the analyses, or replicated the results comment with their findings. This discussion is common in topical Facebook groups, though these communications could be utilized much more efficiently. Like other online outlets (e.g. research gate, PPPR), comments can be viewed by impact (which is most liked/ sub commented), date, or other metrics.
The utility comes from the information within the ‘social’ system.
Not only does the computer use information about your clicks and comments to place things in the feed (bringing you articles you are interested in; Bian, Liu, Zhou, Agichtein, & Zha, 2009), the information can also be used to determine which individuals find or create quality content (e.g., content that generates likes, downloads, and further discussion).
The system knows who uploads content that gets many return comments or shares. The system knows who makes comments on other’s work that receive many likes and response comments. The system knows which thinkers are in which fields, how often papers are linked together, which papers generate discussion, which syntax protocols or data files are often downloaded, which authors, or keywords are trending and much more.
This information can be used for many things; for instance, to develop ‘network maps’ of the research space, which could help theory development, newcomers to the field, or science researchers generally. These maps could also be constructed for authors, papers, keywords, or many other pieces of the science game.
More importantly, this information can be used to reward pro-group behaviors that are currently undervalued and thus not done. The current impact metric is how often the author’s paper gets cited; this could easily be nuanced to include, clicks, comments, shares, downloads, and formal citations on all content including uploaded syntax files, datasets, or comments on other’s work that elicit more discussion (Altmetrics; Priem, Taraborelli, Groth, & Neylon, 2010).
For instance, let’s say someone uploads an interesting, easy to use, dataset into the system (e.g., book reviews for 5,000 goodreads profiles, 1,000 bank employee satisfaction surveys), if many people like, share, download, and use the dataset, it could be made to help their impact metric as much as (or more than!) a just average paper. If one reports that the statistics make sense with the data or report statistical errors in other’s papers, that could be made to help their impact factor, if one consistently makes interesting comments on others work, that could be made to help their impact factor (endorsing good reviews). In this way, these behaviors become relevant and interesting for the individual to do and the group benefits enormously.
These steps not only alleviate the need to cheat (by broadening the reward for things others than perfect, novel, papers), they make it much harder to do so. For instance, under this system, it is reasonable to imagine scientists making their entire careers putting their badge of approval that the statistics and data in a paper are valid, because the community comes to trust these individuals and their posts are appreciated with likes and sub comments.
The same principle holds for making interesting datasets available, designing useful protocols or holding virtual office hours/ discussion sessions on the profiles (like a Google hangout). It becomes worthwhile to upload the data and syntax not because you need to in order to publish, but because it helps your ‘impact factor’ when individuals download these documents. Suddenly, doing the good thing by the group is not utility negative and it will stick.
The system outlined so far could be implemented without changing the fundamental peer review system. The proposed changes will improve the system by encouraging (through ease and likes/ comments) open practices and endorsing group centered behavior, but only adding this to the current system does not adequately deal with the need for competition (as this is only after it has been published), the time papers spend sitting on desks, or the excess cost of the current system (Rennie, 1999; Edlin, & Rubinfeld, 2004). It is time to re-examine the core of the system in light of the advances afforded by the system.
Citations can be read here.
What do you think, could a system like this work? What would it take to bring it about? Leave a comment below! 😀