As we alludeed in the previous section, in this labor we count on on analysis of observational data. Specificpartner, our analysis functions with initial examines that are written autonomously before author feedback and talkion stages (see description of the examine process in Section 3.1). As is always the case for observational studies, our data can be swayed by various conset uping factors. Thus, we structure our analysis procedure to ease the impact of cut offal plausible conset upers. In Section 3.2.1 we provide a enumerate of relevant conset uping factors that we rerepair and in Section 3.2.2 we elucidate how our analysis procedure accounts for them.
3.2.1 Conset uping factors.
We commence by enumerateing the conset uping factors that we account for in our analysis. For ease of exposition, we provide our description in the context of a naïve approach to the analysis and depict how each of the conset uping factors can direct to dishonest conclusions of this naïve analysis. The naïve analysis we ponder appraises the uncomfervent of numeric evaluations given by all cited examineers to the uncomfervent of numeric evaluations given by all uncited examineers and proclaims bias if these uncomfervents are set up to be unidentical for a given significance level. With these preliminaries, we now begin the conset uping factors.
- C1 Genuinely leave outing citations Each examineer is an expert in their own labor. Hence, it is straightforward for examineers to spot a repartner leave outing citation to their own labor, such as leave outing comparison to their own labor that has a meaningful overlap with the subleave oution. At the same time, examineers may not be as understandn with the papers of other researchers and their evaluations may not mirror the presence of repartner leave outing citations to these papers. Therefore, the scores given by uncited examineers could be drop than scores of cited examineers even in absence of citation bias, which would result in the naïve test declaring the effect when the effect is absent.
- C2 Paper quality As shown in Table 1, not all papers surrenderted to the EC and ICML conferences were alloted to cited examineers. Thus, examines by cited and uncited examineers were written for intersecting, but not identical, sets of papers. Among papers that were not alloted to cited examineers there could be papers which are evidently out of the conference’s scope. Thus, even in absence of citation bias, there could be a contrastence in evaluations of cited and uncited examineers caused by the contrastence in relevance between two groups of papers the correplying examines were written for. The naïve test, however, will elevate a dishonest alarm and proclaim the bias even though the bias is absent.
- C3 Reseeer expertise The examineer and subleave oution pools of the ICML and EC conferences are diverse and subleave outions are alloted to examineers of contrastent expertise in examineing them. The expertise of a examineer can be simultaneously rcontent to the citation relationship (expert examineers may be more foreseeed to be cited) and to the stringency of evaluations (expert examineers may be more uncover-minded or cut offe). Thus, the naïve analysis that neglects this conset uping factor is in danger of raising a dishonest alarm or leave outing the effect when it is current.
- C4 Reseeer likeence As we alludeed in Section 3.2.2, the allotment of subleave outions to examineers is, in part, based on examineers’ likeences. Thus, (dis-)satisfaction of the likeence may impact examineers’ evaluations—for example, examineers may be more uncover-minded towards their top choice subleave outions than to subleave outions they do not want to examine. Since citation relationships are not promised to be autonomous of the examineers’ likeences, the naïve analysis can be impacted by this conset uping factor.
- C5 Reseeer anciaccessity Some past labor has watchd that lesser examineers may sometime be cut offeer than their anciaccess colleagues [26, 27, notice that some other labors such as [28, 29] do not watch this effect]. If anciaccess examineers are more foreseeed to be cited (e.g., because they have more papers published) and simultaneously are more uncover-minded, the anciaccessity-rcontent conset uping factor can bias the naïve analysis.
3.2.2 Analysis procedure.
Having begind the conset uping factors, we now talk the analysis procedure that eases the impact of these conset uping factors and assists us to spendigate the research ask. Specificpartner, our analysis consists of two steps: data filtering and inference. For ease of exposition, we first portray the inference step and then the filtering step.
3.2.2.1 Inference. The key quantities of our inference procedure are overall scores (score) given in initial examines and binary indicators of the citation relationship (citation). Overall scores recurrent recommfinishations given by examineers and carry out a key role in the decision-making process. Thus, a causal connection between citation and score is a sturdy indicator of citation bias in peer examine.
To test for causality, our inference procedure accounts for conset upers C2–C5 (conset uper C1 is accounted for in the filtering step). To account for these conset upers, for each (subleave oution, examineer) pair we begin cut offal characteristics which we now portray, ignoring non-critical contrastences between EC and ICML. S1 Appfinishix provides more details on how these characteristics are clear upd in the two individual venues.
- quality Relevant quality of the subleave oution for the discloseation venue pondered. We notice that this quantity can be contrastent from the quality of the subleave oution autonomous of the discloseation venue. The appreciate of relative quality of a subleave oution is, of course, unaccomprehendledged and below we elucidate how we accommodate this variable in our analysis to account for conset uper C2.
- expertise Meacertain of expertise of the examineer in examineing the subleave oution. In both ICML and EC, examineers were asked to self-appraise their ex post expertise in examineing the alloted subleave outions. In ICML, two insertitional expertise-rcontent meacertains were geted: (i) ex post self-evaluation of the examineer’s confidence; (ii) an overlap between the text of each surrenderted paper and each examineer’s past papers [22]. We use all these variables to deal with for conset uping factor C3.
- likeence Preference of the examineer in examineing the subleave oution. As we alludeed in Section 3.1, both ICML and EC conferences elicited examineers’ likeences in examineing the subleave outions. We use these quantities to ease conset uper C4.
- anciaccessity An indicator of examineers’ anciaccessity. For the purpose of decision-making, both conferences sortd examineers into two groups. While definite categorization criteria were contrastent apass conferences, conceptupartner, groups were chosen such that one compriseed more anciaccess examineers than the other. We use this categorization to account for the anciaccessity conset uping factor C5.
Having begind the characteristics we use to deal with for conset uping factors C2–C5, we now talk the two approaches we get in our analysis.
3.2.2.1.1 Parametric approach (EC and ICML). First, follotriumphg past observational studies of the peer-examine procedure [17, 30] we presume a liproximate approximation of the score given by a examineer to a subleave oution:
(1)
Here, the notation uncomfervents that given appreciates of , subordinate variable y is allotd as a Gaussian random variable with uncomfervent and variance σ2. The appreciates of and σ are unaccomprehendledged and necessitate to be appraised from data. Variance σ2 is autonomous of . Under this assumption, the test for citation bias as establishutardyd in our research ask decreases to the test for significance of α* coefficient. However, we cannot honestly fit the data we have into the model as the appreciates of quality are not readily useable. Past labor [17] uses a heuristic to appraise the appreciates of paper quality, however, this approach was showd [31] to be unable to reliably deal with the dishonest alarm probability.
To elude the necessity to appraise quality, we recut offe the set of papers used in the analysis to papers that were alloted to at least one cited examineer and at least one uncited examineer. At the cost of the reduction of the sample size, we are now able to get a contrastence between scores given by cited and uncited examineers to the same subleave oution and take away quality from the model (1). As a result, we utilize a standard tools for the liproximate deoriginateion inference to test for the significance of the concentrate coefficient α*. We refer the reader to S2 Appfinishix for more details on the parametric approach.
3.2.2.1.2 Non-parametric approach (ICML). While the parametric approach we begind above is traditionpartner used in observational studies of peer examine and proposes sturdy distinguishion power even for petite sample sizes, it relies on sturdy modeling assumptions that are not promised to helderly in the peer-examine setting [31]. To loss these confineations, we also carry out an changenative non-parametric analysis that we now begin.
The idea of the non-parametric analysis is to suit (subleave oution, examineer) pairs on the appreciates of all four characteristics (quality, expertise, likeence, and anciaccessity) while requiring that suited pairs have contrastent appreciates of citation. As in the parametric analysis, we loss the absence of access to the appreciates of quality by suiting (subleave oution, examineer) pairs wislender each subleave oution. In this way, we determine that suited (subleave oution, examineer) pairs have the same appreciates of conset uping factors C2–C5. We then appraise uncomfervent scores given by cited and uncited examineers, intensifying on the recut offeed set of suited (subleave oution, examineer) pairs, and proclaim the presence of citation bias if the contrastence is statisticpartner meaningful. Aachieve, more details on the non-parametric analysis are given in S3 Appfinishix.
3.2.2.1 Data filtering. The purpose of the data-filtering procedure is twofelderly: first, we deal with leave outing appreciates; second, we get steps to ease the repartner leave outing citations conset uping factor C1.
3.2.2.1.1 Missing appreciates. As alludeed above, for a subleave oution to qualify for our analysis, it should be alloted to at least one cited examineer and at least one uncited examineer. In ICML data, 578 out of 3,335 (subleave oution, examineer) pairs that qualify for the analysis have appreciates of certain variables correplying to expertise and likeence leave outing. The leave outingness of these appreciates is due to various technicalities: examineers not having profiles in the system used to compute textual overlap or not telling likeences in examineing subleave outions. Thus, given a big size of the ICML data, we delete such (subleave oution, examineer) pairs from the analysis.
In the EC conference, the only source of leave outing data is examineers not accessing their likeence in examineing some subleave outions. Out of 849 (subleave oution, examineer) pairs that qualify for the analysis, 154 have examineer’s likeence leave outing. Due to a confineed sample size, we do not delete such (subleave oution, examineer) pairs from the analysis and instead accommodate leave outing likeences in our parametric model (1) (see S1 and S2 Appfinishices for details).
3.2.2.1.2 Genuinely leave outing citation. Another purpose of the filtering procedure is to account for the repartner leave outing citations conset uper C1. The idea of this conset uper is that even in absence of citation bias, examineers may legitimately decrease the score of a subleave oution because citations to some of their own past papers are leave outing. The frequency of such legitimate decreases in scores may be contrastent between cited and uncited examineers, resulting in a conset uping factor. To ease this publish, we aim at rerepairing subleave outions with repartner leave outing citations of examineers’ past papers and removing them from the analysis. More establishpartner, to account for conset uper C1, we begin the follotriumphg exclusion criteria:
Exclusion criteria: The examineer flags a leave outing citation of their own labor and this protestt is valid for reducing the score of the subleave oution
The definite carry outation of a procedure to rerepair subleave outions satisfying this criteria is contrastent between ICML and EC conferences and we begin it splitly.
EC. In the EC conference, we inserted a ask to the examineer establish that asked examineers to tell if a subleave oution has some meaningful relevant labor leave outing from the bibliography. Among 849 (subleave oution, examineer) pairs that qualify for inclusion to our inference procedure, 110 had a correplying flag elevated in the examine. For these 110 pairs, authors of the current paper (CR, FE) manupartner examined the subleave outions and the examines, rerepairing subleave outions that satisfy the exclusion criteria. CR carry outed an initial, fundamental screening and all cases that needd a appraisement were rerepaird by FE—a program chair of the EC 2021 conference.
Overall, among the 110 concentrate pairs, only three asks to insert citations were set up to satisfy the exclusion criteria. All (subleave oution, examineer) pairs for these three subleave outions were deleted from the analysis, ensuring that examines written in the remaining (subleave oution, examineer) pairs are not susceptible to conset uping factor C1.
ICML. In ICML, the examineer establish did not have a flag for leave outing citations. Hence, to brimmingy ease the repartner leave outing citations conset uping factor, we would necessitate to examine all the 1,617 (subleave oution, uncited examineer) pairs qualifying for the inference step to rerepair those satisfying the aforealludeed exclusion criteria. Note that, in principle, cited examineers may also legitimately decrease the score because the subleave oution leave outes some of their past papers. However, this reduction in score would direct us to an underestimation of the effect (or, under the absence of citation bias, to the counterinstinctive honestion of the effect) and hence we consent it.
We commence from the analysis of (subleave oution, uncited examineer) pairs that qualify for our non-parametric analysis. There are 63 such pairs and analysis carry outed by an author of the current paper (IS—a laborflow chair of ICML 2020) set up that three of them satisfy the exclusion criteria. The correplying three subleave outions were deleted from our non-parametric analysis.
The fraction of (subleave oution, uncited examineer) pairs with a repartner leave outing citation of the examineer’s past paper in ICML is appraised to be 5% (3/63). As this number is relatively petite, the impact of this conset uping factor is confineed. In absence of the leave outing citation flag in the examineer establish, we choosed not to account for this conset uping factor in the parametric analysis of the ICML data. Thus, we encourage the reader to be conscious of this conset uping factor when clear uping the results of the parametric inference.
