Understanding Research Credibility: HDR Analysts Contribute to SCORE

Understanding Research Credibility: HDR Analysts Contribute to SCORE

by , | Jun 8, 2026 | Methods, Psychology

Systematizing Confidence in Open Research and Evidence (SCORE) was a large collaborative research project designed to improve judgments about scientific credibility in the social and behavioral sciences. Back when Peter and I were psychology professors, we independently chose to join this initiative, the results of which were recently published in Nature.  

In working with clients at HDR, we are often faced with the question of how much confidence should be placed in a model result or empirical finding. Decision-making happens under uncertainty, so part of the job is deciding not just what the evidence says, but how much weight it should carry. That broader question is one reason why the results of the SCORE project are relevant to our current work.  

SCORE was a DARPA-funded multi-method collaboration involving 865 researchers. As part of the initiative, the credibility of published findings was evaluated across three dimensions: reproducibility, robustness, and replicability.  

 

  • Researchers for the reproducibility study (Nature | Open Access) examined whether re-running an original analysis on the original data from published research articles will produce the same result reported by the original authors. Only approximately 54% of sampled papers were precisely reproducible. Papers from political science and economics journals had higher reproducibility rates compared to those from other disciplines. Paper recency and journal data sharing policies also predicted reproducibility.  
  • Researchers for robustness study (Nature | Open Access) tested whether conclusions hold when reasonable alternative analytical choices are applied to the same data. While 74% of the re-analyses reached the same conclusion as the original authors, quantitative results like effect sizes varied substantially. 
  • Researchers for the replicability study (Nature | Open Access) attempted independent replications of 274 claims drawn from 164 published papers. The replications were carefully designed, used the original materials when possible, and were peer-reviewed in advance. Only Fifty-five percent of claims replicated with statistically significant results in the original direction (see Figure 1). Replication rates varied somewhat across discipline and replication criteria. 
Figure 1: Each point shows the original and replication effect sizes for a replicated claim. Point size reflects the number of claims per paper. Replication effect sizes are shown as positive when the observed relationship follows the same direction as the original effect, and negative when the relationship is in the opposite direction. Points are classified as successful if the replication is statistically significant (p <  .05, two-sided) and in the same direction as the original effect; otherwise, they are classified as failed. 

These investigations remind us that research credibility is not a single property. A finding can survive one test and fail another. Reproducing an analysis, obtaining similar conclusions under alternative specifications, and observing the same result in new data each tell us something different. When the same findings are repeatedly observed, confidence in the robustness and reliability of the results increases.  In fact, replication rates are essential for estimating the probability of a hypothesis being true (see Doug’s paper in the American Statistician for more on this point).

Confidence about research findings has value beyond the academic community. In applied work, we rely on leveraging empirical research to support our claims.  Many HDR projects involve integrating multiple forms of evidence, each with different strengths and limitations. Historical observations, expert judgment, and external benchmarks may all serve as inputs into a model. Methods rooted in probabilistic reasoning and uncertainty quantification provide a framework for combining these sources while making confidence levels explicit. Rather than treating evidence as simply true or false, such approaches recognize that confidence should increase as findings remain consistent across multiple lines of inquiry and decrease when conclusions depend heavily on particular assumptions or analytical choices.  

SCORE’s datasets, methods, and findings are openly available. Take a look! This initiative represents one of the most comprehensive efforts to quantify reliability in published social and behavioral science, and Peter and I are proud to have played a small part in it. 

The Role of Calibration in Risk Analysis

The Role of Calibration in Risk Analysis

by , | Feb 19, 2026 | Calibration, Computing Value of Information, Methods

HDR’s Calibration Training: Team Calibrator – Hubbard Decision Research

Managing risk requires making decisions under uncertainty, often before complete information is available. One of the most common objections we encounter when working with clients concerns the lack of data to inform quantitative model inputs. When data are easily accessible, leveraging them to generate empirical inputs is straightforward. Gaps still arise, however, or data collection becomes impractical, especially early in a project. Under such conditions, we rely on “calibrated estimates” from subject matter experts (SMEs).

Every measurement instrument requires calibration, whether the instrument involves a precision manufacturing tool or human judgment used in model building. Calibration depends on consistent and unambiguous feedback. Prior to calibration, measurement error is often quite large. Humans tend to be systematically overconfident when making estimates, which introduces error and reduces model realism. Such overconfidence appears both in 90% confidence-interval range estimates and in probability estimates for binary events.

In training more than 3,000 individuals through consulting engagements and standalone programs, HDR has repeatedly observed this pattern of overconfidence. Calibration exercises demonstrably improve performance. Our methods, along with those developed by Philip Tetlock and Roger Cooke—whose pioneering work in this field is well worth reading—align stated confidence with empirical accuracy. Calibration in this context means that a claim of 90% confidence in a range estimate corresponds, across repeated estimates, to correctness approximately 90% of the time within a statistically allowable error range.

Figure 1 illustrates the typical pattern observed for calibration improvement over time. Despite systematic improvement, several confidence levels remain difficult for aggregated groups to calibrate perfectly. Slight overconfidence commonly appears when individuals state 50% confidence in a binary event. Such statements suggest complete uncertainty, yet outcomes across many trials indicate the presence of some informational advantage. Slight overconfidence also appears near the 100% confidence level, where allowable error approaches zero. To address these residual effects, estimates are aggregated across multiple experts and adjusted using each expert’s observed calibration performance. Aggregation reduces individual bias, and final calibration adjustments further fine-tune estimates, producing more reliable inputs for decision models.

Figure 1

 

Improved estimation quality forms a critical component of the Applied Information Economics (AIE) framework. Organizations frequently face data gaps. A common reaction treats further analysis as impossible until those gaps are filled, prompting immediate, large-scale data collection. In contrast, AIE emphasizes decision definition and measurement of current knowledge before engaging in such efforts. As illustrated in Figure 2, the framework uses quantitative analysis to show where reducing uncertainty would meaningfully affect the decision.

Figure 2

 

AIE helps organizations avoid a common decision-making pitfall: Measurement Inversion. As termed by Doug Hubbard, the Measurement Inversion describes a repeatedly observed pattern in which organizations measure and collect data on factors that have little or no effect on decisions. Millions of dollars can be poured into these efforts. Doug Hubbard often remarks, “I honestly wonder how this doesn’t impact the GDP.”  A reasonable response is that it probably does.

The first step of AIE, defining the decision, focuses on the choices under consideration, the outcomes that matter, and the uncertain variables that influence those outcomes. Risk analysis supports better decisions about which risk-reduction actions best serve the organization. Every organization faces many possible mitigations, controls, and initiatives, but determining which are justified requires quantitative analysis. Clear decision definition provides the foundation for prioritization.

Identification of variables that merit additional measurement follows from the next two AIE steps: modeling current knowledge and computing the value of additional information. Modeling current knowledge involves populating the model with “arm’s-reach” data and calibrated estimates. Calibration training ensures that uncertainty around each estimate is represented appropriately. Once the model is populated, analysis proceeds to calculation of the value of information (VOI), which indicates where additional measurement is worth the effort.

For example, consider a hypothetical capital project planning a major facility upgrade. Early cost and schedule data are incomplete, and the team considers delaying approval to collect detailed estimates across all work packages. AIE modeling using calibrated estimates shows that uncertainty in a small number of long-lead components drives most of the risk, while uncertainty in routine tasks has little impact on the decision. VOI analysis confirms that broad data collection would not change the outcome, whereas targeted measurement would.

VOI quantifies the economic impact of reducing uncertainty in specific model inputs. Ron Howard, a founder of decision analysis, introduced the concept in the 1960s, yet organizations still apply it infrequently. Many variables exhibit negligible information value, indicating that additional data collection or analysis would not affect decisions.

Before taking on a large data-collection effort, pause and ask whether that effort is actually justified. Avoid falling prey to Measurement Inversion. In many cases, decisions improve more from well-calibrated estimates than from indiscriminate data gathering. AIE provides a structured way to use calibrated judgment and value-of-information analysis to focus measurement on uncertainty that truly matters and to support better decisions.