Research-Problem Validity in Primary Research: Precision and Transparency in Characterizing Past Knowledge

Four validity types evaluate the approximate truth of inferences communicated by primary research. However, current validity frameworks ignore the truthfulness of empirical inferences that are central to research problem statements. Problem statements contrast a review of past research with other knowledge that extend, contradict, or call into question specific features of past research. Authors communicate empirical inferences, or quantitative judgments about the frequency (e.g., “few,” “most”) and variability (e.g., “on the one hand, on the other hand”) in their reviews of existing theories, measures, samples, or results. The authors code a random sample of primary research articles and show that 83% of quantitative judgments in their sample are both vague and their origin non-transparent, making it difficult to assess their validity. The authors review validity threats of current practices. The authors propose that documenting the literature search, how the search was coded, along with quantification facilitates more precise judgments and makes their origin transparent. This practice enables research questions that are more closely tied to the existing body of knowledge and allows for more informed evaluations of the contribution of primary research articles, their design choices, and how they advance knowledge. The authors discuss potential limitations of their proposed framework.