In the field of epidemiology, the challenges posed by personalization are combined with complex issues regarding the relationship between the collective and the individual dimension, which have accompanied the entire history of the discipline and touch upon its very nature and the definition of its tasks. A standard definition of epidemiology is: “the study of the distribution and determinants of health-related states or events in specified populations, and the application of this study to the prevention and control of health problems” [3]. To inform prevention and control, epidemiology adopts quantitative methods and in particular statistical tools. “Through a constantly evolving collection of cohort studies, randomised trials, and case-control studies, decades of epidemiological research have worked to quantify the nature and magnitude of associations between risk exposures and outcomes in studied populations” [4].

The definition of epidemiology presented above, like alternative ones, leaves a fundamental question open: how do we specify the population? Does one refer to the community as a whole or to narrower groups within it, limited to people at higher risk for various reasons? If the focus is on the broad population, the “prevention and control of health problems” will concern collective efforts organized by society to promote general health, targeting poverty, inequality, social and cultural determination of disease, pollution. Relevant expertise involves sociology, economics, behavioral and environmental science, media communication and policy making [5]. If the focus is on small groups, by contrast, prevention will address physiological and psychological determinants of disease, targeting lifestyle and metabolic factors like blood pressure, body mass index or gene mapping, or environmental exposure. Therefore, the contribution of clinicians, geneticists and molecular biologist becomes prevalent. Public health strategies, priorities and disciplinary positioning are different in the two cases.

In the first half of the 1900, epidemiology tendentially chose the first path, adopting a macro approach aimed at the population as a whole. Research in population epidemiology was focused on social and economic factors contributing to the spread of infectious diseases such as cholera and malaria or of conditions such as rickets and scurvy, whose control depends on water sanitation, food hygiene, sewage draining and disposal, and housing conditions. “Immense changes occurred during the second half of the 20th century. There was a shift of emphasis to non-communicable diseases, which reflected the decline of many infections and an increase in heart disease and cancer in many countries” [6]. The focus shifted from community ecology to individual risk factors and individual patients, bringing epidemiology closer to clinical practice. Methods and principles of epidemiology were applied to support decision-making in the management of individual cases. The expression “clinical epidemiology” was introduced at that time, to indicate a basic medical science devoted to “the application, by a physician who provides direct patient care, of epidemiological and biostatistical methods to the study of diagnostic and therapeutic processes in order to effect an improvement in health” [7]. The approach led to evidence-based medicine (EBM) which aims at “integrating individual clinical expertise with the best available external clinical evidence from systematic research” [8]. Here the goal is the care of individual patients, and the evidence is obtained with statistical tools, primarily the extremely powerful and successful RCTs (Randomized Control Trials, or in this context rather Randomized Clinical Trials).

The difference between population epidemiology and clinical epidemiology was already based on the distinction between general (macro) and individual (micro) reference, i.e., on the issues brought to the fore by the recent development of precision medicine, which has a specifical history in the field [9]. Since the 1980s, the debate within epidemiology highlighted the possible contrast between the two approaches. According to Last [10] the term “clinical epidemiology” is an oxymoron. By definition, in fact, epidemiology addresses a population, maybe reduced when it is a restricted population of patients rather than a community-based population. The CDC states that “In epidemiology, the patient is the community and individuals are viewed collectively” [11]. In Last’s opinion, therefore, the object of epidemiology cannot be individual persons as in the case of clinical epidemiology, because this would call into question the identity of the discipline and its research agenda linked to community-based preventive medicine and general health promoting activities.

Does then the contrast between individualization and generalization in PM introduce no real novelty in the field of epidemiology, besides the further sophistication of the quantitative techniques used by the discipline? Our research explores an alternative possibility, referring to the innovative aspects of recent algorithmic techniques.

Prior to ML, the statistical techniques and tools used in population epidemiology and clinical epidemiology were largely the same. The second half of the twentieth century had witnessed “the victory of statistics” [12–14]. The difference between the macro and the micro approach is that statistical tools refer in the former to data on a multiplicity of individuals (the population) and in the latter to narrower groups or even to single individuals (up to N-of-1 or single subject trials [15–18]).

In the last decades of the 20th century, epidemiology, biomedicine and clinical practice gradually came closer. RCTs became the standard testing procedure in all three fields. In cancer medicine and cardiology, for example, statistical designs of clinical trials created the very platform for clinical experimentation [13, 14, 19], and epidemiology became a core discipline in cancer control [20], (setting the agendas of public campaigns of cancer prevention [21, 22]). Epidemiology handbooks introduce both approaches seamlessly [11, 23].

Within this framework, the approach of PM might introduce a more radical discontinuity [24–26]. Models using ML algorithms can identify multifaceted non-linear patterns in large amounts of unsampled and uncontrolled data without a driving hypothesis, only searching for regularities that may anticipate future trends. For epidemiological research this makes a big difference. Researchers using machine learning have begun to explore heterogeneous treatment effects instead of overall average effects [27, 28]. Machine learning models can evaluate many variables to define heterogeneous effects without increasing statistical error, detecting very specific clusters of individuals who show different treatment effects [29]. The promises of PM are based on these opportunities.

But epidemiology, as we have seen, is defined precisely by its focus on population. Is this changing and how, in the era of precision medicine and machine learning? In epidemiological research using big data and machine learning, the chronic opposition between micro and macro dimensions might be entering a new evolutionary stage. Is this affecting the methodological approach and the distribution of goals and methods between population epidemiology and clinical epidemiology? Are the relationships with other disciplines changing? The next section of our paper explores these questions empirically.

To investigate these issues, we carried out a quantitative analysis of a dataset of articles in the years 2000–2019 collected on PubMed. We decided to restrict our analysis to articles published until 2020 because of the impact of the pandemic on research and on publications in population epidemiology, combined with the exponential growth in the use of ML and other AI techniques in recent years. The procedure used in our analysis is presented in detail in the Supplementary Appendixes. Here we describe the main steps of our investigation and their results.

We queried PubMed for research articles on epidemiology in journals in medicine, epidemiology, bioinformatics, and artificial intelligence in medicine, using the MeSH (Medical Subject Headings) terms for epidemiology and epidemiological methods. MeSH (Medical Subject Headings) terms are a standardized vocabulary indexing biomedical literature, ensuring precise and efficient searches in databases like PubMed. These terms are hierarchically organized and include synonyms to cover variations in terminology. Automatic term mapping enhances search accuracy by converting common language phrases into MeSH terms, ensuring comprehensive retrieval of relevant articles and improving search consistency. We used MeshTerms to query PubMed without specifying which of the Mesh Terms belongs to which journal (and thus to a specific research community). We found that Mesh Terms associated with machine learning and those associated with statistics were used in distinct journals. In general, standardization reveals areas of specialization, research trends, and interdisciplinary collaborations, making it easier to identify and differentiate between fields such as public health and clinical medicine. Additionally, MeSH terms streamline literature searches and assess journal scope. To distinguish between different methods, we used MeSH terms for methods associated with machine learning and for “classical” statistical methods (see Supplementary Appendix SA.1.1.1. for the query strings). The query results exceeded 1 million publications; we decided to restrict our analysis to the most important journals. We identified the top journals on medicine and health, epidemiology, computational biology, and medical informatics using Google scholar (https://scholar.google.com/citations?view_op=top_venues&hl=en; see Supplementary Appendix SA.1.1.4).

The queries resulted in 10,822 publications for machine learning methods, 66,408 publications for statistical methods (5,706 overlapping publications). We cleaned up our results excluding first duplicate publications in both corpora, then publications without abstracts and finally publications whose abstract was shorter than 50 words (that would not be suited for an exploration of the dataset with unsupervised topic modeling). Figure 1 presents a flow chart explaining the inclusion and exclusion criteria to select the publications for our analysis.

We have collected N = 67,838 publications in total, including abstracts, authors, keywords, departments, and year of publication for further inspection. These publications produced again two corpora, distinguishing publications using machine learning methods (n = 10,731) and publication using statistical methods (n = 57,107).

In the first step of our analysis, we investigated whether researchers in epidemiology using statistical methods and researchers using machine learning methods constitute two distinct research communities that publish their work in different journals. We inspected the journals where publications with the MeSH terms associated to machine learning methods and statistical methods (i.e., the two communities) present their research, finding in fact two distinct groups of journals, with only a slight overlap represented by PLOS One, where research from both communities appear) (Figure 2; see Supplementary Appendix SA.2).

To understand and properly interpret these preliminary findings, we conducted some interviews with bioinformaticians, genome researchers and epidemiologists, questioning them about the relationship between bioinformatics and epidemiology in the medical field. The interviews confirmed a trend towards an increase in the use of big data and machine learning, but at the same time the presence of significant obstacles. Epidemiologists increasingly need the results of bioinformatics, but the two disciplines still seem to be separated in approach, methods, and reference journals.

To investigate further, we reran the analysis using structural topic modeling (STM) [30]. STM is a method in natural language processing that enhances traditional topic modeling by incorporating document metadata, such as publication dates, authors, or journal types, into the analysis. This allows STM to identify latent topics within a corpus and understand how these topics vary with different covariates.

For our analysis, we used the publication date as the covariate (see Supplementary Appendix SA.3). The analysis was conducted using the R package stm (version 1.3.7) with R version 4.3. In topic modeling, it is necessary to fix the total number of topics. Initially, we ran a model comparison with topic numbers ranging from 5 to 150, with a step-size of 10. Based on statistical measures of exclusivity and coherence, we found no clear models with an optimal trade-off between these metrics. Therefore, we qualitatively inspected the models with k values from 50 to 90 and k = 150.

For each model, we labeled each topic and investigated which model provided sufficient resolution for our research question, combining expertise in epidemiology and data science. In doubtful cases, we further examined the abstracts for each topic and classified the topics as either “population epidemiology” or “clinical epidemiology.”

After labeling topics and inspecting abstracts, we concluded that k = 150 provided sufficient resolution for our purposes. We then analyzed the topics over time (see Supplementary Appendix SAB). To investigate our hypothesis, we divided the first 40 topics into three groups: those closer to the clinical side of epidemiology (17 topics), those closer to the population side of epidemiology (15 topics), and those that could not be plausibly assigned to either side (black-labeled topics) (8 topics). See Supplementary Appendix SA.3 for the list of topics.