4  Research Transparency Report

4.1 Timeline

Puzzled by a finding in Jankin et al. (2024), two authors experimented with the idea of multiverse analysis for topic modeling.

Artefact: First git commit in a private repository. See also Footnote 8 in the manuscript.

After an internal presentation of the idea to the research team, more authors joined the project. The team developed the initial idea into a systematic investigation. Guided by Pipal et al. (2022), the team worked on a registration.

The protocol was preregistered on OSF as an open end preregisration.

Artefact: Preregistration on Open Science Framework (OSF)

First active phase of the research.

First draft was submitted for the ICA Conference (Computational Methods Division).

First submission to Communication Methods and Measures (CMM).

ICA submission was accepted

First decision from CMM: Revise and resubmit

Second active phase of the research.

Artefacts:

The following were developed based on the comments and queries from the reviewers.

• The forensic investigation of the reproducibility of Curini and Vignoli (2021). (Git commit)
• The variance decomposition analysis in Coda. (Git commit)
• The distribution \(\rho\) in Coda. (Git commit)

Based on the forensic investigation, we identified issues with the dictionary of Curini and Vignoli (2021), which was fixed. (Git commit) (See below) This also led to a rerun of the multiverse analysis of Curini and Vignoli (2021). We have kept the old binary artefacts. (intermediate/curini/oldruns)

Presentation at the ICA Conference.

ICA computational methods division best paper award.

Revised version #1 submitted to CMM.

Second decision from CMM: Revise and resubmit.

Revised version #2 submitted to CMM.

Final decision from CMM: Accept.

Git repository made public.

Binary artefacts deployed on Zenodo.

Artefact: Zenodo

Online appendix website made public.

Artefact: Website

Fixed an inconsistency in the code. (See below)

Artefact: Git commit

4.2 Preregistration deviations

(Also reported in the manuscript)

The current investigation was deviated from the preregistration in the following two ways. First, in the original preregistration, we included an additional inclusion criterion: “Studies represent a diverse selection of cases, such as text genres or domains as well as covering different languages.” This criterion was practically not used because the second criterion was too restrictive and we could not include enough studies with complete code and data. Using the typology of Lakens (2024), this deviation is an unforeseen event. The impact of this deviation should be negligible, as we did not reject any primary study because of text genres, domains, or languages.

Second, in the original preregistration, we based on Denny and Spirling (2018) and Maier et al. (2018) and included stemming as a considered analytical path. As mentioned above, this analytical path was not considered for Takano et al. (2023) because stemming is not possible for the Japanese language. Using the typology of Lakens (2024), this deviation is a violation of untested assumption. We incorrectly assumed in the preregistration that any text can be stemmed. The impact of this deviation is relatively higher, as that makes the multiverse size not comparable between that of Takano et al. (2023) and the rest of the five cases.

4.3 Other deviations

4.3.1 Italian dictionary in Curini and Vignoli (2021)

The Italian dictionary for Curini and Vignoli (2021) was modified during the course of the study. The modification was to make the dictionary also applicable to the stemming and the lemmatization cases (e.g., democrazia to democraz*).

The change led to a rerun. The binary artefacts of the previous run before the change were preserved as intermediate/curini/oldruns on Zenodo.

4.3.2 Dummy variable in Tvinnereim and Fløttum (2015)

Tvinnereim and Fløttum (2015) entered education as numerical variable (edu3) in the topic modeling, i.e.,

# Select the 4 best models out of 20 runs (see Vignette, pp. 8-9)
ncpSelect2 <- selectModel(out$documents,
                          out$vocab,
                          K=4,
                          prevalence =~ meta$concern+meta$humanmade+meta$efficacy+meta$edu3+meta$gender+meta$age, 
                          max.em.its=100, 
                          data=out$meta,
                          runs=20,
                          seed=5926696) #, emtol=1)

However, the variable was entered as a dummy variable in the effect estimation.

# Figures
# Co-variation with demographics and concern
# Generate education binary: university or not
univdummy <- as.numeric(meta$edu3==3)
print(univdummy)  
summary(univdummy)
table(meta$edu3, univdummy)
meta$univdummy <- univdummy

# Figure 1: Topical prevalence over co-variates
prep <- estimateEffect(1:4 ~ concern+univdummy+gender+age, 
                       ncpPrevFit,
                       meta=meta, 
                       uncertainty = "Global")

In the original analysis, we also entered this as a binary variable in the anchoring procedure. We fixed it in the code after acceptance to not follow this in our code (Git commit). The reason was that the binary dummy was only used for another visualization. We tested that entering education as numerical variable or binary dummy does not influence the anchoring and therefore also not influence the subsequent effect estimation of age.

library(stm)
#> stm v1.3.7 successfully loaded. See ?stm for help. 
#>  Papers, resources, and other materials at structuraltopicmodel.com

args <- tmmv.parse_args_train(slug = "tvinnereim", .current_run = 1)

analytical_paths <- list(
    token_normalization = c("none", "lemmatization", "stemming"),
    stopword_removal = c(TRUE, FALSE),
    trimming = c(TRUE, FALSE),
    alternative_model = c(TRUE, FALSE),
    k_setting = c(1, 2, 3), #K original, alt1, alt2
    iteration_setting = c(1, 2, 3) #iter original, alt1, alt2    
)

settings_df <- expand.grid(analytical_paths, stringsAsFactors = FALSE)

settings <- settings_df |> purrr::transpose() 


.read <- function(setting, args) {
    readRDS(tmmv.get_rds_filename(
        setting,
        here::here(args$output_dir)
    ))
}

models <- purrr::map(settings, .read, args = args)
names(models) <- purrr::map_chr(settings,
                                rlang::hash)

anchor_setting <- list(token_normalization = "stemming",
                       stopword_removal = TRUE,
                       trimming = TRUE,
                       alternative_model = FALSE,
                       k_setting = 1, 
                       iteration_setting = 1)

anchor_mod <- models[[rlang::hash(anchor_setting)]]

.gen_anchor <- function(anchor_mod, edu3 = TRUE) {
    if (edu3) {
        est <- stm::estimateEffect(
    ~ concern + edu3 + gender + age,
    stmobj = anchor_mod$mod,
    metadata = anchor_mod$docvars
    )
    } else {
        anchor_mod$docvars$univdummy <- as.numeric(anchor_mod$docvars$edu3 == 3)
        est <- stm::estimateEffect(
    ~ concern + univdummy + gender + age,
    stmobj = anchor_mod$mod,
    metadata = anchor_mod$docvars
    )
    }
    max_topic_index <- which.max(
    purrr::map_dbl(est$parameters, \(x) {
        mean(purrr::map_dbl(x, \(y) y$est["age"]))
    })
    )

    return(anchor_mod$theta[, max_topic_index])
}


## the anchors are the same
stopifnot(identical(.gen_anchor(anchor_mod, edu3 = TRUE),
                    .gen_anchor(anchor_mod, edu3 = FALSE)))

4.4 Additional discourses

Some ideas in the current study were developed in parallel to Linde et al. (2026) (currently as preprint).

4.5 References

Curini, Luigi, and Valerio Vignoli. 2021. “Committed Moderates and Uncommitted Extremists: Ideological Leaning and Parties’ Narratives on Military Interventions in Italy.” Foreign Policy Analysis 17 (3). https://doi.org/10.1093/fpa/orab016.

Denny, Matthew J., and Arthur Spirling. 2018. “Text Preprocessing for Unsupervised Learning: Why It Matters, When It Misleads, and What to Do about It.” Political Analysis 26 (2): 168–89. https://doi.org/10.1017/pan.2017.44.

Jankin, Slava, Alexander Baturo, and Niheer Dasandi. 2024. “Words to Unite Nations: The Complete United Nations General Debate Corpus, 1946–Present.” Journal of Peace Research, ahead of print, November. https://doi.org/10.1177/00223433241275335.

Lakens, Daniël. 2024. “When and How to Deviate from a Preregistration.” Collabra: Psychology 10 (1). https://doi.org/10.1525/collabra.117094.

Linde, Maximilian, Jun Sun, Paul Balluff, Danica Radovanović, and Chung-hong Chan. 2026. Making Uncertainty Visible: Multiverse Analysis for Robust Computational Social Science. arXiv. https://doi.org/10.48550/ARXIV.2605.19745.

Maier, Daniel, A. Waldherr, P. Miltner, et al. 2018. “Applying LDA Topic Modeling in Communication Research: Toward a Valid and Reliable Methodology.” Communication Methods and Measures 12 (2-3): 93–118. https://doi.org/10.1080/19312458.2018.1430754.

Pipal, Christian, Hyunjin Song, and Hajo G. Boomgaarden. 2022. “If You Have Choices, Why Not Choose (and Share) All of Them? A Multiverse Approach to Understanding News Engagement on Social Media.” Digital Journalism, March, 1–21. https://doi.org/10.1080/21670811.2022.2036623.

Takano, Ryota, Akiko Matsuo, and Kazuaki Kawano. 2023. “Development of a Japanese Version of the Awe Experience Scale (AWE-s): A Structural Topic Modeling Approach.” F1000Research 12: 515. https://doi.org/10.12688/f1000research.134275.2.

Tvinnereim, Endre, and Kjersti Fløttum. 2015. “Explaining Topic Prevalence in Answers to Open-Ended Survey Questions about Climate Change.” Nature Climate Change 5 (8): 744–47. https://doi.org/10.1038/nclimate2663.