Table of Contents¶
Introduction¶
The Wikimedia Foundation's Editing team is working to improve how contributors communicate on Wikipedia using talk pages through a series of incremental improvements that will be released over time.
As part of this effort, the Editing team introduced a new workflow for starting new topic threads on talk pages, across Wikipedia's 16 talk namespaces. This new workflow is intended to make it more intuitive for Junior Contributors to initiate conversations in ways other contributors can easily reply to and to help Senior Contributors do the same, with less effort.
The new topic tool tool provides an an inline form for adding new topics. In addition, the language throughout the workflow was adjusted to be more topic-specific.
The team ran an AB test of the New Topic Tool from 27 January 2022 through 25 March 2022 [^timeline] to assess the efficacy of this new feature. The test included all logged-in and logged-out users that edited a talk page at one the 20 participating Wikipedias during the duration of the AB test (see full list of participating Wikipedias in task description and conditions outlined in the methodology section below). During this test, 50% of users included in the test had the New Topic tool automatically enabled, and 50% did not. This report focues on the results for logged-in users.
[^timeline]: Note that we excluded logged-out data collected from from 27 January 2022 to 17 February 2022 in this analysis due to errors in the bucketing implementation that did not accurately log all the events in the AB test.
You can find more information about features of this tool and project updates on the project page.
Methodology¶
The AB test was run on a per Wikipedia basis and contributors included in the test were randomly assigned to either the control (new topic tool disabled by default) or treatment (new topic tool enabled by default). We excluded logged-in users that have previously used the New Topic tool (as determined by the state of the discussiontools-newtopictool-opened preference) but included users that had explicitly enabled or disabled the overall Discussion tools beta feature. Users at these Wikipedias were still able to turn the tool on or off the tool in Special:Preferences; however, they remained in the same group they were buketed in for the duration of the test.
Upon conclusion of the test on 25 March 2022, we recorded a total of 19,676 new topic attempts initiated across both test groups by 6,258 distinct contributors across all experience levels. A total of 3,630 (58%) of these contributors were identified as Junior Contributors. Data was collected in EditAttemptStep and talk_page_edit .
In this test, a user can add a new topic using the New Topic Tool or using the existing add new section link interface. For the purpose of this analysis, these two types of editing experiences are defined as follows:
New Topic Tool: Any edit to add a new topic (section) to a talk page namespace made with the new topic tool. The new topic tool allows edits using both wikitext and source mode. New Topic tool events were sampled at 100%.
Recorded in EditAttemptStep as: event.action = 'init', event.integration = 'discussiontools', event.init_type = 'section'
Existing Add New Section Link: Any edit to add a new topic (section) using the existing section=new link and workflow. These events were sampled at a rate of 1/16, or 6.125%.
Recorded in the EditAttemptStep as: event.action = 'init', event.integration = 'page' , event.init_type = 'section', event.init_mechanism IN ('url-new', 'new')
We excluded the following types of edits from this analysis: (1) edits made with the reply tool, (2) full-page edits to create a new page or to an existing page, (3) corrective edits to an existing section. Note: It's possible to create a new section using full page editing but these were excluded for the following reasons: (1) we do not have instrumentation to decipher beween an attempt to make an edit to existing text on the page vs an edit attempt to create a new section using full page editing; and (2) this method exists in both the test and control groups and we do not believe would be impacted by the appearance of the new topic tool.
See the following Phabricator tickets for further details regarding instrumentation and implementation of the AB test:
library(IRdisplay)
display_html(
'<script>
code_show=true;
function code_toggle() {
if (code_show){
$(\'div.input\').hide();
} else {
$(\'div.input\').show();
}
code_show = !code_show
}
$( document ).ready(code_toggle);
</script>
<form action="javascript:code_toggle()">
<input type="submit" value="Click here to toggle on/off the raw code.">
</form>'
)
shhh <- function(expr) suppressPackageStartupMessages(suppressWarnings(suppressMessages(expr)))
shhh({
library(tidyverse)
# Modeling
library(brms)
library(lme4)
library(tidybayes)
set.seed(5)
# Tables:
library(gt)
library(gtsummary)
})
options(repr.plot.width = 15, repr.plot.height = 10)
# collect all new topic tool attempts and saves
# Add a join with talk_page_edit to ensure all saved edits are included due to bug https://phabricator.wikimedia.org/T305541
query <-
"
--find all edit attempts
WITH edit_attempts AS (
SELECT
wiki AS wiki,
event.user_id as user_id,
event.editing_session_id as edit_attempt_id,
event.bucket AS experiment_group,
event.is_oversample AS is_oversample,
event.integration AS editing_method,
If(event.integration == 'discussiontools', 1, 0) AS new_topic_tool_used,
CASE
WHEN event.init_type = 'section' AND event.integration == 'discussiontools' THEN 'new_topic_tool'
WHEN event.init_type = 'section' AND event.integration == 'page' AND event.init_mechanism IN ('url-new', 'new') THEN 'new_section_link'
ELSE 'NA' -- check to make sure all edit types accounted for in above list
END AS section_edit_type,
event.user_editcount AS experience_level
FROM event.editattemptstep
WHERE
-- only in participating wikis
wiki IN ('amwiki', 'bnwiki', 'zhwiki', 'nlwiki', 'arzwiki', 'frwiki', 'hewiki', 'hiwiki',
'idwiki', 'itwiki', 'jawiki', 'kowiki', 'omwiki', 'fawiki', 'plwiki', 'ptwiki', 'eswiki', 'thwiki',
'ukwiki', 'viwiki')
-- since deployment
AND year = 2022
AND ((month = 01 and day >= 27) OR (month = 02) OR
(month = 03 and day <= 25))
-- remove bots
AND useragent.is_bot = false
-- look at only desktop events
AND event.platform = 'desktop'
-- review all talk namespaces
AND event.page_ns % 2 = 1
-- only users in AB test
AND event.bucket IN ('test', 'control')
-- only registered user
AND event.user_id != 0
AND event.action = 'init'
-- discard VE/Wikieditor edits to create new page or reply tool edits
AND NOT (
-- not a reply tool edit
(event.init_type = 'page' AND event.integration = 'discussiontools') OR
-- not an wikitext edit to create a new page
(event.init_type = 'page' AND event.init_mechanism IN ('url-new', 'new') AND event.integration = 'page') OR
-- not a corrective edit to an existing section
(event.init_type = 'section' AND event.init_mechanism IN ('click', 'url') AND event.integration == 'page') OR
-- not a full page edit
(event.init_type = 'page' AND event.init_mechanism IN ('click', 'url') AND event.integration = 'page')
)),
-- find all published comments
published_dt_new_topics AS (
SELECT
performer.user_id AS user_id,
session_id AS edit_save_id,
`database` AS wiki
FROM event.mediawiki_talk_page_edit
WHERE
year = 2022
AND ((month = 01 and day >= 27) OR (month = 02) OR
(month = 03 and day <= 25))
-- only in participating wikis
AND `database` IN ('amwiki', 'bnwiki', 'zhwiki', 'nlwiki', 'arzwiki', 'frwiki', 'hewiki', 'hiwiki',
'idwiki', 'itwiki', 'jawiki', 'kowiki', 'omwiki', 'fawiki', 'plwiki', 'ptwiki', 'eswiki', 'thwiki',
'ukwiki', 'viwiki')
),
published_section_link_new_topics AS (
SELECT
event.user_id as user_id,
event.editing_session_id AS edit_save_id,
wiki AS wiki
FROM event.editattemptstep
WHERE
-- only in participating wikis
wiki IN ('amwiki', 'bnwiki', 'zhwiki', 'nlwiki', 'arzwiki', 'frwiki', 'hewiki', 'hiwiki',
'idwiki', 'itwiki', 'jawiki', 'kowiki', 'omwiki', 'fawiki', 'plwiki', 'ptwiki', 'eswiki', 'thwiki',
'ukwiki', 'viwiki')
AND year = 2022
AND ((month = 01 and day >= 27) OR (month = 02) OR
(month = 03 and day <= 25))
AND event.action = 'saveSuccess'
)
-- main query
SELECT
eas.wiki,
eas.user_id,
edit_attempt_id,
experiment_group,
is_oversample,
editing_method,
new_topic_tool_used,
section_edit_type,
-- was saved in either talk page edit or editattemptstep
IF ((section_edit_type = 'new_topic_tool' AND (tpe_save.edit_save_id IS NOT NULL OR eas_save.edit_save_id IS NOT NULL))
OR (section_edit_type = 'new_section_link' AND (tpe_save.edit_save_id IS NOT NULL OR eas_save.edit_save_id IS NOT NULL)), 1, 0) AS edit_success,
experience_level
FROM edit_attempts eas
LEFT JOIN published_dt_new_topics tpe_save ON
eas.edit_attempt_id = tpe_save.edit_save_id AND
eas.wiki = tpe_save.wiki
LEFT JOIN published_section_link_new_topics eas_save ON
eas.edit_attempt_id = eas_save.edit_save_id AND
eas.wiki = eas_save.wiki
"
new_topic_attempts <- wmfdata::query_hive(query)
Don't forget to authenticate with Kerberos using kinit
# data reformatting and cleanup
#set factor levels with correct baselines
new_topic_attempts$section_edit_type <-
factor(
new_topic_attempts$section_edit_type,
levels = c("NA", "full_page_edits", "new_section_link", "new_topic_tool"),
labels = c("NA", "Full page edits", "Existing add new section link", "New topic tool")
)
new_topic_attempts$edit_success <-
factor(
new_topic_attempts$edit_success,
levels = c(0, 1),
labels = c("Not Complete", "Complete")
)
# reformat user-id and adjust to include wiki to account for duplicate user id instances.
# Users do not have the smae user_id on different wikis
new_topic_attempts$user_id <-
as.character(paste(new_topic_attempts$user_id, new_topic_attempts$wiki, sep ="-"))
#clarfiy wiki names
new_topic_attempts <- new_topic_attempts %>%
mutate(
wiki = case_when(
#clarfiy participating project names
wiki == 'amwiki' ~ "Amharic Wikipedia",
wiki == 'bnwiki' ~ "Bengali Wikipedia",
wiki == 'zhwiki' ~ "Chinese Wikipedia",
wiki == 'nlwiki' ~ 'Dutch Wikipedia',
wiki == 'arzwiki' ~ 'Egyptian Wikipedia',
wiki == 'frwiki' ~ 'French Wikipedia',
wiki == 'hewiki' ~ 'Hebrew Wikipedia',
wiki == 'hiwiki' ~ 'Hindi Wikipedia',
wiki == 'idwiki' ~ 'Indonesian Wikipedia',
wiki == 'itwiki' ~ 'Italian Wikipedia',
wiki == 'jawiki' ~ 'Japanese Wikipedia',
wiki == 'kowiki' ~ 'Korean Wikipedia',
wiki == 'omwiki' ~ 'Oromo Wikipedia',
wiki == 'fawiki' ~ 'Persian Wikipedia',
wiki == 'plwiki' ~ 'Polish Wikipedia',
wiki == 'ptwiki' ~ 'Portuguese Wikipedia',
wiki == 'eswiki' ~ 'Spanish Wikipedia',
wiki == 'thwiki' ~ 'Thai Wikipedia',
wiki == 'ukwiki' ~ 'Ukrainian Wikipedia',
wiki == 'viwiki' ~ 'Vietnamese Wikipedia'
)
)
# Create new column to identify Junior and Non-Junior Contributors
new_topic_attempts <- new_topic_attempts %>%
mutate(
is_junior = case_when(
#clarfiy participating project names
experience_level < 100 ~ "Junior Contributor",
experience_level >= 100 ~ "Non-Junior Contributor"
),
is_junior = factor(is_junior,
levels = c("Non-Junior Contributor", "Junior Contributor")
))
New topic attempts in each group¶
Since all Discussion Tool events including new topic tool events are oversampled (100% sampling rate) in EditAttemptStep, we removed oversampled events so we can directly compare the balance between each experiment group.
new_topic_attempts_bygroup <- new_topic_attempts %>%
filter(is_oversample == 'false') %>% #All Discussion Tool events are oversampled - removing to check balance.
group_by(experiment_group) %>%
summarise(users = n_distinct(user_id),
attempts = n_distinct(edit_attempt_id), .groups = 'drop')
new_topic_attempts_bygroup
| experiment_group | users | attempts |
|---|---|---|
| <chr> | <int> | <int> |
| control | 891 | 1392 |
| test | 881 | 1549 |
New Topic Tool Completion Rate¶
Our key performance indicator (KPI) for this analysis is new topic completion rate. For the purpose of this analysis, we are defining the completion rate as the percent of contributors that successfully published (event.action = 'saveSuccess' in EditAttemptStep)[^instrumentation] at least one comment after clicking the Add topic / New section link interface (event.action = 'init') during the time of the AB test.
Note that this does not take into account the number of attempts it took for the user to publish or the duration of their editing sessions. For comparison purposes, we also reviewed completion rate defined as the percent of all new topic edit attempts by Junior Contributors that were successfully published. This was also the dataset we used to model the impact of the new topic tool as the model accounts for both user and wiki experience on the success of each edit attempt.
[^instrumentation]: During this analysis, we identified a bug where some instances of edits sessions completed using the new topic tool were not correctly recorded as being sucessfully saved in EditAttemptStep. As a result, we also used the talk_page_edit schema to more accurately account for all edits posted by the new topic tool by joining this data with init events identifed in EditAttemptStep.
New Topic Tool Completion Rate by Junior Contributors¶
We first calculated the new topic completion rate for Junior Contributors overall and across each participating Wikipedia.
Overall new topic completion rate by Junior Contributors¶
New topic completion rate defined as percent of edit attempts that are successfully published¶
# Completion Rate By Session
new_topic_attempts_jc_bysession <- new_topic_attempts %>%
filter(is_junior == 'Junior Contributor') %>% #only jc
group_by (wiki, section_edit_type) %>%
summarise(n_attempts = n_distinct(edit_attempt_id),
n_completions = n_distinct(edit_attempt_id[edit_success == 'Complete']),
#n_completions = sum(edit_success == 'Complete'),
new_topic_tool_used = as.integer(ifelse(sum(section_edit_type== 'New topic tool'), 1, 0)),
.groups = 'drop')
# By session
new_topic_attempts_jc_bysession_all <- new_topic_attempts_jc_bysession %>%
group_by(section_edit_type) %>%
summarise(n_attempts = sum(n_attempts),
n_attempts_completed = sum(n_completions),
completion_rate = paste0(round(n_attempts_completed / n_attempts *100, 1), "%"),
.groups = 'drop'
)
new_topic_attempts_jc_bysession_all
| section_edit_type | n_attempts | n_attempts_completed | completion_rate |
|---|---|---|---|
| <fct> | <int> | <int> | <chr> |
| Existing add new section link | 476 | 165 | 34.7% |
| New topic tool | 5799 | 2054 | 35.4% |
New topic completion rate defined as percent of Junior Contributors that successfully publish at least 1 new topic¶
# Completion Rate By Junior Contributors
new_topic_attempts_jc <- new_topic_attempts %>%
filter(is_junior == 'Junior Contributor') %>% #only jc
group_by (wiki, section_edit_type, user_id) %>%
summarise(n_attempts = n_distinct(edit_attempt_id),
n_completions = n_distinct(edit_attempt_id[edit_success == 'Complete']),
edit_success = ifelse(sum(n_completions >= 1), 'Complete', 'Not Complete'), #redefine edit success as user completed at least 1 edit attempt
new_topic_tool_used = as.integer(ifelse(sum(section_edit_type== 'New topic tool'), 1, 0)),
.groups = 'drop')
# By Contributor
# Review edit completion rate by editing method
new_topic_attempts_jc_all <- new_topic_attempts_jc %>%
group_by(section_edit_type) %>%
summarise(n_users = n_distinct(user_id),
n_users_completed = sum(n_completions >= 1), #user completed at least 1 edit
completion_rate = paste0(round(n_users_completed / n_users *100, 1), "%"),
.groups = 'drop'
)
# Create table of completion rate
new_topic_attempts_jc_all_table <- new_topic_attempts_jc_all %>%
gt() %>%
tab_header(
title = "Junior contributors new topic completion rate",
subtitle = "across all participating Wikipedias"
) %>%
cols_label(
section_edit_type = "Editing method",
n_users = "Number of users attempted",
n_users_completed = "Number of users completed",
completion_rate = "New topic completion rate"
) %>%
tab_footnote(
footnote = "Defined as percent of contributors that attempted and published at least 1 new topic",
locations = cells_column_labels(
columns = 'completion_rate'
)
) %>%
tab_footnote(
footnote = "Sampling rate for Non-New Topic Tool events is 6.25%",
locations = cells_body(
columns = 'section_edit_type', rows = 1)
) %>%
tab_footnote(
footnote = "Sampling rate for New Topic Tool events is 100%",
locations = cells_body(
columns = 'section_edit_type', rows = 2)) %>%
gtsave(
"new_topic_attempts_jc_all_table.html", inline_css = TRUE)
IRdisplay::display_html(data = new_topic_attempts_jc_all_table, file = "new_topic_attempts_jc_all_table.html")
| Junior contributors new topic completion rate | |||
|---|---|---|---|
| across all participating Wikipedias | |||
|
1
Defined as percent of contributors that attempted and published at least 1 new topic
2
Sampling rate for Non-New Topic Tool events is 6.25%
3
Sampling rate for New Topic Tool events is 100%
|
p <- new_topic_attempts_jc_all %>%
ggplot(aes(x= section_edit_type, y = n_users_completed / n_users, fill = section_edit_type)) +
geom_col(position = 'dodge') +
geom_text(aes(label = paste(completion_rate), fontface=2), vjust=1.2, size = 8, color = "white") +
scale_y_continuous(labels = scales::percent) +
scale_x_discrete(labels = c("Existing add new section link", "New topic tool")) +
labs (y = "Percent of junior contributors ",
x = "Editing method",
title = "Junior contributors new topic completion rate \n across all participating Wikipedias",
caption = "Defined as percent of contributors that make a new topic attempt and publish at least 1 new topic") +
scale_fill_manual(values= c("#999999", "steelblue2")) +
theme(
panel.grid.minor = element_blank(),
panel.background = element_blank(),
plot.title = element_text(hjust = 0.5),
text = element_text(size=16),
legend.position= "none",
axis.line = element_line(colour = "black"))
p
ggsave("Figures/new_topic_attempts_jc_all.png", p, width = 16, height = 8, units = "in", dpi = 300)
Overall, 44.5% of all Junior Contributors that made a new topic attempt were able to successfully publish at least 1 new topic with the new topic tool, while 36.4% of all Junior Contributors successfully published a new topic using the previous new section link editing workflow. This represents a 8 percentage point (36.4% → 44.5%); 22% observed increase in new topic completion rate.
Note that this does not take into account the number of attempts it took for the user to publish or the duration of their editing sessions. If we look just at the percent of editing sessions succesfully completed, there was a 0.7 percentage point (34.7% → 35.4%); 2% observed increase in new topic completion rate for editing sessions completed using the new topic tool.
New topic completion rate by participating Wikipedia¶
# Review edit attempts by editing method and wiki
new_topic_attempts_jc_bywiki <- new_topic_attempts_jc %>%
group_by(wiki, section_edit_type) %>%
summarise(n_users = n_distinct(user_id),
n_users_completed = sum(n_completions >=1), #user completed at least 1 edit
completion_rate = paste0(round(n_users_completed / n_users *100, 1), "%"),
.groups = 'drop') %>%
#determine credible intervals
cbind(as.data.frame(binom:::binom.bayes(x = .$n_users_completed, n = .$n_users, conf.level = 0.95, tol = 1e-10))) %>%
mutate(lower = round(lower,2),
upper = round(upper, 2))
new_topic_attempts_jc_bywiki_tbl <- new_topic_attempts_jc_bywiki %>%
select(c(1,2,3,4,5,12,13)) %>% #remove unneeded rows
gt() %>%
tab_header(
title = "Junior contributors new topic completion rate by participating Wikipedia"
) %>%
cols_label(
wiki = "Wikipedia",
section_edit_type= "Editing method",
n_users = "Number of users attempted",
n_users_completed = "Number of users completed",
completion_rate = "Completion rate",
lower = "CI (Lower Bound)",
upper = "CI (Upper Bound)"
) %>%
tab_footnote(
footnote = "Defined as percent of contributors that make a new topic attempt and publish at least 1 new topic",
locations = cells_column_labels(
columns = 'completion_rate'
)
) %>%
tab_footnote(
footnote = "Sampling rate for Non-New Topic Tool events is 6.25%",
locations = cells_column_labels(
columns = 'section_edit_type'
)
) %>%
tab_footnote(
footnote = "Sampling rate for New Topic Tool events is 100%",
locations = cells_column_labels(
columns = 'section_edit_type'
)) %>%
tab_footnote(
footnote = "95% credible intervals. There is a 95% probability that the parameter lies in this interval",
locations = cells_column_labels(
columns = c('lower', 'upper')
)) %>%
gtsave(
"new_topic_attempts_jc_bywiki_tbl .html", inline_css = TRUE)
IRdisplay::display_html(data = new_topic_attempts_jc_bywiki_tbl, file = "new_topic_attempts_jc_bywiki_tbl .html")
new_topic_attempts_jc_bywiki_tbl
| Junior contributors new topic completion rate by participating Wikipedia | ||||||
|---|---|---|---|---|---|---|
|
1
Sampling rate for Non-New Topic Tool events is 6.25%
2
Sampling rate for New Topic Tool events is 100%
3
Defined as percent of contributors that make a new topic attempt and publish at least 1 new topic
4
95% credible intervals. There is a 95% probability that the parameter lies in this interval
|
NULL
# Plot edit completion rates for each wiki
dodge <- position_dodge(width=0.9)
p <- new_topic_attempts_jc_bywiki %>%
filter(!(wiki %in% c('Amharic Wikipedia', 'Egyptian Wikipedia', 'Thai Wikipedia'))) %>% # remove wikis where there are under 10 events as we
ggplot(aes(x= section_edit_type, y = n_users_completed / n_users, fill = section_edit_type)) +
geom_col(position = 'dodge') +
geom_errorbar(aes(ymin = lower, ymax = upper), color = 'red', size = 1, alpha = 0.5, position = dodge, width = 0.25) +
geom_text(aes(label = paste(completion_rate), fontface=2), vjust=1.2, size = 5, color = "white") +
facet_wrap(~ wiki) +
scale_y_continuous(labels = scales::percent) +
labs (y = "Percent of junior contributors ",
title = "Junior contributors new topic completion rate by participating Wikipedia",
caption = "Amharic, Egyptian, Thai, and Oromo Wikipedias removed from analysis due to insufficient events \n
Red error bars: Reflect 95% credible intervals") +
scale_fill_manual(values= c("#999999", "steelblue2"), name = "Editing Method", labels = c("Existing add new section link", "New topic tool")) +
theme(
panel.grid.minor = element_blank(),
panel.background = element_blank(),
plot.title = element_text(hjust = 0.5),
text = element_text(size=16),
legend.position="bottom",
axis.text.x = element_blank(),
axis.title.x=element_blank(),
axis.line = element_line(colour = "black"))
p
ggsave("Figures/new_topic_attempts_jc_bywiki_plot.png", p, width = 16, height = 8, units = "in", dpi = 300)