Notebook

Opt-out rate for desktop improvements feature on test wikis¶

Questions/Notes for Olga:¶

https://superset.wikimedia.org/r/310

PrefUpdate captures user preference modifications at registration - not isolated to just user initiated changes. https://phabricator.wikimedia.org/T260867#6403042. Caused by change merged on August 3rd https://phabricator.wikimedia.org/rEWMV30731c2c748a0ea34f798628fe7961b928572a23. This is likely why only the numbers for users with 0 edits are so high.
Is the table https://docs.google.com/spreadsheets/d/1svlRDsYjOhjXkIV1a-DqLYTcLIFuYoc6pHHz8_VmQcc/edit#gid=625518477 correct? Numbers not matching. For example, on frwiki there are more users changing to legacy, which doesn't make sense if legact is default. We would first need them to opt-in to modern before opt-out to legacy.
For thos shown the modern vector as default, we will need another way to find the total number of people that had access prior to opt-out (number that viewed a page?)

Purpose¶

We would like to know the number of logged-in users that have opted out from the desktop improvements projects from the beginning of the deployments on the test wikis (target is less than 40%)

Metrics¶

Report on the percentage of logged-in users that have opted out from the new version of vector:

per test wiki
per user edit bucket

Approach¶

Data comes from the PrefUpdate table, which records preference changes for registered users.

Due to the issue of accidental opt-ins/opt-outs, we will have to only consider the percentage of opt-outs after the date of deployment on each wiki as listed below:

euwiki: Jul 22nd (Default Skin Version: 2)
frwikitionary: (Default Skin Version: 2)
officewiki: (Default Skin Version: 2)
ptwikiversity: (Default Skin Version: 2)
fawiki: July 28 2020 (Default Skin Version: 1)
frwiki: Aug 5 2020(Default Skin Version: 1)
hewiki: July 28 2020(Default Skin Version: 1)

PrefUpdateBugs:

The event.isDefault field is not recorded correctly for the VectorSkinVersion property. For the PrefUpdate instrumentation, the default value of the VectorSkinVersion preference is always null (not "1" or "2").See T258971.
PrefUpdate captures user preference modifications at registration - not isolated to just user initiated changes. https://phabricator.wikimedia.org/T260867#6403042. Caused by change merged on August 3rd https://phabricator.wikimedia.org/rEWMV30731c2c748a0ea34f798628fe7961b928572a23. This is likely why only the numbers for users with 0 edits are so high.

As a result, we will use the event.value field to determine the opt-out rate. event.value = 1 "legacy vector", event.value = 2 "modern vector". The below table what the default is for each wiki: https://docs.google.com/spreadsheets/d/1svlRDsYjOhjXkIV1a-DqLYTcLIFuYoc6pHHz8_VmQcc/edit#gid=625518477

Note: Data reflects total number of opt-outs so this will include any users that might have opt-out multiple times.

In [ ]:

#Opt-Out rate based on deployment dates on each wiki

query<- "SELECT
    wiki,
    event.bucketedUserEditCount AS user_edit_bucket,
    Count(*) AS n_opt
FROM event.prefupdate 
WHERE 
    event.property = 'VectorSkinVersion' AND
    year = 2020 and month >= 07 AND
    ((wiki = 'euwiki' AND event.savetimestamp >= '20200728' ) OR
     (wiki = 'frwikitionary' AND event.savetimestamp >= '20200728') OR
     (wiki = 'officewiki' AND event.savetimestamp >= '20200728') OR
     (wiki = 'ptwikiversity' AND event.savetimestamp >= '20200728' ) OR
     (wiki = 'fawiki' AND event.savetimestamp >= '20200728') OR
     (wiki = 'frwiki' AND event.savetimestamp >= '20200728') OR
     (wiki = 'hewiki' AND event.savetimestamp >='20200728' )) AND
    event.value = '"1"'
GROUP BY wiki, event.bucketedUserEditCount"

User User Preferences Table Instead to Obtain Total Current Users with Preference Set¶

References: https://github.com/wikimedia-research/Active-Editor-Skin-Statistics/blob/master/Re-run%20active%20editors%20skin%20statistics.ipynb

Pending resolution of PrefUpdate issues, we will use the user properties table to identify the total number of users with each vector skin preference set.

In [ ]:

query='''
SELECT 
  up_value AS skin, 
  COUNT(*) AS users
FROM user_properties
WHERE up_user in ({users})
AND up_property = "skin"
GROUP BY up_value
'''