This project investigates the level of equal opportunities in the field of professional developers, following the CRISP-DM process.
Survey results from Stackoverflow, which are freely accessible at https://insights.stackoverflow.com/survey, are used as the data basis.
*Note: The notebook does not claim to fully reflect the complexity of this question. It only elaborates on what the Stackoverflow survey results indicate.
Framing the Problem
The questions to be answered in this notebook are:
# Necessary libraries
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import numpy as np
Let's get a feel for the data set
# Read in the data - gather the data
df = pd.read_csv("2017survey_results_public.csv")
schema = pd.read_csv("2017survey_results_schema.csv")
schema.set_index("Column", inplace=True)
# Size of the dataset - assess the data
print("shape of the data:", df.shape)
print(f"Survey participants: {df.shape[0]}, Questions asked: {df.shape[1]}\n")
# Summary of the df
print(f"df.info:")
print({df.info()})
pd.set_option('display.max_columns', 200)
df.head()
shape of the data: (51392, 154) Survey participants: 51392, Questions asked: 154 df.info: <class 'pandas.core.frame.DataFrame'> RangeIndex: 51392 entries, 0 to 51391 Columns: 154 entries, Respondent to ExpectedSalary dtypes: float64(6), int64(1), object(147) memory usage: 60.4+ MB {None}
Respondent | Professional | ProgramHobby | Country | University | EmploymentStatus | FormalEducation | MajorUndergrad | HomeRemote | CompanySize | CompanyType | YearsProgram | YearsCodedJob | YearsCodedJobPast | DeveloperType | WebDeveloperType | MobileDeveloperType | NonDeveloperType | CareerSatisfaction | JobSatisfaction | ExCoderReturn | ExCoderNotForMe | ExCoderBalance | ExCoder10Years | ExCoderBelonged | ExCoderSkills | ExCoderWillNotCode | ExCoderActive | PronounceGIF | ProblemSolving | BuildingThings | LearningNewTech | BoringDetails | JobSecurity | DiversityImportant | AnnoyingUI | FriendsDevelopers | RightWrongWay | UnderstandComputers | SeriousWork | InvestTimeTools | WorkPayCare | KinshipDevelopers | ChallengeMyself | CompetePeers | ChangeWorld | JobSeekingStatus | HoursPerWeek | LastNewJob | AssessJobIndustry | AssessJobRole | AssessJobExp | AssessJobDept | AssessJobTech | AssessJobProjects | AssessJobCompensation | AssessJobOffice | AssessJobCommute | AssessJobRemote | AssessJobLeaders | AssessJobProfDevel | AssessJobDiversity | AssessJobProduct | AssessJobFinances | ImportantBenefits | ClickyKeys | JobProfile | ResumePrompted | LearnedHiring | ImportantHiringAlgorithms | ImportantHiringTechExp | ImportantHiringCommunication | ImportantHiringOpenSource | ImportantHiringPMExp | ImportantHiringCompanies | ImportantHiringTitles | ImportantHiringEducation | ImportantHiringRep | ImportantHiringGettingThingsDone | Currency | Overpaid | TabsSpaces | EducationImportant | EducationTypes | SelfTaughtTypes | TimeAfterBootcamp | CousinEducation | WorkStart | HaveWorkedLanguage | WantWorkLanguage | HaveWorkedFramework | WantWorkFramework | HaveWorkedDatabase | WantWorkDatabase | HaveWorkedPlatform | WantWorkPlatform | IDE | AuditoryEnvironment | Methodology | VersionControl | CheckInCode | ShipIt | OtherPeoplesCode | ProjectManagement | EnjoyDebugging | InTheZone | DifficultCommunication | CollaborateRemote | MetricAssess | EquipmentSatisfiedMonitors | EquipmentSatisfiedCPU | EquipmentSatisfiedRAM | EquipmentSatisfiedStorage | EquipmentSatisfiedRW | InfluenceInternet | InfluenceWorkstation | InfluenceHardware | InfluenceServers | InfluenceTechStack | InfluenceDeptTech | InfluenceVizTools | InfluenceDatabase | InfluenceCloud | InfluenceConsultants | InfluenceRecruitment | InfluenceCommunication | StackOverflowDescribes | StackOverflowSatisfaction | StackOverflowDevices | StackOverflowFoundAnswer | StackOverflowCopiedCode | StackOverflowJobListing | StackOverflowCompanyPage | StackOverflowJobSearch | StackOverflowNewQuestion | StackOverflowAnswer | StackOverflowMetaChat | StackOverflowAdsRelevant | StackOverflowAdsDistracting | StackOverflowModeration | StackOverflowCommunity | StackOverflowHelpful | StackOverflowBetter | StackOverflowWhatDo | StackOverflowMakeMoney | Gender | HighestEducationParents | Race | SurveyLong | QuestionsInteresting | QuestionsConfusing | InterestedAnswers | Salary | ExpectedSalary | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 1 | Student | Yes, both | United States | No | Not employed, and not looking for work | Secondary school | NaN | NaN | NaN | NaN | 2 to 3 years | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | With a soft "g," like "jiff" | Strongly agree | Strongly agree | Agree | Disagree | Strongly agree | Agree | Agree | Disagree | Somewhat agree | Disagree | Strongly agree | Strongly agree | Strongly disagree | Agree | Agree | Disagree | Agree | I'm not actively looking, but I am open to new... | 0.0 | Not applicable/ never | Very important | Very important | Important | Very important | Very important | Very important | Important | Very important | Very important | Very important | Very important | Very important | Somewhat important | Not very important | Somewhat important | Stock options; Vacation/days off; Remote options | Yes | Other | NaN | NaN | Important | Important | Important | Somewhat important | Important | Not very important | Not very important | Not at all important | Somewhat important | Very important | NaN | NaN | Tabs | NaN | Online course; Open source contributions | NaN | NaN | NaN | 6:00 AM | Swift | Swift | NaN | NaN | NaN | NaN | iOS | iOS | Atom; Xcode | Turn on some music | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | Somewhat satisfied | Not very satisfied | Not at all satisfied | Very satisfied | Satisfied | Not very satisfied | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | I have created a CV or Developer Story on Stac... | 9.0 | Desktop; iOS app | At least once each week | Haven't done at all | Once or twice | Haven't done at all | Haven't done at all | Several times | Several times | Once or twice | Somewhat agree | Strongly disagree | Strongly disagree | Strongly agree | Agree | Strongly agree | Strongly agree | Strongly disagree | Male | High school | White or of European descent | Strongly disagree | Strongly agree | Disagree | Strongly agree | NaN | NaN |
1 | 2 | Student | Yes, both | United Kingdom | Yes, full-time | Employed part-time | Some college/university study without earning ... | Computer science or software engineering | More than half, but not all, the time | 20 to 99 employees | Privately-held limited company, not in startup... | 9 to 10 years | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | With a hard "g," like "gift" | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | No | Other | NaN | Some other way | Important | Important | Important | Important | Somewhat important | Somewhat important | Not very important | Somewhat important | Not very important | Very important | British pounds sterling (£) | NaN | Spaces | NaN | Online course; Self-taught; Hackathon; Open so... | Official documentation; Stack Overflow Q&A; Other | NaN | NaN | 10:00 AM | JavaScript; Python; Ruby; SQL | Java; Python; Ruby; SQL | .NET Core | .NET Core | MySQL; SQLite | MySQL; SQLite | Amazon Web Services (AWS) | Linux Desktop; Raspberry Pi; Amazon Web Servic... | Atom; Notepad++; Vim; PyCharm; RubyMine; Visua... | Put on some ambient sounds (e.g. whale songs, ... | NaN | Git | Multiple times a day | Agree | Disagree | Strongly disagree | Agree | Somewhat agree | Disagree | Strongly disagree | Customer satisfaction; On time/in budget; Peer... | Not very satisfied | Satisfied | Satisfied | Satisfied | Somewhat satisfied | Satisfied | No influence at all | No influence at all | No influence at all | No influence at all | No influence at all | No influence at all | No influence at all | No influence at all | No influence at all | No influence at all | No influence at all | I have created a CV or Developer Story on Stac... | 8.0 | Desktop; iOS browser; iOS app; Android browser... | Several times | Several times | Once or twice | Once or twice | Once or twice | Haven't done at all | Several times | At least once each week | Disagree | Strongly disagree | Strongly disagree | Strongly agree | Agree | Strongly agree | Strongly agree | Strongly disagree | Male | A master's degree | White or of European descent | Somewhat agree | Somewhat agree | Disagree | Strongly agree | NaN | 37500.0 |
2 | 3 | Professional developer | Yes, both | United Kingdom | No | Employed full-time | Bachelor's degree | Computer science or software engineering | Less than half the time, but at least one day ... | 10,000 or more employees | Publicly-traded corporation | 20 or more years | 20 or more years | NaN | Other | NaN | NaN | NaN | 8.0 | 9.0 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | With a hard "g," like "gift" | Strongly agree | Strongly agree | Strongly agree | Somewhat agree | Agree | Strongly agree | Agree | Somewhat agree | Disagree | Disagree | Agree | Somewhat agree | Disagree | Somewhat agree | Agree | Disagree | Agree | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | Yes | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | British pounds sterling (£) | Neither underpaid nor overpaid | Spaces | Not very important | Self-taught; Coding competition; Hackathon; Op... | Official documentation; Trade book; Textbook; ... | NaN | NaN | 9:00 AM | Java; PHP; Python | C; Python; Rust | NaN | NaN | MySQL | NaN | NaN | NaN | Sublime Text; Vim; IntelliJ | Turn on some music | Agile; Lean; Scrum; Extreme; Pair; Kanban | Mercurial | Multiple times a day | Agree | Disagree | Disagree | Agree | Agree | Disagree | Somewhat agree | Customer satisfaction; Benchmarked product per... | Very satisfied | Somewhat satisfied | Satisfied | Satisfied | Somewhat satisfied | Very satisfied | A lot of influence | Some influence | Some influence | Some influence | A lot of influence | Some influence | Some influence | Some influence | Some influence | Some influence | Some influence | I have created a CV or Developer Story on Stac... | 8.0 | Desktop; iOS browser; iOS app | Once or twice | Haven't done at all | Haven't done at all | Haven't done at all | Haven't done at all | Haven't done at all | At least once each day | At least once each day | Disagree | Disagree | Strongly disagree | Strongly agree | Agree | Agree | Agree | Disagree | Male | A professional degree | White or of European descent | Somewhat agree | Agree | Disagree | Agree | 113750.0 | NaN |
3 | 4 | Professional non-developer who sometimes write... | Yes, both | United States | No | Employed full-time | Doctoral degree | A non-computer-focused engineering discipline | Less than half the time, but at least one day ... | 10,000 or more employees | Non-profit/non-governmental organization or pr... | 14 to 15 years | 9 to 10 years | NaN | NaN | NaN | NaN | Data scientist | 6.0 | 3.0 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | With a soft "g," like "jiff" | Strongly agree | Strongly agree | Strongly agree | Disagree | Somewhat agree | Agree | Agree | Agree | Somewhat agree | Strongly disagree | Strongly agree | Agree | Disagree | Strongly agree | Strongly agree | Somewhat agree | Agree | I am actively looking for a job | 5.0 | Between 2 and 4 years ago | Somewhat important | Somewhat important | Somewhat important | Important | Important | Very important | Important | Very important | Important | Somewhat important | Not very important | Very important | Important | Very important | Very important | Stock options; Annual bonus; Health benefits; ... | Yes | LinkedIn; Other | NaN | A friend, family member, or former colleague t... | Somewhat important | Somewhat important | Very important | Very important | Somewhat important | Somewhat important | Not very important | Not very important | Important | Very important | NaN | NaN | Spaces | NaN | NaN | NaN | NaN | NaN | 9:00 AM | Matlab; Python; R; SQL | Matlab; Python; R; SQL | React | Hadoop; Node.js; React | MongoDB; Redis; SQL Server; MySQL; SQLite | MongoDB; Redis; SQL Server; MySQL; SQLite | Windows Desktop; Linux Desktop; Mac OS; Amazon... | Windows Desktop; Linux Desktop; Mac OS; Amazon... | Notepad++; Sublime Text; TextMate; Vim; IPytho... | Turn on some music | Agile | Git | Multiple times a day | Somewhat agree | Agree | Somewhat agree | Somewhat agree | Strongly agree | Disagree | Somewhat agree | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | I have created a CV or Developer Story on Stac... | 10.0 | Desktop; iOS browser; iOS app | At least once each week | Several times | At least once each week | Several times | At least once each week | Several times | At least once each day | At least once each day | Agree | Strongly disagree | Strongly disagree | Strongly agree | Strongly agree | Agree | Strongly agree | Disagree | Male | A doctoral degree | White or of European descent | Agree | Agree | Somewhat agree | Strongly agree | NaN | NaN |
4 | 5 | Professional developer | Yes, I program as a hobby | Switzerland | No | Employed full-time | Master's degree | Computer science or software engineering | Never | 10 to 19 employees | Privately-held limited company, not in startup... | 20 or more years | 10 to 11 years | NaN | Mobile developer; Graphics programming; Deskto... | NaN | NaN | NaN | 6.0 | 8.0 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | With a soft "g," like "jiff" | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | Satisfied | Satisfied | Satisfied | Satisfied | Satisfied | Satisfied | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
The most interesting columns in regard to the questions are:
Q1: Is there equal opportunity in the programming profession?
Note: The country is probably strongly related to ethnicity and needs to be considered in the study.
Q2: How does your social environment influence your chances of being successful as a developer?
Q3: How important is an open mind and tolerance to succeed as a programmer?
To evaluate whether a developer is successful, the following parameters are evaluated:
Of course, these evaluation criteria must take into account whether the survey participant is a professional programmer.
# Columns of interest
INITIAL_CONDITIONS = ["Race", "HighestEducationParents", "Gender"]
PART_INFLUENCEABLE_CONDITIONS = ["Country"]
SOCIAL_ENVIRONMENT = ["HighestEducationParents", "FriendsDevelopers"]
MINDSET = ["RightWrongWay", "DiversityImportant"]
EVALUATION_PARAMETERS = ["JobSatisfaction", "CareerSatisfaction", "Salary"]
ADDITIONAL_PARAMETERS = ["Professional"]
columns_of_interest = list(set([item for list in [INITIAL_CONDITIONS, PART_INFLUENCEABLE_CONDITIONS, SOCIAL_ENVIRONMENT,
MINDSET, ADDITIONAL_PARAMETERS, EVALUATION_PARAMETERS] for item in list]))
# Let's take a look at the questions behind the columns
for index, row in schema.loc[columns_of_interest].iterrows():
question = schema.loc[index, "Question"]
print(f"{index}: {question}")
Gender: Which of the following do you currently identify as? Salary: What is your current annual base salary, before taxes, and excluding bonuses, grants, or other compensation? Race: Which of the following do you identify as? RightWrongWay: There's a right and a wrong way to do everything FriendsDevelopers: Most of my friends are developers, engineers, or scientists HighestEducationParents: What is the highest level of education received by either of your parents? DiversityImportant: Diversity in the workplace is important CareerSatisfaction: Career satisfaction rating Professional: Which of the following best describes you? Country: In which country do you currently live? JobSatisfaction: Job satisfaction rating
# Build a df containing only the columns of interest
df_filtered = df[columns_of_interest].copy()
df_filtered.head()
Gender | Salary | Race | RightWrongWay | FriendsDevelopers | HighestEducationParents | DiversityImportant | CareerSatisfaction | Professional | Country | JobSatisfaction | |
---|---|---|---|---|---|---|---|---|---|---|---|
0 | Male | NaN | White or of European descent | Somewhat agree | Disagree | High school | Agree | NaN | Student | United States | NaN |
1 | Male | NaN | White or of European descent | NaN | NaN | A master's degree | NaN | NaN | Student | United Kingdom | NaN |
2 | Male | 113750.0 | White or of European descent | Disagree | Somewhat agree | A professional degree | Strongly agree | 8.0 | Professional developer | United Kingdom | 9.0 |
3 | Male | NaN | White or of European descent | Somewhat agree | Agree | A doctoral degree | Agree | 6.0 | Professional non-developer who sometimes write... | United States | 3.0 |
4 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | 6.0 | Professional developer | Switzerland | 8.0 |
# Datatypes in the dataset
print(df_filtered.dtypes)
cat_cols = list(df_filtered.columns[df_filtered.dtypes=="object"])
print(f"categorical cols: {cat_cols}")
Gender object Salary float64 Race object RightWrongWay object FriendsDevelopers object HighestEducationParents object DiversityImportant object CareerSatisfaction float64 Professional object Country object JobSatisfaction float64 dtype: object categorical cols: ['Gender', 'Race', 'RightWrongWay', 'FriendsDevelopers', 'HighestEducationParents', 'DiversityImportant', 'Professional', 'Country']
# Ditribution of the quantitative parameters
df_filtered.describe()
Salary | CareerSatisfaction | JobSatisfaction | |
---|---|---|---|
count | 12891.000000 | 42695.000000 | 40376.000000 |
mean | 56298.480641 | 7.300574 | 6.957078 |
std | 39880.905277 | 1.955444 | 2.167652 |
min | 0.000000 | 0.000000 | 0.000000 |
25% | 26440.371839 | 6.000000 | 6.000000 |
50% | 50000.000000 | 8.000000 | 7.000000 |
75% | 80000.000000 | 9.000000 | 8.000000 |
max | 197000.000000 | 10.000000 | 10.000000 |
# Proportions of missing evaluation parameters
prop_missing_salary = df_filtered["Salary"].isnull().mean()
prop_missing_jobsf = df_filtered["JobSatisfaction"].isnull().mean()
prop_missing_careersf = df_filtered["CareerSatisfaction"].isnull().mean()
print("Missing evaluation parameters")
print(f"Salary: {round(prop_missing_salary*100, 1)} %")
print(f"JobSatisfaction: {round(prop_missing_jobsf*100, 1)} %")
print(f"CareerSatisfaction: {round(prop_missing_careersf*100, 1)} %")
# Missing evaluation parameters combined
print("\nData left after removing every row with missing evaluation parameters:")
# We can safely drop rows where all parameters are nan
df_filtered.dropna(subset=cat_cols, how="all", inplace=True)
# Build a df containing only rows with the needed evaluation criteria - clean the data
df_eval = df_filtered.dropna(subset=EVALUATION_PARAMETERS, axis=0).copy()
# Due to the questions above we are only interested in data from professional developers
data_left = round((df_eval[df_eval["Professional"]=="Professional developer"].shape[0]/
df_filtered[df_filtered["Professional"]=="Professional developer"].shape[0])*100, 1)
print(f"{data_left} %, -> {df_eval.shape[0]}")
print(f"\nshape of the df: {df_eval.shape}")
Missing evaluation parameters Salary: 74.9 % JobSatisfaction: 21.4 % CareerSatisfaction: 16.9 % Data left after removing every row with missing evaluation parameters: 35.6 %, -> 12847 shape of the df: (12847, 11)
In order not to lose that much data, we could impute values in this step. However, this reduces the significance of our data, since we reduce the variability.
Because our df has a lot of rows in proportion to the columns, we continue without imputation for now.
In the df df_eval, due to the filtering with the salary, there should be only survey participants whose professional status is professional developer.
This can be easily checked using the "Professional" column.
print(df_eval["Professional"].value_counts())
Professional developer 12847 Name: Professional, dtype: int64
The df is now filtered so we can take a look at the distribution of the selected parameters in the data set.
# analyze the data
# Plot only results where there are at least 50 survey participants
ethnicity_counts = df_eval["Race"].value_counts()[df_eval["Race"].value_counts() >=50]
# Plot only results where there are at least 400 survey participants
country_counts = df_eval["Country"].value_counts()[df_eval["Country"].value_counts() >=400]
# Plot only result where there are at least 10 survey participants
gender_counts = df_eval["Gender"].value_counts()[df_eval["Gender"].value_counts()>=10]
# Default plot for data overview
def do_standard_plot(df, title, plot_type, size=[6,6], use_y_grid=False):
"""[Takes a df containing a distribution and creates a plot from it]
Args:
df ([pandas.DataFrame]): [df containing a distribution]
title ([str]): [title of the plot]
size (list, optional): [size of the figure]. Defaults to [8,8].
size ([bool, optional]): [True for y-grid]. Defaults to False.
"""
if use_y_grid:
_, ax = plt.subplots()
df.plot(kind=plot_type, figsize=size);
plt.title(title)
if use_y_grid:
plt.grid(axis="y")
ax.set_axisbelow(True)
plt.ylabel("Survey participants")
# Ethnicity distribution
print(ethnicity_counts)
do_standard_plot(ethnicity_counts, "Ethnicity distribution", "pie")
White or of European descent 8685 South Asian 682 Hispanic or Latino/Latina 419 East Asian 308 Middle Eastern 212 Hispanic or Latino/Latina; White or of European descent 155 Black or of African descent 155 I prefer not to say 150 I don’t know 131 Middle Eastern; White or of European descent 66 Name: Race, dtype: int64
It is clearly evident that one ethnicity in the chart is clearly overweighted.
Since only ethnicities to which at least 50 survey participants belong are shown here,
it should still be possible to make a reasonably statistically reliable statement with these.
We can neglect the answers 'I prefer not to say' and 'I don't know'.
So we focus on the following groups:
['White or of European descent', 'South Asian',
'Hispanic or Latino/Latina', 'East Asian', 'Middle Eastern',
'Hispanic or Latino/Latina; White or of European descent',
'Black or of African descent', 'Middle Eastern; White or of European descent']
ethnicities_in_focus = ['White or of European descent', 'South Asian',
'Hispanic or Latino/Latina', 'East Asian', 'Middle Eastern',
'Hispanic or Latino/Latina; White or of European descent',
'Black or of African descent', 'Middle Eastern; White or of European descent']
# Country distribution
do_standard_plot(country_counts, "Country distribution", "bar", [8,6], True)
In terms of countries, there is a bias towards the U.S., although the data is much more balanced on this point.
The countries that we want to consider in the evaluation are:
['United States', 'United Kingdom', 'Germany', 'India', 'Canada', 'France']
countries_in_focus = list(country_counts.index)
# Gender distribution
do_standard_plot(gender_counts, "Gender distribution", "bar", [8,6], True)
If you look at the gender distribution, you can see, as you would expect, that the profession of developer is strongly dominated by males.
Since we can't make reasonable statements, with too few survey participants for a gender, we will divide this property into 3 groups.
The first two groups are female and male. All remaining answers are summarized under the term Other.
# Disable chained assignments
pd.options.mode.chained_assignment = None # Source: https://stackoverflow.com/questions/49728421/pandas-dataframe-settingwithcopywarning-a-value-is-trying-to-be-set-on-a-copy
# Reworked gender distribution
df_eval["Gender"][[element not in ["Male", "Female", np.nan] for element in df_eval["Gender"]]] = "Other"
print(df_eval["Gender"].value_counts())
do_standard_plot(df_eval["Gender"].value_counts(), "Gender distribution", "pie")
Male 10632 Female 816 Other 222 Name: Gender, dtype: int64
#Missing values left
df_eval.isnull().sum()
Gender 1177 Salary 0 Race 1605 RightWrongWay 4017 FriendsDevelopers 4000 HighestEducationParents 1182 DiversityImportant 4014 CareerSatisfaction 0 Professional 0 Country 0 JobSatisfaction 0 dtype: int64
This should be enough for a first overview.
So let's take a look at question number 1:
Q1 Is there equal opportunity in the programming profession?
Note: The OverallSatisfaction is introduced as the sum of the job and the career satisfaction
def add_overall_sf(df):
"""[Adds the OverallSatifsction column to a df as the sum of job and career satisfaction]
Args:
df ([pandas.DataFrame]): [Pandas df that contains the columns "JobSatisfaction" and "CareerSatisfaction"]
"""
df["OverallSatisfaction"] = df["JobSatisfaction"] + df["CareerSatisfaction"]
# Evaluation parameter by ethnicity
ethnicity_satifaction = df_eval.groupby("Race").mean().loc[ethnicities_in_focus].sort_values("Salary", ascending=False)
add_overall_sf(ethnicity_satifaction)
ethnicity_satifaction
Salary | CareerSatisfaction | JobSatisfaction | OverallSatisfaction | |
---|---|---|---|---|
Race | ||||
Middle Eastern; White or of European descent | 69035.926739 | 7.575758 | 7.090909 | 14.666667 |
White or of European descent | 61582.525312 | 7.559125 | 7.057110 | 14.616235 |
Hispanic or Latino/Latina; White or of European descent | 60179.626160 | 7.741935 | 7.219355 | 14.961290 |
East Asian | 52027.612807 | 7.113636 | 6.779221 | 13.892857 |
Black or of African descent | 50068.650250 | 7.212903 | 6.187097 | 13.400000 |
Hispanic or Latino/Latina | 41648.064699 | 8.069212 | 7.248210 | 15.317422 |
Middle Eastern | 37802.317456 | 7.537736 | 6.849057 | 14.386792 |
South Asian | 30528.177506 | 7.076246 | 6.373900 | 13.450147 |
The table shows that a low salary does not necessarily mean low satisfaction.
The salary varies so significantly that in the next step we have to take the countries into account.
It is important to note that different countries have different salary levels and that there is an uneven distribution of ethnic groups across countries.
def plot_multiple_bar_plot(data, x="Country", y="Salary", hue="Race", title="Salary by country and ethnicity",
palette=["tab:blue","tab:green","tab:red","tab:cyan","tab:pink","tab:olive","tab:gray","tab:orange"]):
"""[Creates a plot in which each x-tick is divided again using several bars in different colors]
Args:
data ([pandas.DataFrame]): [A df containing the x-ticks, the categories for the hues, and the corresponding y-values]
x (str, optional): [Name of the column with the values for the x-ticks]. Defaults to "Country".
y (str, optional): [Name of the column with the values for the y-axis]. Defaults to "Salary".
hue (str, optional): [Name of the column with the categories that are divided into different colored bars.]. Defaults to "Race".
title (str, optional): [Title of the plot]. Defaults to "Salary by country and ethnicity".
palette (list, optional): [seaborn.color_palette or list of matplotlib colors.]. Defaults to ["tab:blue","tab:green","tab:red",
"tab:cyan","tab:pink","tab:olive",
"tab:gray","tab:orange"].
"""
_, ax = plt.subplots()
sns.barplot(x=x, y=y, hue=hue, data=data, palette=palette)
plt.gcf().set_size_inches(14, 7)
plt.legend(loc="upper center")
plt.title(title)
plt.grid(axis="y")
ax.set_axisbelow(True)
# Plot salary by country and ethnicity
ethnicity_sf_countries = df_eval.groupby(["Race", "Country"], as_index=False)[EVALUATION_PARAMETERS].mean()
# Select only ethnicities and countries in focus
ethnicity_sf_countries_if = ethnicity_sf_countries[(ethnicity_sf_countries["Race"].isin(ethnicities_in_focus)) &
(ethnicity_sf_countries["Country"].isin(countries_in_focus))]
# visualize results
plot_multiple_bar_plot(ethnicity_sf_countries_if)
We can quickly see that there is not enough data available for India, so the plot is made again without India.
# Select only ethnicities and countries in focus - without India
countries_in_focus_no_india = countries_in_focus.copy()
countries_in_focus_no_india.remove("India")
ethnicity_sf_countries_if_mod = ethnicity_sf_countries[(ethnicity_sf_countries["Race"].isin(ethnicities_in_focus)) &
(ethnicity_sf_countries["Country"].isin(countries_in_focus_no_india))]
plot_multiple_bar_plot(ethnicity_sf_countries_if_mod)
We can clearly see that some ethnicities are disadvantaged. The "Black or of African descent" group in particular has the lowest or second lowest salary in all countries considered.
# Standard diagram for the visualization of the distribution of the evaluation parameters
def plot_eval_param_dist(df, title, parameter, x_lim=[13, 16],
palette=["tab:blue","tab:green","tab:red","tab:cyan","tab:pink","tab:olive",
"tab:gray","tab:orange","tab:purple","tab:brown"]):
"""[Plots the distribution of a parameter in a bar chart]
Args:
df ([pandas.DataFrame]): [A df containing a distribution over "parameter"]
title ([str]): [Title of the plot]
parameter ([str]): [Evaluation parameter to plot]
x_lim (list, optional): [x-Axis limit of the plot]. Defaults to [12, 16].
palette (list, optional): [seaborn.color_palette or list of matplotlib colors.]. Defaults to ["tab:blue","tab:green","tab:red",
"tab:cyan","tab:pink","tab:olive",
"tab:gray","tab:orange","tab:purple",
"tab:brown"].
"""
df.sort_values(parameter, ascending=False, inplace=True)
_, ax = plt.subplots()
sns.barplot(x=df[parameter], y=df.index, palette=palette)
plt.title(title)
plt.xlim(x_lim)
plt.grid(axis="x")
ax.set_axisbelow(True)
# Plot overall satisfaction by ethnicity
plot_eval_param_dist(ethnicity_satifaction, "Overall satisfaction by ethnicity", "OverallSatisfaction",
palette=["tab:red", "tab:cyan", "tab:olive", "tab:orange", "tab:pink", "tab:green", "tab:gray", "tab:blue"])
We can see again that the group "Black or of African descent" is clearly disadvantaged. People from Asia also only achieve a low level of overall satisfaction.
In order to see which parameters affect general satisfaction and to what extent,
the ranges between maximum and minimum satisfaction are stored in a dict.
range_oa_sf = {}
def save_range_in_oa_sf(df, parameter):
"""[The function saves the range between maximum and minimum OverallSatisfaction depending on the category/parameter in a dict]
Args:
df ([pandas.DataFrame]): [A df containing a distribution over the OverallSatisfaction]
parameter ([str]): [The category to which the distribution belongs]
"""
global range_oa_sf
range_oa_sf[parameter] = round(df["OverallSatisfaction"].max()-df["OverallSatisfaction"].min(),2)
# Safe range
save_range_in_oa_sf(ethnicity_satifaction, "Ethnicity")
# Available data HighestEducationParents
available_data_hep = len(df_eval["HighestEducationParents"]) - sum(df_eval["HighestEducationParents"].isnull())
print(f"Available data HighestEducationParents: {available_data_hep}")
# Evaluation parameter by HighestEducationParents
pe_satifaction = df_eval.groupby("HighestEducationParents").mean().sort_values("Salary", ascending=False)
add_overall_sf(pe_satifaction)
pe_satifaction
Available data HighestEducationParents: 11665
Salary | CareerSatisfaction | JobSatisfaction | OverallSatisfaction | |
---|---|---|---|---|
HighestEducationParents | ||||
A doctoral degree | 68414.472092 | 7.553226 | 7.185484 | 14.738710 |
A professional degree | 64326.306664 | 7.480000 | 6.998182 | 14.478182 |
A master's degree | 60409.909827 | 7.522377 | 7.033951 | 14.556327 |
Some college/university study, no bachelor's degree | 59926.831012 | 7.663360 | 6.996900 | 14.660260 |
A bachelor's degree | 57102.290990 | 7.494773 | 6.945182 | 14.439955 |
High school | 50924.049281 | 7.463867 | 6.899902 | 14.363770 |
I don't know/not sure | 49553.558412 | 7.233533 | 6.898204 | 14.131737 |
I prefer not to answer | 45081.722501 | 7.523077 | 6.861538 | 14.384615 |
Primary/elementary school | 39351.855125 | 7.246973 | 6.733656 | 13.980630 |
No education | 38277.773591 | 7.551724 | 6.689655 | 14.241379 |
# Plot salary by HighestEducationParents
plot_eval_param_dist(pe_satifaction, "Salary by HighestEducationParents", "Salary", x_lim=[37500,70000])
The chart clearly shows that the salary earned is heavily dependent on the education of the parents.
# Plot overall satisfaction by HighestEducationParents
plot_eval_param_dist(pe_satifaction, "Overall satisfaction by HighestEducationParents", "OverallSatisfaction", x_lim=[13.5,15],
palette=["tab:blue","tab:cyan","tab:red","tab:green","tab:pink","tab:orange","tab:olive","tab:brown","tab:gray","tab:purple"])
# Save satisfaction range for HighestEducationParents
save_range_in_oa_sf(pe_satifaction, "HighestEducationParents")
As far as overall satisfaction is concerned, the graph is a little less clear. Nevertheless, we can clearly see that a high level
of overall satisfaction is more likely to be achieved by people whose parents have achieved a high academic degree. The parents of
the three most satisfied groups on average all have an academic degree.
# Evaluation parameter by Gender
gender_satifaction = df_eval.groupby("Gender").mean().sort_values("Salary", ascending=False)
add_overall_sf(gender_satifaction)
gender_satifaction
Salary | CareerSatisfaction | JobSatisfaction | OverallSatisfaction | |
---|---|---|---|---|
Gender | ||||
Other | 61633.921918 | 7.162162 | 6.576577 | 13.738739 |
Female | 57969.628276 | 7.409314 | 6.985294 | 14.394608 |
Male | 56924.546269 | 7.525677 | 6.977897 | 14.503574 |
# Plot salary by Gender
plot_eval_param_dist(gender_satifaction, "Salary by gender", "Salary", x_lim=[55000,63000],
palette=["tab:red","tab:blue","tab:green"])
Contrary to what one might expect, men in the developer profession earn the lowest salaries.
One reason for this could be that the proportion of men in lower-income countries is significantly higher.
In order to check this, the salary distribution of the different genders across the countries must be considered.
# Plot salary by country and Gender
gender_sf_countries = df_eval.groupby(["Gender", "Country"], as_index=False)[EVALUATION_PARAMETERS].mean()
# Select only countries in focus
gender_sf_countries_if = gender_sf_countries[gender_sf_countries["Country"].isin(countries_in_focus)]
plot_multiple_bar_plot(gender_sf_countries_if, hue="Gender", title="Salary by country and gender")
When comparing countries, we can see that women are clearly disadvantaged.
For people who assign themselves to a gender other than male or female,
whether they belong to the best or the lowest paid group depends heavily on the country.
# Plot overall satisfaction by Gender
plot_eval_param_dist(gender_satifaction, "Overall satisfaction by gender", "OverallSatisfaction", x_lim=[13.5,14.8],
palette=["tab:green","tab:blue","tab:red"])
# Save satisfaction range for Gender
save_range_in_oa_sf(gender_satifaction, "Gender")
When it comes to general job and career satisfaction, the graph is reversed. Here the men are the happiest.
# range overall satifaction for the initial conditions
range_oa_sf
{'Ethnicity': 1.92, 'HighestEducationParents': 0.76, 'Gender': 0.76}
Overall, we can state that ethnicity has the greatest impact on the level of satisfaction achieved.
As the graphics show, all initial conditions have an impact on salary and satisfaction.
Question Q1 can be clearly answered in the negative. There is no equal opportunity in the job profile of professional developers.
Let's continue with question 2:
Q2 How does your social environment influence your chances of being successful as a developer?
We have already seen clearly that the parents education not only has an extreme impact on salary, but also on job satisfaction.
Now we have to check how the circle of friends affect the success. Here, however, it must be taken into account that a correlation does not necessarily mean a causality.
If you are successful as a programmer, you have a lot to do with other developers and the probability is high that you will find new friends among them.
In addition, many friendships are made during an academic career and in our society people with high degrees tend to earn higher incomes.
# Available data FriendsDevelopers
available_data_fd = len(df_eval["FriendsDevelopers"]) - sum(df_eval["FriendsDevelopers"].isnull())
print(f"Available data FriendsDevelopers: {available_data_fd}")
# Evaluation parameter by FriendsDevelopers
friends_dev_satifaction = df_eval.groupby("FriendsDevelopers").mean().sort_values("Salary", ascending=False)
add_overall_sf(friends_dev_satifaction)
friends_dev_satifaction
Available data FriendsDevelopers: 8847
Salary | CareerSatisfaction | JobSatisfaction | OverallSatisfaction | |
---|---|---|---|---|
FriendsDevelopers | ||||
Disagree | 60290.451849 | 7.556188 | 7.024182 | 14.580370 |
Somewhat agree | 56516.277812 | 7.492390 | 6.952104 | 14.444494 |
Strongly disagree | 55530.701540 | 7.436782 | 6.959770 | 14.396552 |
Agree | 53857.007633 | 7.447246 | 6.874416 | 14.321662 |
Strongly agree | 52200.128657 | 7.539924 | 6.978454 | 14.518378 |
The table shows that the general level of satisfaction is very balanced.
What is more interesting for a graphical visualisation is the correlation between social circle and salary.
# Plot salary in relation to friends from the same professional field
plot_eval_param_dist(friends_dev_satifaction, "Salary / Are your friends developers?", "Salary", x_lim=[50000,62000])
Contrary to my original assumption, we can see from the graph that people whose circle of friends includes few developers have the highest salary.
The people who mainly surround themselves with developers earn even the lowest salary.
With regard to Q2, it can be said that the social environment has a significant impact on success as a programmer.
The parents in particular have a major impact. The higher the educational level of the parents,
the more likely it is to achieve a high salary and the greater the chances of being satisfied in one's job/career.
As far as the circle of friends is concerned, it is advantageous in terms of salary not only to surround oneself
with friends of the same professional orientation. However, this parameter is irrelevant for the level of satisfaction achieved.
Finally, we continue with question 3:
Q3 How important is an open mind and tolerance to succeed as a programmer?
# Available data RightWrongWay
available_data_rw = len(df_eval["RightWrongWay"]) - sum(df_eval["RightWrongWay"].isnull())
print(f"Available data RightWrongWay: {available_data_rw}")
# Evaluation parameter by RightWrongWay
right_wrong_satifaction = df_eval.groupby("RightWrongWay").mean().sort_values("Salary", ascending=False)
add_overall_sf(right_wrong_satifaction)
right_wrong_satifaction
Available data RightWrongWay: 8830
Salary | CareerSatisfaction | JobSatisfaction | OverallSatisfaction | |
---|---|---|---|---|
RightWrongWay | ||||
Strongly disagree | 64572.123211 | 7.548686 | 7.134467 | 14.683153 |
Disagree | 63007.050758 | 7.469945 | 7.028583 | 14.498529 |
Somewhat agree | 60474.853731 | 7.529251 | 6.974341 | 14.503592 |
Agree | 49374.558330 | 7.523661 | 6.925117 | 14.448778 |
Strongly agree | 38073.552483 | 7.393528 | 6.698330 | 14.091858 |
As with the circle of friends, this parameter does not have a high impact on satisfaction.
It remains to be said, however, that the people who believe that there is a right and wrong way for everything are the most unsatisfied.
In terms of salary, the more open a developer is to different solutions, the higher the salary he achieves.
# Plot the relation of salary to whether people believe there is a right way and a wrong way to do everything
plot_eval_param_dist(right_wrong_satifaction, "Salary / There's a right and a wrong way to do everything", "Salary", x_lim=[35000,67000],
palette=["tab:pink","tab:green","tab:red","tab:blue","tab:cyan"])
Since the difference in salary is so significant, we should look at this again in a country comparison.
# Plot salary by country and RightWrongWay
right_wrong_sf_countries = df_eval.groupby(["RightWrongWay", "Country"], as_index=False)[EVALUATION_PARAMETERS].mean()
# Select only countries in focus
right_wrong_sf_countries_if = right_wrong_sf_countries[right_wrong_sf_countries["Country"].isin(countries_in_focus)]
plot_multiple_bar_plot(right_wrong_sf_countries_if, hue="RightWrongWay", title="Salary / There's a right and a wrong way to do everything")
We can see that in all the countries considered, people who believe that there is only one right way to solve a problem earn less than those who strongly reject this belief.
# Available data DiversityImportant
available_data_di = len(df_eval["DiversityImportant"]) - sum(df_eval["DiversityImportant"].isnull())
print(f"Available data DiversityImportant: {available_data_di}")
# Evaluation parameter by DiversityImportant
diversity_imp_satifaction = df_eval.groupby("DiversityImportant").mean().sort_values("Salary", ascending=False)
add_overall_sf(diversity_imp_satifaction)
diversity_imp_satifaction
Available data DiversityImportant: 8833
Salary | CareerSatisfaction | JobSatisfaction | OverallSatisfaction | |
---|---|---|---|---|
DiversityImportant | ||||
Strongly disagree | 63195.713791 | 7.260736 | 6.687117 | 13.947853 |
Strongly agree | 61762.894720 | 7.619968 | 7.078532 | 14.698500 |
Disagree | 57684.292744 | 7.358865 | 6.809929 | 14.168794 |
Agree | 53805.346354 | 7.487527 | 6.948568 | 14.436095 |
Somewhat agree | 53270.827345 | 7.455715 | 6.919842 | 14.375557 |
# Plot the relation of overall satisfaction to whether people believe that Diversity in the workplace is important in percent
diversity_imp_satifaction.sort_values("OverallSatisfaction", ascending=False, inplace=True)
_, ax = plt.subplots()
sns.barplot(x=100*(diversity_imp_satifaction["OverallSatisfaction"]/max(diversity_imp_satifaction["OverallSatisfaction"])), y=diversity_imp_satifaction.index)
plt.title("Overall satisfaction / Diversity in the workplace is important")
plt.xlabel("OverallSatisfaction[%]")
plt.xlim([90, 100])
plt.grid(axis="x")
ax.set_axisbelow(True)
On the question of whether diversity is important in the workplace, it appears that people with a clear opinion on this earn the most.
The group that is strongly opposed to diversity has an average income that is about €1500 higher, but is also more than 5% less satisfied than those for whom diversity is important.
To answer Q3, it is important to note that it is extremely important to be open to different solution approaches when it comes to maximizing salary.
In addition, a clear opinion can help to maximize the salary. It should be taken into account that open people are happier.
For the condensed presentation of the results, check out this medium post: https://medium.com/@florianpue/an-inconvenient-truth-about-social-justice-in-programming-f78fb17c614b