Guided Project: Clean And Analyze Employee Exit Surveys

In this guided project, we'll work with exit surveys from employees of the Department of Education, Training and Employment (DETE) and the Technical and Further Education (TAFE) institute in Queensland, Australia. You can find the TAFE exit survey here and the survey for the DETE here. We've made some slight modifications to these datasets to make them easier to work with, including changing the encoding to UTF-8 (the original ones are encoded using cp1252.)

In this project, we'll play the role of data analyst and pretend our stakeholders want to know the following:

Are employees who only worked for the institutes for a short period of time resigning due to some kind of dissatisfaction? What about employees who have been there longer?

Are younger employees resigning due to some kind of dissatisfaction? What about older employees?

A data dictionary wasn't provided with the dataset. For this project, we'll use our general knowledge to define the columns.

Below is a preview of a couple columns we'll work with from the dete_survey.csv:

  • ID: An id used to identify the participant of the survey
  • SeparationType: The reason why the person's employment ended
  • Cease Date: The year or month the person's employment ended
  • DETE Start Date: The year the person began employment with the DETE

Below is a preview of a couple columns we'll work with from the tafe_survey.csv:

  • Record ID: An id used to identify the participant of the survey
  • Reason for ceasing employment: The reason why the person's employment ended
  • LengthofServiceOverall. Overall Length of Service at Institute (in years): The length of the person's employment (in years)

Importing the libraries we will be working with in this project

In [120]:
import pandas as pd
import numpy as np

Reading the csv files into pandas

In [121]:
dete_survey = pd.read_csv('dete_survey.csv')

tafe_survey = pd.read_csv('tafe_survey.csv')

Exploring the DETE dataset

In [122]:
dete_survey.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 822 entries, 0 to 821
Data columns (total 56 columns):
 #   Column                               Non-Null Count  Dtype 
---  ------                               --------------  ----- 
 0   ID                                   822 non-null    int64 
 1   SeparationType                       822 non-null    object
 2   Cease Date                           822 non-null    object
 3   DETE Start Date                      822 non-null    object
 4   Role Start Date                      822 non-null    object
 5   Position                             817 non-null    object
 6   Classification                       455 non-null    object
 7   Region                               822 non-null    object
 8   Business Unit                        126 non-null    object
 9   Employment Status                    817 non-null    object
 10  Career move to public sector         822 non-null    bool  
 11  Career move to private sector        822 non-null    bool  
 12  Interpersonal conflicts              822 non-null    bool  
 13  Job dissatisfaction                  822 non-null    bool  
 14  Dissatisfaction with the department  822 non-null    bool  
 15  Physical work environment            822 non-null    bool  
 16  Lack of recognition                  822 non-null    bool  
 17  Lack of job security                 822 non-null    bool  
 18  Work location                        822 non-null    bool  
 19  Employment conditions                822 non-null    bool  
 20  Maternity/family                     822 non-null    bool  
 21  Relocation                           822 non-null    bool  
 22  Study/Travel                         822 non-null    bool  
 23  Ill Health                           822 non-null    bool  
 24  Traumatic incident                   822 non-null    bool  
 25  Work life balance                    822 non-null    bool  
 26  Workload                             822 non-null    bool  
 27  None of the above                    822 non-null    bool  
 28  Professional Development             808 non-null    object
 29  Opportunities for promotion          735 non-null    object
 30  Staff morale                         816 non-null    object
 31  Workplace issue                      788 non-null    object
 32  Physical environment                 817 non-null    object
 33  Worklife balance                     815 non-null    object
 34  Stress and pressure support          810 non-null    object
 35  Performance of supervisor            813 non-null    object
 36  Peer support                         812 non-null    object
 37  Initiative                           813 non-null    object
 38  Skills                               811 non-null    object
 39  Coach                                767 non-null    object
 40  Career Aspirations                   746 non-null    object
 41  Feedback                             792 non-null    object
 42  Further PD                           768 non-null    object
 43  Communication                        814 non-null    object
 44  My say                               812 non-null    object
 45  Information                          816 non-null    object
 46  Kept informed                        813 non-null    object
 47  Wellness programs                    766 non-null    object
 48  Health & Safety                      793 non-null    object
 49  Gender                               798 non-null    object
 50  Age                                  811 non-null    object
 51  Aboriginal                           16 non-null     object
 52  Torres Strait                        3 non-null      object
 53  South Sea                            7 non-null      object
 54  Disability                           23 non-null     object
 55  NESB                                 32 non-null     object
dtypes: bool(18), int64(1), object(37)
memory usage: 258.6+ KB
In [123]:
dete_survey.head()
Out[123]:
ID SeparationType Cease Date DETE Start Date Role Start Date Position Classification Region Business Unit Employment Status ... Kept informed Wellness programs Health & Safety Gender Age Aboriginal Torres Strait South Sea Disability NESB
0 1 Ill Health Retirement 08/2012 1984 2004 Public Servant A01-A04 Central Office Corporate Strategy and Peformance Permanent Full-time ... N N N Male 56-60 NaN NaN NaN NaN Yes
1 2 Voluntary Early Retirement (VER) 08/2012 Not Stated Not Stated Public Servant AO5-AO7 Central Office Corporate Strategy and Peformance Permanent Full-time ... N N N Male 56-60 NaN NaN NaN NaN NaN
2 3 Voluntary Early Retirement (VER) 05/2012 2011 2011 Schools Officer NaN Central Office Education Queensland Permanent Full-time ... N N N Male 61 or older NaN NaN NaN NaN NaN
3 4 Resignation-Other reasons 05/2012 2005 2006 Teacher Primary Central Queensland NaN Permanent Full-time ... A N A Female 36-40 NaN NaN NaN NaN NaN
4 5 Age Retirement 05/2012 1970 1989 Head of Curriculum/Head of Special Education NaN South East NaN Permanent Full-time ... N A M Female 61 or older NaN NaN NaN NaN NaN

5 rows × 56 columns

In [124]:
# Checking missing data

dete_survey.isnull().sum()
Out[124]:
ID                                       0
SeparationType                           0
Cease Date                               0
DETE Start Date                          0
Role Start Date                          0
Position                                 5
Classification                         367
Region                                   0
Business Unit                          696
Employment Status                        5
Career move to public sector             0
Career move to private sector            0
Interpersonal conflicts                  0
Job dissatisfaction                      0
Dissatisfaction with the department      0
Physical work environment                0
Lack of recognition                      0
Lack of job security                     0
Work location                            0
Employment conditions                    0
Maternity/family                         0
Relocation                               0
Study/Travel                             0
Ill Health                               0
Traumatic incident                       0
Work life balance                        0
Workload                                 0
None of the above                        0
Professional Development                14
Opportunities for promotion             87
Staff morale                             6
Workplace issue                         34
Physical environment                     5
Worklife balance                         7
Stress and pressure support             12
Performance of supervisor                9
Peer support                            10
Initiative                               9
Skills                                  11
Coach                                   55
Career Aspirations                      76
Feedback                                30
Further PD                              54
Communication                            8
My say                                  10
Information                              6
Kept informed                            9
Wellness programs                       56
Health & Safety                         29
Gender                                  24
Age                                     11
Aboriginal                             806
Torres Strait                          819
South Sea                              815
Disability                             799
NESB                                   790
dtype: int64

Exploring the TAFE dataset

In [125]:
tafe_survey.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 702 entries, 0 to 701
Data columns (total 72 columns):
 #   Column                                                                                                                                                         Non-Null Count  Dtype  
---  ------                                                                                                                                                         --------------  -----  
 0   Record ID                                                                                                                                                      702 non-null    float64
 1   Institute                                                                                                                                                      702 non-null    object 
 2   WorkArea                                                                                                                                                       702 non-null    object 
 3   CESSATION YEAR                                                                                                                                                 695 non-null    float64
 4   Reason for ceasing employment                                                                                                                                  701 non-null    object 
 5   Contributing Factors. Career Move - Public Sector                                                                                                              437 non-null    object 
 6   Contributing Factors. Career Move - Private Sector                                                                                                             437 non-null    object 
 7   Contributing Factors. Career Move - Self-employment                                                                                                            437 non-null    object 
 8   Contributing Factors. Ill Health                                                                                                                               437 non-null    object 
 9   Contributing Factors. Maternity/Family                                                                                                                         437 non-null    object 
 10  Contributing Factors. Dissatisfaction                                                                                                                          437 non-null    object 
 11  Contributing Factors. Job Dissatisfaction                                                                                                                      437 non-null    object 
 12  Contributing Factors. Interpersonal Conflict                                                                                                                   437 non-null    object 
 13  Contributing Factors. Study                                                                                                                                    437 non-null    object 
 14  Contributing Factors. Travel                                                                                                                                   437 non-null    object 
 15  Contributing Factors. Other                                                                                                                                    437 non-null    object 
 16  Contributing Factors. NONE                                                                                                                                     437 non-null    object 
 17  Main Factor. Which of these was the main factor for leaving?                                                                                                   113 non-null    object 
 18  InstituteViews. Topic:1. I feel the senior leadership had a clear vision and direction                                                                         608 non-null    object 
 19  InstituteViews. Topic:2. I was given access to skills training to help me do my job better                                                                     613 non-null    object 
 20  InstituteViews. Topic:3. I was given adequate opportunities for personal development                                                                           610 non-null    object 
 21  InstituteViews. Topic:4. I was given adequate opportunities for promotion within %Institute]Q25LBL%                                                            608 non-null    object 
 22  InstituteViews. Topic:5. I felt the salary for the job was right for the responsibilities I had                                                                615 non-null    object 
 23  InstituteViews. Topic:6. The organisation recognised when staff did good work                                                                                  607 non-null    object 
 24  InstituteViews. Topic:7. Management was generally supportive of me                                                                                             614 non-null    object 
 25  InstituteViews. Topic:8. Management was generally supportive of my team                                                                                        608 non-null    object 
 26  InstituteViews. Topic:9. I was kept informed of the changes in the organisation which would affect me                                                          610 non-null    object 
 27  InstituteViews. Topic:10. Staff morale was positive within the Institute                                                                                       602 non-null    object 
 28  InstituteViews. Topic:11. If I had a workplace issue it was dealt with quickly                                                                                 601 non-null    object 
 29  InstituteViews. Topic:12. If I had a workplace issue it was dealt with efficiently                                                                             597 non-null    object 
 30  InstituteViews. Topic:13. If I had a workplace issue it was dealt with discreetly                                                                              601 non-null    object 
 31  WorkUnitViews. Topic:14. I was satisfied with the quality of the management and supervision within my work unit                                                609 non-null    object 
 32  WorkUnitViews. Topic:15. I worked well with my colleagues                                                                                                      605 non-null    object 
 33  WorkUnitViews. Topic:16. My job was challenging and interesting                                                                                                607 non-null    object 
 34  WorkUnitViews. Topic:17. I was encouraged to use my initiative in the course of my work                                                                        610 non-null    object 
 35  WorkUnitViews. Topic:18. I had sufficient contact with other people in my job                                                                                  613 non-null    object 
 36  WorkUnitViews. Topic:19. I was given adequate support and co-operation by my peers to enable me to do my job                                                   609 non-null    object 
 37  WorkUnitViews. Topic:20. I was able to use the full range of my skills in my job                                                                               609 non-null    object 
 38  WorkUnitViews. Topic:21. I was able to use the full range of my abilities in my job. ; Category:Level of Agreement; Question:YOUR VIEWS ABOUT YOUR WORK UNIT]  608 non-null    object 
 39  WorkUnitViews. Topic:22. I was able to use the full range of my knowledge in my job                                                                            608 non-null    object 
 40  WorkUnitViews. Topic:23. My job provided sufficient variety                                                                                                    611 non-null    object 
 41  WorkUnitViews. Topic:24. I was able to cope with the level of stress and pressure in my job                                                                    610 non-null    object 
 42  WorkUnitViews. Topic:25. My job allowed me to balance the demands of work and family to my satisfaction                                                        611 non-null    object 
 43  WorkUnitViews. Topic:26. My supervisor gave me adequate personal recognition and feedback on my performance                                                    606 non-null    object 
 44  WorkUnitViews. Topic:27. My working environment was satisfactory e.g. sufficient space, good lighting, suitable seating and working area                       610 non-null    object 
 45  WorkUnitViews. Topic:28. I was given the opportunity to mentor and coach others in order for me to pass on my skills and knowledge prior to my cessation date  609 non-null    object 
 46  WorkUnitViews. Topic:29. There was adequate communication between staff in my unit                                                                             603 non-null    object 
 47  WorkUnitViews. Topic:30. Staff morale was positive within my work unit                                                                                         606 non-null    object 
 48  Induction. Did you undertake Workplace Induction?                                                                                                              619 non-null    object 
 49  InductionInfo. Topic:Did you undertake a Corporate Induction?                                                                                                  432 non-null    object 
 50  InductionInfo. Topic:Did you undertake a Institute Induction?                                                                                                  483 non-null    object 
 51  InductionInfo. Topic: Did you undertake Team Induction?                                                                                                        440 non-null    object 
 52  InductionInfo. Face to Face Topic:Did you undertake a Corporate Induction; Category:How it was conducted?                                                      555 non-null    object 
 53  InductionInfo. On-line Topic:Did you undertake a Corporate Induction; Category:How it was conducted?                                                           555 non-null    object 
 54  InductionInfo. Induction Manual Topic:Did you undertake a Corporate Induction?                                                                                 555 non-null    object 
 55  InductionInfo. Face to Face Topic:Did you undertake a Institute Induction?                                                                                     530 non-null    object 
 56  InductionInfo. On-line Topic:Did you undertake a Institute Induction?                                                                                          555 non-null    object 
 57  InductionInfo. Induction Manual Topic:Did you undertake a Institute Induction?                                                                                 553 non-null    object 
 58  InductionInfo. Face to Face Topic: Did you undertake Team Induction; Category?                                                                                 555 non-null    object 
 59  InductionInfo. On-line Topic: Did you undertake Team Induction?process you undertook and how it was conducted.]                                                555 non-null    object 
 60  InductionInfo. Induction Manual Topic: Did you undertake Team Induction?                                                                                       555 non-null    object 
 61  Workplace. Topic:Did you and your Manager develop a Performance and Professional Development Plan (PPDP)?                                                      608 non-null    object 
 62  Workplace. Topic:Does your workplace promote a work culture free from all forms of unlawful discrimination?                                                    594 non-null    object 
 63  Workplace. Topic:Does your workplace promote and practice the principles of employment equity?                                                                 587 non-null    object 
 64  Workplace. Topic:Does your workplace value the diversity of its employees?                                                                                     586 non-null    object 
 65  Workplace. Topic:Would you recommend the Institute as an employer to others?                                                                                   581 non-null    object 
 66  Gender. What is your Gender?                                                                                                                                   596 non-null    object 
 67  CurrentAge. Current Age                                                                                                                                        596 non-null    object 
 68  Employment Type. Employment Type                                                                                                                               596 non-null    object 
 69  Classification. Classification                                                                                                                                 596 non-null    object 
 70  LengthofServiceOverall. Overall Length of Service at Institute (in years)                                                                                      596 non-null    object 
 71  LengthofServiceCurrent. Length of Service at current workplace (in years)                                                                                      596 non-null    object 
dtypes: float64(2), object(70)
memory usage: 395.0+ KB
In [126]:
tafe_survey.head()
Out[126]:
Record ID Institute WorkArea CESSATION YEAR Reason for ceasing employment Contributing Factors. Career Move - Public Sector Contributing Factors. Career Move - Private Sector Contributing Factors. Career Move - Self-employment Contributing Factors. Ill Health Contributing Factors. Maternity/Family ... Workplace. Topic:Does your workplace promote a work culture free from all forms of unlawful discrimination? Workplace. Topic:Does your workplace promote and practice the principles of employment equity? Workplace. Topic:Does your workplace value the diversity of its employees? Workplace. Topic:Would you recommend the Institute as an employer to others? Gender. What is your Gender? CurrentAge. Current Age Employment Type. Employment Type Classification. Classification LengthofServiceOverall. Overall Length of Service at Institute (in years) LengthofServiceCurrent. Length of Service at current workplace (in years)
0 6.341330e+17 Southern Queensland Institute of TAFE Non-Delivery (corporate) 2010.0 Contract Expired NaN NaN NaN NaN NaN ... Yes Yes Yes Yes Female 26 30 Temporary Full-time Administration (AO) 1-2 1-2
1 6.341337e+17 Mount Isa Institute of TAFE Non-Delivery (corporate) 2010.0 Retirement - - - - - ... Yes Yes Yes Yes NaN NaN NaN NaN NaN NaN
2 6.341388e+17 Mount Isa Institute of TAFE Delivery (teaching) 2010.0 Retirement - - - - - ... Yes Yes Yes Yes NaN NaN NaN NaN NaN NaN
3 6.341399e+17 Mount Isa Institute of TAFE Non-Delivery (corporate) 2010.0 Resignation - - - - - ... Yes Yes Yes Yes NaN NaN NaN NaN NaN NaN
4 6.341466e+17 Southern Queensland Institute of TAFE Delivery (teaching) 2010.0 Resignation - Career Move - Private Sector - - - ... Yes Yes Yes Yes Male 41 45 Permanent Full-time Teacher (including LVT) 3-4 3-4

5 rows × 72 columns

In [ ]:
 

First Observations:

  • "Not Stated" values that indicate they are missing values, but they are not represented by NaN.
  • Both dataframes have several columns we will not need for this project
  • Both dataframes have many of the same columns with different names.
  • There are several columns related to satisfaction/dissatisfaction.

Fixing the first item "Not Stated" by reading the csv file again and passing the parameter na_values to read Not Started as NaN.

In [127]:
#read in the data again
dete_survey = pd.read_csv('dete_survey.csv', na_values='Not Stated')

#checking if the changes were correctly applied

dete_survey.head()
Out[127]:
ID SeparationType Cease Date DETE Start Date Role Start Date Position Classification Region Business Unit Employment Status ... Kept informed Wellness programs Health & Safety Gender Age Aboriginal Torres Strait South Sea Disability NESB
0 1 Ill Health Retirement 08/2012 1984.0 2004.0 Public Servant A01-A04 Central Office Corporate Strategy and Peformance Permanent Full-time ... N N N Male 56-60 NaN NaN NaN NaN Yes
1 2 Voluntary Early Retirement (VER) 08/2012 NaN NaN Public Servant AO5-AO7 Central Office Corporate Strategy and Peformance Permanent Full-time ... N N N Male 56-60 NaN NaN NaN NaN NaN
2 3 Voluntary Early Retirement (VER) 05/2012 2011.0 2011.0 Schools Officer NaN Central Office Education Queensland Permanent Full-time ... N N N Male 61 or older NaN NaN NaN NaN NaN
3 4 Resignation-Other reasons 05/2012 2005.0 2006.0 Teacher Primary Central Queensland NaN Permanent Full-time ... A N A Female 36-40 NaN NaN NaN NaN NaN
4 5 Age Retirement 05/2012 1970.0 1989.0 Head of Curriculum/Head of Special Education NaN South East NaN Permanent Full-time ... N A M Female 61 or older NaN NaN NaN NaN NaN

5 rows × 56 columns

Dropping the columns from each dataframe that we will not use in our analysis

In [128]:
dete_survey_updated = dete_survey.drop(dete_survey.columns[28:49], axis=1)

tafe_survey_updated = tafe_survey.drop(tafe_survey.columns[17:66], axis=1)

Now both dataframes have only the columns that are relevant for this project. Let's confirm that by printing the first 5 rows.

In [129]:
dete_survey_updated.head()
Out[129]:
ID SeparationType Cease Date DETE Start Date Role Start Date Position Classification Region Business Unit Employment Status ... Work life balance Workload None of the above Gender Age Aboriginal Torres Strait South Sea Disability NESB
0 1 Ill Health Retirement 08/2012 1984.0 2004.0 Public Servant A01-A04 Central Office Corporate Strategy and Peformance Permanent Full-time ... False False True Male 56-60 NaN NaN NaN NaN Yes
1 2 Voluntary Early Retirement (VER) 08/2012 NaN NaN Public Servant AO5-AO7 Central Office Corporate Strategy and Peformance Permanent Full-time ... False False False Male 56-60 NaN NaN NaN NaN NaN
2 3 Voluntary Early Retirement (VER) 05/2012 2011.0 2011.0 Schools Officer NaN Central Office Education Queensland Permanent Full-time ... False False True Male 61 or older NaN NaN NaN NaN NaN
3 4 Resignation-Other reasons 05/2012 2005.0 2006.0 Teacher Primary Central Queensland NaN Permanent Full-time ... False False False Female 36-40 NaN NaN NaN NaN NaN
4 5 Age Retirement 05/2012 1970.0 1989.0 Head of Curriculum/Head of Special Education NaN South East NaN Permanent Full-time ... True False False Female 61 or older NaN NaN NaN NaN NaN

5 rows × 35 columns

In [130]:
tafe_survey_updated.head()
Out[130]:
Record ID Institute WorkArea CESSATION YEAR Reason for ceasing employment Contributing Factors. Career Move - Public Sector Contributing Factors. Career Move - Private Sector Contributing Factors. Career Move - Self-employment Contributing Factors. Ill Health Contributing Factors. Maternity/Family ... Contributing Factors. Study Contributing Factors. Travel Contributing Factors. Other Contributing Factors. NONE Gender. What is your Gender? CurrentAge. Current Age Employment Type. Employment Type Classification. Classification LengthofServiceOverall. Overall Length of Service at Institute (in years) LengthofServiceCurrent. Length of Service at current workplace (in years)
0 6.341330e+17 Southern Queensland Institute of TAFE Non-Delivery (corporate) 2010.0 Contract Expired NaN NaN NaN NaN NaN ... NaN NaN NaN NaN Female 26 30 Temporary Full-time Administration (AO) 1-2 1-2
1 6.341337e+17 Mount Isa Institute of TAFE Non-Delivery (corporate) 2010.0 Retirement - - - - - ... - Travel - - NaN NaN NaN NaN NaN NaN
2 6.341388e+17 Mount Isa Institute of TAFE Delivery (teaching) 2010.0 Retirement - - - - - ... - - - NONE NaN NaN NaN NaN NaN NaN
3 6.341399e+17 Mount Isa Institute of TAFE Non-Delivery (corporate) 2010.0 Resignation - - - - - ... - Travel - - NaN NaN NaN NaN NaN NaN
4 6.341466e+17 Southern Queensland Institute of TAFE Delivery (teaching) 2010.0 Resignation - Career Move - Private Sector - - - ... - - - - Male 41 45 Permanent Full-time Teacher (including LVT) 3-4 3-4

5 rows × 23 columns

Exploring the Columns names

Because we eventually want to combine them, we'll have to standardize the column names. We will start by doing the following in the dete_survey_updated dataframe:

  • Make all the capitalization lowercase.
  • Remove any trailing whitespace from the end of the strings.
  • Replace spaces with underscores ('_').
In [131]:
#cleaning strings
dete_survey_updated.columns = dete_survey_updated.columns.str.replace(" ", "_").str.strip().str.lower()

#Confirming the changes were successfully applied
dete_survey_updated.head()
Out[131]:
id separationtype cease_date dete_start_date role_start_date position classification region business_unit employment_status ... work_life_balance workload none_of_the_above gender age aboriginal torres_strait south_sea disability nesb
0 1 Ill Health Retirement 08/2012 1984.0 2004.0 Public Servant A01-A04 Central Office Corporate Strategy and Peformance Permanent Full-time ... False False True Male 56-60 NaN NaN NaN NaN Yes
1 2 Voluntary Early Retirement (VER) 08/2012 NaN NaN Public Servant AO5-AO7 Central Office Corporate Strategy and Peformance Permanent Full-time ... False False False Male 56-60 NaN NaN NaN NaN NaN
2 3 Voluntary Early Retirement (VER) 05/2012 2011.0 2011.0 Schools Officer NaN Central Office Education Queensland Permanent Full-time ... False False True Male 61 or older NaN NaN NaN NaN NaN
3 4 Resignation-Other reasons 05/2012 2005.0 2006.0 Teacher Primary Central Queensland NaN Permanent Full-time ... False False False Female 36-40 NaN NaN NaN NaN NaN
4 5 Age Retirement 05/2012 1970.0 1989.0 Head of Curriculum/Head of Special Education NaN South East NaN Permanent Full-time ... True False False Female 61 or older NaN NaN NaN NaN NaN

5 rows × 35 columns

Updating the names of some columns in the tafe_survey_updated dataset to match the dete_survey_updated

In [132]:
#Mapping columns
col_name_mapping = ({'Record ID': 'id', 'CESSATION YEAR': 'cease_date', 
                     'Reason for ceasing employment': 'separationtype',
                     'Gender. What is your Gender?': 'gender', 
                     'CurrentAge. Current Age': 'age',
                     'Employment Type. Employment Type': 'employment_status',
                     'Classification. Classification': 'position',
                     'LengthofServiceOverall. Overall Length of Service at Institute (in years)': 'institute_service',
                     'LengthofServiceCurrent. Length of Service at current workplace (in years)': 'role_service'

})

tafe_survey_updated.rename(columns=col_name_mapping, inplace=True)

#Printing first 5 rows
tafe_survey_updated.head()
Out[132]:
id Institute WorkArea cease_date separationtype Contributing Factors. Career Move - Public Sector Contributing Factors. Career Move - Private Sector Contributing Factors. Career Move - Self-employment Contributing Factors. Ill Health Contributing Factors. Maternity/Family ... Contributing Factors. Study Contributing Factors. Travel Contributing Factors. Other Contributing Factors. NONE gender age employment_status position institute_service role_service
0 6.341330e+17 Southern Queensland Institute of TAFE Non-Delivery (corporate) 2010.0 Contract Expired NaN NaN NaN NaN NaN ... NaN NaN NaN NaN Female 26 30 Temporary Full-time Administration (AO) 1-2 1-2
1 6.341337e+17 Mount Isa Institute of TAFE Non-Delivery (corporate) 2010.0 Retirement - - - - - ... - Travel - - NaN NaN NaN NaN NaN NaN
2 6.341388e+17 Mount Isa Institute of TAFE Delivery (teaching) 2010.0 Retirement - - - - - ... - - - NONE NaN NaN NaN NaN NaN NaN
3 6.341399e+17 Mount Isa Institute of TAFE Non-Delivery (corporate) 2010.0 Resignation - - - - - ... - Travel - - NaN NaN NaN NaN NaN NaN
4 6.341466e+17 Southern Queensland Institute of TAFE Delivery (teaching) 2010.0 Resignation - Career Move - Private Sector - - - ... - - - - Male 41 45 Permanent Full-time Teacher (including LVT) 3-4 3-4

5 rows × 23 columns

Removing more of the data that we don't need for our analysis

The column separationtype contains a couple of different separation types. We will only analyze survey respondents who resigned. We're going to select the rows of respondents who have a Resignation separation type.

In [133]:
#Counting values in the separationtype column - DETE
dete_survey_updated['separationtype'].value_counts()
Out[133]:
Age Retirement                          285
Resignation-Other reasons               150
Resignation-Other employer               91
Resignation-Move overseas/interstate     70
Voluntary Early Retirement (VER)         67
Ill Health Retirement                    61
Other                                    49
Contract Expired                         34
Termination                              15
Name: separationtype, dtype: int64

For dete_survey_updated, as it has 3 different types of resignation, we will use a regex to extra all rows that has the word "Resignation" as part of the string value in the column separationtype.

In [134]:
#Counting values in the separationtype column - TAFE

tafe_survey_updated['separationtype'].value_counts()
Out[134]:
Resignation                 340
Contract Expired            127
Retrenchment/ Redundancy    104
Retirement                   82
Transfer                     25
Termination                  23
Name: separationtype, dtype: int64
In [135]:
#we are adding the df.copy() to the end of the new dataframe to avoid the SettingWithCopy warning
pattern = r"[Rr]esignation"

dete_resignations = dete_survey_updated[dete_survey_updated['separationtype'].str.contains(pattern)].copy()

dete_resignations.head()
Out[135]:
id separationtype cease_date dete_start_date role_start_date position classification region business_unit employment_status ... work_life_balance workload none_of_the_above gender age aboriginal torres_strait south_sea disability nesb
3 4 Resignation-Other reasons 05/2012 2005.0 2006.0 Teacher Primary Central Queensland NaN Permanent Full-time ... False False False Female 36-40 NaN NaN NaN NaN NaN
5 6 Resignation-Other reasons 05/2012 1994.0 1997.0 Guidance Officer NaN Central Office Education Queensland Permanent Full-time ... False False False Female 41-45 NaN NaN NaN NaN NaN
8 9 Resignation-Other reasons 07/2012 2009.0 2009.0 Teacher Secondary North Queensland NaN Permanent Full-time ... False False False Female 31-35 NaN NaN NaN NaN NaN
9 10 Resignation-Other employer 2012 1997.0 2008.0 Teacher Aide NaN NaN NaN Permanent Part-time ... False False False Female 46-50 NaN NaN NaN NaN NaN
11 12 Resignation-Move overseas/interstate 2012 2009.0 2009.0 Teacher Secondary Far North Queensland NaN Permanent Full-time ... False False False Male 31-35 NaN NaN NaN NaN NaN

5 rows × 35 columns

In [136]:
#We have to clean from Nan values to be able to do a boolean index
tafe_survey_updated_na = tafe_survey_updated[~tafe_survey_updated['separationtype'].isnull()]

tafe_resignations = tafe_survey_updated_na[tafe_survey_updated_na['separationtype'].str.contains(pattern)].copy()

#Priting first 5 rows
tafe_resignations.head()
Out[136]:
id Institute WorkArea cease_date separationtype Contributing Factors. Career Move - Public Sector Contributing Factors. Career Move - Private Sector Contributing Factors. Career Move - Self-employment Contributing Factors. Ill Health Contributing Factors. Maternity/Family ... Contributing Factors. Study Contributing Factors. Travel Contributing Factors. Other Contributing Factors. NONE gender age employment_status position institute_service role_service
3 6.341399e+17 Mount Isa Institute of TAFE Non-Delivery (corporate) 2010.0 Resignation - - - - - ... - Travel - - NaN NaN NaN NaN NaN NaN
4 6.341466e+17 Southern Queensland Institute of TAFE Delivery (teaching) 2010.0 Resignation - Career Move - Private Sector - - - ... - - - - Male 41 45 Permanent Full-time Teacher (including LVT) 3-4 3-4
5 6.341475e+17 Southern Queensland Institute of TAFE Delivery (teaching) 2010.0 Resignation - - - - - ... - - Other - Female 56 or older Contract/casual Teacher (including LVT) 7-10 7-10
6 6.341520e+17 Barrier Reef Institute of TAFE Non-Delivery (corporate) 2010.0 Resignation - Career Move - Private Sector - - Maternity/Family ... - - Other - Male 20 or younger Temporary Full-time Administration (AO) 3-4 3-4
7 6.341537e+17 Southern Queensland Institute of TAFE Delivery (teaching) 2010.0 Resignation - - - - - ... - - Other - Male 46 50 Permanent Full-time Teacher (including LVT) 3-4 3-4

5 rows × 23 columns

Checking for inconsistent data in the columns cease_date and dete_start_date on both datasets

we'll focus on verifying that the years in the cease_date and dete_start_date columns make sense.

Since the cease_date is the last year of the person's employment and the dete_start_date is the person's first year of employment, it wouldn't make sense to have years after the current date.

Given that most people in this field start working in their 20s, it's also unlikely that the dete_start_date was before the year 1940.

dete_resignations

In [137]:
#Column cease_date

dete_resignations['cease_date'].value_counts()
Out[137]:
2012       126
2013        74
01/2014     22
12/2013     17
06/2013     14
09/2013     11
11/2013      9
07/2013      9
10/2013      6
08/2013      4
05/2012      2
05/2013      2
07/2012      1
09/2010      1
07/2006      1
2010         1
Name: cease_date, dtype: int64

Some of the values have month and year combined. We will remove the month and leave only the year

In [138]:
#Extracting the year and assigning it back to the series.

pattern = r"(?P<Years>[1-2][0-9]{3})"

dete_resignations['cease_date'] = dete_resignations['cease_date'].str.extract(pattern)

#Converting dtype to float

dete_resignations['cease_date'] = dete_resignations['cease_date'].astype(float)

# checking if the changes were correctly applied
dete_resignations['cease_date'].value_counts()
Out[138]:
2013.0    146
2012.0    129
2014.0     22
2010.0      2
2006.0      1
Name: cease_date, dtype: int64
In [139]:
#Column dete_start_date

dete_resignations['dete_start_date'].value_counts().sort_index(ascending=True)
Out[139]:
1963.0     1
1971.0     1
1972.0     1
1973.0     1
1974.0     2
1975.0     1
1976.0     2
1977.0     1
1980.0     5
1982.0     1
1983.0     2
1984.0     1
1985.0     3
1986.0     3
1987.0     1
1988.0     4
1989.0     4
1990.0     5
1991.0     4
1992.0     6
1993.0     5
1994.0     6
1995.0     4
1996.0     6
1997.0     5
1998.0     6
1999.0     8
2000.0     9
2001.0     3
2002.0     6
2003.0     6
2004.0    14
2005.0    15
2006.0    13
2007.0    21
2008.0    22
2009.0    13
2010.0    17
2011.0    24
2012.0    21
2013.0    10
Name: dete_start_date, dtype: int64

The data in those columns seem fine, between the date range we expected - the oldest year is 1963 and there is no date greater than the current date

tafe_resignations Column cease_date

In [140]:
tafe_resignations['cease_date'].value_counts()
Out[140]:
2011.0    116
2012.0     94
2010.0     68
2013.0     55
2009.0      2
Name: cease_date, dtype: int64

The column cease_date in tafe_resignation also looks okay and matches the dtype from the dataset dete_resignations.

In the Human Resources field, the length of time an employee spent in a workplace is referred to as their years of service.

The tafe_resignations dataframe already contains a "service" column, which we renamed to institute_service. In order to analyze both surveys together, we'll have to create a corresponding institute_service column in dete_resignations.

Let's check if we have data that can be used to calculate the length of time the employee spent in their workplace before moving on.

In [141]:
dete_resignations.head()
Out[141]:
id separationtype cease_date dete_start_date role_start_date position classification region business_unit employment_status ... work_life_balance workload none_of_the_above gender age aboriginal torres_strait south_sea disability nesb
3 4 Resignation-Other reasons 2012.0 2005.0 2006.0 Teacher Primary Central Queensland NaN Permanent Full-time ... False False False Female 36-40 NaN NaN NaN NaN NaN
5 6 Resignation-Other reasons 2012.0 1994.0 1997.0 Guidance Officer NaN Central Office Education Queensland Permanent Full-time ... False False False Female 41-45 NaN NaN NaN NaN NaN
8 9 Resignation-Other reasons 2012.0 2009.0 2009.0 Teacher Secondary North Queensland NaN Permanent Full-time ... False False False Female 31-35 NaN NaN NaN NaN NaN
9 10 Resignation-Other employer 2012.0 1997.0 2008.0 Teacher Aide NaN NaN NaN Permanent Part-time ... False False False Female 46-50 NaN NaN NaN NaN NaN
11 12 Resignation-Move overseas/interstate 2012.0 2009.0 2009.0 Teacher Secondary Far North Queensland NaN Permanent Full-time ... False False False Male 31-35 NaN NaN NaN NaN NaN

5 rows × 35 columns

We can calculate the years of service for dete_resignations by subtracting the cease_date from the dete_start_date. Let's go ahead and create the institute_service column

In [142]:
dete_resignations['institute_service'] = dete_resignations['cease_date'] - dete_resignations['dete_start_date']
In [143]:
#Checking new column in our dataset
dete_resignations.head()
Out[143]:
id separationtype cease_date dete_start_date role_start_date position classification region business_unit employment_status ... workload none_of_the_above gender age aboriginal torres_strait south_sea disability nesb institute_service
3 4 Resignation-Other reasons 2012.0 2005.0 2006.0 Teacher Primary Central Queensland NaN Permanent Full-time ... False False Female 36-40 NaN NaN NaN NaN NaN 7.0
5 6 Resignation-Other reasons 2012.0 1994.0 1997.0 Guidance Officer NaN Central Office Education Queensland Permanent Full-time ... False False Female 41-45 NaN NaN NaN NaN NaN 18.0
8 9 Resignation-Other reasons 2012.0 2009.0 2009.0 Teacher Secondary North Queensland NaN Permanent Full-time ... False False Female 31-35 NaN NaN NaN NaN NaN 3.0
9 10 Resignation-Other employer 2012.0 1997.0 2008.0 Teacher Aide NaN NaN NaN Permanent Part-time ... False False Female 46-50 NaN NaN NaN NaN NaN 15.0
11 12 Resignation-Move overseas/interstate 2012.0 2009.0 2009.0 Teacher Secondary Far North Queensland NaN Permanent Full-time ... False False Male 31-35 NaN NaN NaN NaN NaN 3.0

5 rows × 36 columns

Analyzing the columns related to job satisfaction on both dataframes

Checking the values in the columns 'Contributing Factors. Dissatisfaction' and 'Contributing Factors. Job Dissatisfaction' in the tafe_resignations dataframe.

In [144]:
tafe_resignations['Contributing Factors. Dissatisfaction'].value_counts()
Out[144]:
-                                         277
Contributing Factors. Dissatisfaction      55
Name: Contributing Factors. Dissatisfaction, dtype: int64
In [145]:
tafe_resignations['Contributing Factors. Job Dissatisfaction'].value_counts()
Out[145]:
-                      270
Job Dissatisfaction     62
Name: Contributing Factors. Job Dissatisfaction, dtype: int64

Creating column dissatisfied for tafe_resignations

In [146]:
#Creating a function to update values in both columns

def update_vals(value):
    if pd.isnull(value):
        return np.nan
    elif value == '-':
        return False 
    else:
        return True

Applying the function we created to update the values in the new column 'dissatisfied' for TAFE dataset

In [147]:
cols = ['Contributing Factors. Dissatisfaction', 'Contributing Factors. Job Dissatisfaction']

#creating new column dissatisfied
tafe_resignations['dissatisfied'] = tafe_resignations[cols].applymap(update_vals).any(axis=1, skipna=False)

#creating a copy of the dataset to avoid SettingWithCopy warn
tafe_resignations_up = tafe_resignations.copy()

#Displaying value counts
tafe_resignations_up['dissatisfied'].value_counts()
Out[147]:
False    241
True      91
Name: dissatisfied, dtype: int64

Creating a dissatisfied column to dete_resignations

In [148]:
dete_resignations['dissatisfied'] = dete_resignations[['job_dissatisfaction',
        'dissatisfaction_with_the_department','physical_work_environment',
        'lack_of_recognition','lack_of_job_security','work_location',
        'employment_conditions','work_life_balance',
        'workload']].any(axis = 1,skipna = False)

dete_resignations_up = dete_resignations.copy()

dete_resignations_up['dissatisfied'].value_counts(dropna= False)
Out[148]:
False    162
True     149
Name: dissatisfied, dtype: int64

We've successfully created the column dissatisfied for both datasets with values True, False and NaN. If the employee indicated any of the factors(column names listed above in the code), we will mark them as dissatisfied in the new column.

Now, we're finally ready to combine our datasets! Our end goal is to aggregate the data according to the institute_service column, so when you combine the data, think about how to get the data into a form that's easy to aggregate.

First, let's add a column institute to each dataframe that will allow us to easily distinguish between the two.

In [149]:
dete_resignations_up['institute'] = 'DETE'
tafe_resignations_up['institute'] = 'TAFE'

Combining the two datasets

In [150]:
combined = pd.concat([dete_resignations_up,tafe_resignations_up],ignore_index = True)

#Printing fist 5 rows
combined.head()
Out[150]:
id separationtype cease_date dete_start_date role_start_date position classification region business_unit employment_status ... Contributing Factors. Ill Health Contributing Factors. Maternity/Family Contributing Factors. Dissatisfaction Contributing Factors. Job Dissatisfaction Contributing Factors. Interpersonal Conflict Contributing Factors. Study Contributing Factors. Travel Contributing Factors. Other Contributing Factors. NONE role_service
0 4.0 Resignation-Other reasons 2012.0 2005.0 2006.0 Teacher Primary Central Queensland NaN Permanent Full-time ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
1 6.0 Resignation-Other reasons 2012.0 1994.0 1997.0 Guidance Officer NaN Central Office Education Queensland Permanent Full-time ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
2 9.0 Resignation-Other reasons 2012.0 2009.0 2009.0 Teacher Secondary North Queensland NaN Permanent Full-time ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
3 10.0 Resignation-Other employer 2012.0 1997.0 2008.0 Teacher Aide NaN NaN NaN Permanent Part-time ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
4 12.0 Resignation-Move overseas/interstate 2012.0 2009.0 2009.0 Teacher Secondary Far North Queensland NaN Permanent Full-time ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN

5 rows × 53 columns

In [151]:
#Quick Exploration of the new dataset
combined.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 651 entries, 0 to 650
Data columns (total 53 columns):
 #   Column                                               Non-Null Count  Dtype  
---  ------                                               --------------  -----  
 0   id                                                   651 non-null    float64
 1   separationtype                                       651 non-null    object 
 2   cease_date                                           635 non-null    float64
 3   dete_start_date                                      283 non-null    float64
 4   role_start_date                                      271 non-null    float64
 5   position                                             598 non-null    object 
 6   classification                                       161 non-null    object 
 7   region                                               265 non-null    object 
 8   business_unit                                        32 non-null     object 
 9   employment_status                                    597 non-null    object 
 10  career_move_to_public_sector                         311 non-null    object 
 11  career_move_to_private_sector                        311 non-null    object 
 12  interpersonal_conflicts                              311 non-null    object 
 13  job_dissatisfaction                                  311 non-null    object 
 14  dissatisfaction_with_the_department                  311 non-null    object 
 15  physical_work_environment                            311 non-null    object 
 16  lack_of_recognition                                  311 non-null    object 
 17  lack_of_job_security                                 311 non-null    object 
 18  work_location                                        311 non-null    object 
 19  employment_conditions                                311 non-null    object 
 20  maternity/family                                     311 non-null    object 
 21  relocation                                           311 non-null    object 
 22  study/travel                                         311 non-null    object 
 23  ill_health                                           311 non-null    object 
 24  traumatic_incident                                   311 non-null    object 
 25  work_life_balance                                    311 non-null    object 
 26  workload                                             311 non-null    object 
 27  none_of_the_above                                    311 non-null    object 
 28  gender                                               592 non-null    object 
 29  age                                                  596 non-null    object 
 30  aboriginal                                           7 non-null      object 
 31  torres_strait                                        0 non-null      object 
 32  south_sea                                            3 non-null      object 
 33  disability                                           8 non-null      object 
 34  nesb                                                 9 non-null      object 
 35  institute_service                                    563 non-null    object 
 36  dissatisfied                                         643 non-null    object 
 37  institute                                            651 non-null    object 
 38  Institute                                            340 non-null    object 
 39  WorkArea                                             340 non-null    object 
 40  Contributing Factors. Career Move - Public Sector    332 non-null    object 
 41  Contributing Factors. Career Move - Private Sector   332 non-null    object 
 42  Contributing Factors. Career Move - Self-employment  332 non-null    object 
 43  Contributing Factors. Ill Health                     332 non-null    object 
 44  Contributing Factors. Maternity/Family               332 non-null    object 
 45  Contributing Factors. Dissatisfaction                332 non-null    object 
 46  Contributing Factors. Job Dissatisfaction            332 non-null    object 
 47  Contributing Factors. Interpersonal Conflict         332 non-null    object 
 48  Contributing Factors. Study                          332 non-null    object 
 49  Contributing Factors. Travel                         332 non-null    object 
 50  Contributing Factors. Other                          332 non-null    object 
 51  Contributing Factors. NONE                           332 non-null    object 
 52  role_service                                         290 non-null    object 
dtypes: float64(4), object(49)
memory usage: 269.7+ KB
In [152]:
#Check for not null values
combined.notnull().sum().sort_values()
Out[152]:
torres_strait                                            0
south_sea                                                3
aboriginal                                               7
disability                                               8
nesb                                                     9
business_unit                                           32
classification                                         161
region                                                 265
role_start_date                                        271
dete_start_date                                        283
role_service                                           290
none_of_the_above                                      311
work_life_balance                                      311
traumatic_incident                                     311
ill_health                                             311
study/travel                                           311
relocation                                             311
maternity/family                                       311
employment_conditions                                  311
workload                                               311
lack_of_job_security                                   311
career_move_to_public_sector                           311
career_move_to_private_sector                          311
interpersonal_conflicts                                311
work_location                                          311
dissatisfaction_with_the_department                    311
physical_work_environment                              311
lack_of_recognition                                    311
job_dissatisfaction                                    311
Contributing Factors. Job Dissatisfaction              332
Contributing Factors. Travel                           332
Contributing Factors. Maternity/Family                 332
Contributing Factors. Ill Health                       332
Contributing Factors. Career Move - Self-employment    332
Contributing Factors. Career Move - Private Sector     332
Contributing Factors. Career Move - Public Sector      332
Contributing Factors. Dissatisfaction                  332
Contributing Factors. Other                            332
Contributing Factors. Interpersonal Conflict           332
Contributing Factors. NONE                             332
Contributing Factors. Study                            332
Institute                                              340
WorkArea                                               340
institute_service                                      563
gender                                                 592
age                                                    596
employment_status                                      597
position                                               598
cease_date                                             635
dissatisfied                                           643
separationtype                                         651
institute                                              651
id                                                     651
dtype: int64

Dropping any columns with less than 500 non null values

In [153]:
combined_updated = combined.dropna(axis = 1, thresh = 500).copy()

combined_updated.head()
Out[153]:
id separationtype cease_date position employment_status gender age institute_service dissatisfied institute
0 4.0 Resignation-Other reasons 2012.0 Teacher Permanent Full-time Female 36-40 7 False DETE
1 6.0 Resignation-Other reasons 2012.0 Guidance Officer Permanent Full-time Female 41-45 18 True DETE
2 9.0 Resignation-Other reasons 2012.0 Teacher Permanent Full-time Female 31-35 3 False DETE
3 10.0 Resignation-Other employer 2012.0 Teacher Aide Permanent Part-time Female 46-50 15 True DETE
4 12.0 Resignation-Move overseas/interstate 2012.0 Teacher Permanent Full-time Male 31-35 3 False DETE

To recap, we've accomplished the following so far:

  • Renamed our columns
  • Dropped any data not needed for our analysis
  • Verified the quality of our data
  • Created a new institute_service column
  • Cleaned the Contributing Factors columns
  • Created a new column indicating if an employee resigned because they were dissatisfied in some way
  • Combined both datasets
  • Dropped columns with missing values

Before we can proceed with our analysis, we need to clean up the institue_service column. It contains values in different formats.

In [154]:
#Check for unique values
combined_updated['institute_service'].unique()
Out[154]:
array([7.0, 18.0, 3.0, 15.0, 14.0, 5.0, nan, 30.0, 32.0, 39.0, 17.0, 9.0,
       6.0, 1.0, 35.0, 38.0, 36.0, 19.0, 4.0, 26.0, 10.0, 8.0, 2.0, 0.0,
       23.0, 13.0, 16.0, 12.0, 21.0, 20.0, 24.0, 33.0, 22.0, 28.0, 49.0,
       11.0, 41.0, 27.0, 42.0, 25.0, 29.0, 34.0, 31.0, '3-4', '7-10',
       '1-2', 'Less than 1 year', '11-20', '5-6', 'More than 20 years'],
      dtype=object)

To analyze the data, we'll convert these numbers into categories:

  • New: Less than 3 years at a company
  • Experienced: 3-6 years at a company
  • Established: 7-10 years at a company
  • Veteran: 11 or more years at a company

Extracting the years of service from each value in the institute_service column.

In [155]:
#Convert column to string type
combined_updated['institute_service'] = combined_updated['institute_service'].astype(str)
In [156]:
#Changing the type to str to extract the year and then converting to float

combined_updated['institute_service'] = combined_updated['institute_service'].str.extract(r'(\d+)').astype('float')
In [157]:
#Check if changes applied correctly

combined_updated['institute_service'].unique()
Out[157]:
array([ 7., 18.,  3., 15., 14.,  5., nan, 30., 32., 39., 17.,  9.,  6.,
        1., 35., 38., 36., 19.,  4., 26., 10.,  8.,  2.,  0., 23., 13.,
       16., 12., 21., 20., 24., 33., 22., 28., 49., 11., 41., 27., 42.,
       25., 29., 34., 31.])

Creating a function to map each value to one of the carrier stages definitions that categorize employees according to the amount of years spent in their workplace:

  • New: Less than 3 years at a company
  • Experienced: 3-6 years at a company
  • Established: 7-10 years at a company
  • Veteran: 11 or more years at a company
In [158]:
# Define function
def map_years_to_category (years):
    if pd.isnull(years):
        return 'Unknown'   # in case years-of-service is unknown, let's make it 'Unknown'
    elif years < 3:
        return ('New')
    elif years < 7:
        return ('Eperienced')
    elif years < 11:
        return ('Established')
    else:
        return ('Veteran')

Creating a new column 'service_cat' to add the carrier categories values

In [159]:
combined_updated['service_cat'] = combined_updated['institute_service'].apply(map_years_to_category)

#Check if changed applied

combined_updated['service_cat'].unique()
Out[159]:
array(['Established', 'Veteran', 'Eperienced', 'Unknown', 'New'],
      dtype=object)

Handling missing values in the column dissatisfied

In [160]:
#Checking the number of True, False and NaN in the dissatisfied column

combined_updated['dissatisfied'].value_counts(dropna=False)
Out[160]:
False    403
True     240
NaN        8
Name: dissatisfied, dtype: int64
In [161]:
#Replacing the NaN values with the value that occurs most frequently - False.

combined_updated['dissatisfied'].fillna(False, inplace=True)

#double checking the changes

combined_updated['dissatisfied'].value_counts(dropna=False)
Out[161]:
False    411
True     240
Name: dissatisfied, dtype: int64

Using the pd.pivot_table method to calculate the percentage of dissatisfied employees in each service_cat group.

In [162]:
pv_combined_updated = pd.pivot_table(combined_updated, index='service_cat', values='dissatisfied')

pv_combined_updated.head()
Out[162]:
dissatisfied
service_cat
Eperienced 0.343023
Established 0.516129
New 0.295337
Unknown 0.295455
Veteran 0.485294
In [163]:
#Plotting the results

%matplotlib inline

pv_combined_updated.plot(kind = 'bar', ylim = (0,1), title = 'Dissatisfied % per Career Stage', legend=False)
Out[163]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f518e986880>

Observations:

  • The highest dissatisfied rates are among employees with 7 - 10 years of service, 51.6%.
  • Veteran employees, 11+ years of service, are the second highest dissatisfied group, with a rate of 48.5%.
  • The third place in dissatisfaction is for the Experienced group, with 34.3%.

There seems to be a direct correlation between years of service and satisfaction level, the longer an employee works for the company, the higher is their dissatisfaction.

Calculating how many people in each age group resigned due to dissatisfaction

In [164]:
combined_updated['age'].unique()
Out[164]:
array(['36-40', '41-45', '31-35', '46-50', '61 or older', '56-60',
       '51-55', '21-25', '26-30', nan, '20 or younger', '41  45',
       '56 or older', '46  50', '36  40', '21  25', '31  35', '26  30'],
      dtype=object)

The column age needs cleaning. Some of the age ranges are missing the dash and have double space and the range 56 or order will be renamed to 56-60. As there is a range 61 or older, it is reasonable to assume that 56 or older actually means 56-60.

We will also drop the nan values.

In [176]:
# Cleaning extra space and adding the -
combined_updated['age'] = combined_updated['age'].str.replace("  ","-").str.replace('56 or older', '56-60')
combined_updated['age'].value_counts(dropna=False).sort_index()
Out[176]:
20 or younger    10
21-25            62
26-30            67
31-35            61
36-40            73
41-45            93
46-50            81
51-55            71
56-60            55
61 or older      23
Name: age, dtype: int64
In [180]:
#Creating a pivot table to calculate the percentage of dissatisfaction by age group

pv_age = pd.pivot_table(combined_updated, index='age', values='dissatisfied')

pv_age
Out[180]:
dissatisfied
age
20 or younger 0.200000
21-25 0.306452
26-30 0.417910
31-35 0.377049
36-40 0.342466
41-45 0.376344
46-50 0.382716
51-55 0.422535
56-60 0.381818
61 or older 0.521739
In [185]:
pv_age.plot(kind = 'bar', ylim = (0,0.7), title = 'Dissatisfaction by Age Group', legend=False)
Out[185]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f518bb272b0>

Employees over 60 years old are the majority - 52.1% - of the dissatisfied age groups.

Let's now analyze each survey separately to see what institute DETE or TAFE had more employees leaving due to job dissatisfaction

In [200]:
pv_dete_tafe = pd.pivot_table(combined_updated, index='service_cat', columns='institute', values='dissatisfied', margins=True)

pv_dete_tafe
Out[200]:
institute DETE TAFE All
service_cat
Eperienced 0.460526 0.250000 0.343023
Established 0.609756 0.333333 0.516129
New 0.375000 0.262774 0.295337
Unknown 0.315789 0.280000 0.295455
Veteran 0.560000 0.277778 0.485294
All 0.479100 0.267647 0.368664
In [201]:
pv_dete_tafe.plot(kind = 'bar', ylim = (0,1), title = 'Dissatisfaction by Employee Group and Institute')
Out[201]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f518b53c2b0>

Conclusion

Let's conclude our analysis by answers the two initial questions we asked at the beginning of this project:

1 - Are employees who only worked for the institutes for a short period of time resigning due to some kind of dissatisfaction? What about employees who have been there longer?

The data shows that the longer the years of service the higher is the dissatisfaction. If you take for instance employees with 7 - 10 years of work with, 51.6% of them pointed job dissatisfaction as a reason for their resignation. For the Veteran group (11+ years), 48.5% of resignations were due to dissatisfaction.

2 - Are younger employees resigning due to some kind of dissatisfaction? What about older employees?

We uncovered that the age group 61+ had the highest disssatisfaction rate, 52.1% of them left the company due to dissatisfaction. With the expception of the range 26-30, the younger groups' dissatisfaction was below 40%

Comparing both institutes DETE and TAFE: The DETE institute has the highest rate of employees leaving due to job dissatisfaction, 47.9%. For the TAFE institute, only 26.7 former employees pointed that they were leaving due to not being satisfied.

In [ ]: