Cleaning and Analyzing Employee Exit Surveys

Introduction

About the Data

The datasets for this project will be taken from the results of exit surveys conducted on employees from two institutes. These are the Department of Education, Training, and Employment (DETE) and the Technical and Further Education (TAFE) institute in Queensland, Australia. The DETE data can be found here and the TAFE data can be found here. The encoding for the CSV files have already been converted to UTF-8 from their original cp1252. The datasets were created on 2014.

About this project

This project will seek to answer the following questions:

  • Are employees who only worked for the institutes for a short period of time resigning due to some kind of dissatisfaction? What about employees who have been there longer?
  • Are younger employees resigning due to some kind of dissatisfaction? What about older employees?

Results for both surveys will be combined to answer the questions.

Results

An employee resigning out of dissatisfaction can happen regardless of their length of service. However, this is more likely to happen to more tenured employees as compared to those who have served for a shorter number of years.

Both younger and older employees are resigning due to some kind of dissatisfaction. However, there is a higher chance of this occurring to middle-aged employees.

Preliminary Steps

Since a data dictionary wasn't provided with the datasets, the columns will be defined using general knowledge.

Some columns from the Department of Education, Training and Employment (DETE) Survey:

Header Description
ID An id used to identify the participant of the survey
SeparationType The reason why the person's employment ended
Cease Date The year or month the person's employment ended
DETE Start Date The year the person began employment with the DETE

Some columns from the Technical and Further Education (TAFE) Survey:

Header Description
Record ID An id used to identify the participant of the survey
Reason for ceasing employment The reason why the person's employment ended
LengthofServiceOverall. Overall Length of Service at Institute (in years) The length of the person's employment (in years)

Importing the Libraries and Reading the CSV Files

In [1]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline

dete_survey = pd.read_csv('dete_survey.csv')
tafe_survey = pd.read_csv('tafe_survey.csv')

Exploring the Datasets

DETE Survey

In [2]:
dete_survey.head()
Out[2]:
ID SeparationType Cease Date DETE Start Date Role Start Date Position Classification Region Business Unit Employment Status ... Kept informed Wellness programs Health & Safety Gender Age Aboriginal Torres Strait South Sea Disability NESB
0 1 Ill Health Retirement 08/2012 1984 2004 Public Servant A01-A04 Central Office Corporate Strategy and Peformance Permanent Full-time ... N N N Male 56-60 NaN NaN NaN NaN Yes
1 2 Voluntary Early Retirement (VER) 08/2012 Not Stated Not Stated Public Servant AO5-AO7 Central Office Corporate Strategy and Peformance Permanent Full-time ... N N N Male 56-60 NaN NaN NaN NaN NaN
2 3 Voluntary Early Retirement (VER) 05/2012 2011 2011 Schools Officer NaN Central Office Education Queensland Permanent Full-time ... N N N Male 61 or older NaN NaN NaN NaN NaN
3 4 Resignation-Other reasons 05/2012 2005 2006 Teacher Primary Central Queensland NaN Permanent Full-time ... A N A Female 36-40 NaN NaN NaN NaN NaN
4 5 Age Retirement 05/2012 1970 1989 Head of Curriculum/Head of Special Education NaN South East NaN Permanent Full-time ... N A M Female 61 or older NaN NaN NaN NaN NaN

5 rows × 56 columns

In [3]:
dete_survey.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 822 entries, 0 to 821
Data columns (total 56 columns):
 #   Column                               Non-Null Count  Dtype 
---  ------                               --------------  ----- 
 0   ID                                   822 non-null    int64 
 1   SeparationType                       822 non-null    object
 2   Cease Date                           822 non-null    object
 3   DETE Start Date                      822 non-null    object
 4   Role Start Date                      822 non-null    object
 5   Position                             817 non-null    object
 6   Classification                       455 non-null    object
 7   Region                               822 non-null    object
 8   Business Unit                        126 non-null    object
 9   Employment Status                    817 non-null    object
 10  Career move to public sector         822 non-null    bool  
 11  Career move to private sector        822 non-null    bool  
 12  Interpersonal conflicts              822 non-null    bool  
 13  Job dissatisfaction                  822 non-null    bool  
 14  Dissatisfaction with the department  822 non-null    bool  
 15  Physical work environment            822 non-null    bool  
 16  Lack of recognition                  822 non-null    bool  
 17  Lack of job security                 822 non-null    bool  
 18  Work location                        822 non-null    bool  
 19  Employment conditions                822 non-null    bool  
 20  Maternity/family                     822 non-null    bool  
 21  Relocation                           822 non-null    bool  
 22  Study/Travel                         822 non-null    bool  
 23  Ill Health                           822 non-null    bool  
 24  Traumatic incident                   822 non-null    bool  
 25  Work life balance                    822 non-null    bool  
 26  Workload                             822 non-null    bool  
 27  None of the above                    822 non-null    bool  
 28  Professional Development             808 non-null    object
 29  Opportunities for promotion          735 non-null    object
 30  Staff morale                         816 non-null    object
 31  Workplace issue                      788 non-null    object
 32  Physical environment                 817 non-null    object
 33  Worklife balance                     815 non-null    object
 34  Stress and pressure support          810 non-null    object
 35  Performance of supervisor            813 non-null    object
 36  Peer support                         812 non-null    object
 37  Initiative                           813 non-null    object
 38  Skills                               811 non-null    object
 39  Coach                                767 non-null    object
 40  Career Aspirations                   746 non-null    object
 41  Feedback                             792 non-null    object
 42  Further PD                           768 non-null    object
 43  Communication                        814 non-null    object
 44  My say                               812 non-null    object
 45  Information                          816 non-null    object
 46  Kept informed                        813 non-null    object
 47  Wellness programs                    766 non-null    object
 48  Health & Safety                      793 non-null    object
 49  Gender                               798 non-null    object
 50  Age                                  811 non-null    object
 51  Aboriginal                           16 non-null     object
 52  Torres Strait                        3 non-null      object
 53  South Sea                            7 non-null      object
 54  Disability                           23 non-null     object
 55  NESB                                 32 non-null     object
dtypes: bool(18), int64(1), object(37)
memory usage: 258.6+ KB

Above, two main data types can be found in the dataset, object and bool. The int64 data type is solely for the ID column.

Most of the columns can be seen to have a majority of non-null entries. All of the bool type columns have complete 822 non-null entries. The last columns have significantly low amounts of non-null entries.

For the DETE Start Date and Role Start Date columns, there are elements which contain Not Stated (index 1) instead of dates but are NOT classified as null objects as can be deduced from the cell above.

The columns names suggest that the questionnaire has questions on three areas:

  • Information on the employee's role (indexes 0 - 9)
  • Actual exit survey questions (indexes 10 -48)
  • Some personal information (indexes 49 - 55)

TAFE Survey

In [4]:
tafe_survey.head()
Out[4]:
Record ID Institute WorkArea CESSATION YEAR Reason for ceasing employment Contributing Factors. Career Move - Public Sector Contributing Factors. Career Move - Private Sector Contributing Factors. Career Move - Self-employment Contributing Factors. Ill Health Contributing Factors. Maternity/Family ... Workplace. Topic:Does your workplace promote a work culture free from all forms of unlawful discrimination? Workplace. Topic:Does your workplace promote and practice the principles of employment equity? Workplace. Topic:Does your workplace value the diversity of its employees? Workplace. Topic:Would you recommend the Institute as an employer to others? Gender. What is your Gender? CurrentAge. Current Age Employment Type. Employment Type Classification. Classification LengthofServiceOverall. Overall Length of Service at Institute (in years) LengthofServiceCurrent. Length of Service at current workplace (in years)
0 6.341330e+17 Southern Queensland Institute of TAFE Non-Delivery (corporate) 2010.0 Contract Expired NaN NaN NaN NaN NaN ... Yes Yes Yes Yes Female 26 30 Temporary Full-time Administration (AO) 1-2 1-2
1 6.341337e+17 Mount Isa Institute of TAFE Non-Delivery (corporate) 2010.0 Retirement - - - - - ... Yes Yes Yes Yes NaN NaN NaN NaN NaN NaN
2 6.341388e+17 Mount Isa Institute of TAFE Delivery (teaching) 2010.0 Retirement - - - - - ... Yes Yes Yes Yes NaN NaN NaN NaN NaN NaN
3 6.341399e+17 Mount Isa Institute of TAFE Non-Delivery (corporate) 2010.0 Resignation - - - - - ... Yes Yes Yes Yes NaN NaN NaN NaN NaN NaN
4 6.341466e+17 Southern Queensland Institute of TAFE Delivery (teaching) 2010.0 Resignation - Career Move - Private Sector - - - ... Yes Yes Yes Yes Male 41 45 Permanent Full-time Teacher (including LVT) 3-4 3-4

5 rows × 72 columns

In [5]:
tafe_survey.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 702 entries, 0 to 701
Data columns (total 72 columns):
 #   Column                                                                                                                                                         Non-Null Count  Dtype  
---  ------                                                                                                                                                         --------------  -----  
 0   Record ID                                                                                                                                                      702 non-null    float64
 1   Institute                                                                                                                                                      702 non-null    object 
 2   WorkArea                                                                                                                                                       702 non-null    object 
 3   CESSATION YEAR                                                                                                                                                 695 non-null    float64
 4   Reason for ceasing employment                                                                                                                                  701 non-null    object 
 5   Contributing Factors. Career Move - Public Sector                                                                                                              437 non-null    object 
 6   Contributing Factors. Career Move - Private Sector                                                                                                             437 non-null    object 
 7   Contributing Factors. Career Move - Self-employment                                                                                                            437 non-null    object 
 8   Contributing Factors. Ill Health                                                                                                                               437 non-null    object 
 9   Contributing Factors. Maternity/Family                                                                                                                         437 non-null    object 
 10  Contributing Factors. Dissatisfaction                                                                                                                          437 non-null    object 
 11  Contributing Factors. Job Dissatisfaction                                                                                                                      437 non-null    object 
 12  Contributing Factors. Interpersonal Conflict                                                                                                                   437 non-null    object 
 13  Contributing Factors. Study                                                                                                                                    437 non-null    object 
 14  Contributing Factors. Travel                                                                                                                                   437 non-null    object 
 15  Contributing Factors. Other                                                                                                                                    437 non-null    object 
 16  Contributing Factors. NONE                                                                                                                                     437 non-null    object 
 17  Main Factor. Which of these was the main factor for leaving?                                                                                                   113 non-null    object 
 18  InstituteViews. Topic:1. I feel the senior leadership had a clear vision and direction                                                                         608 non-null    object 
 19  InstituteViews. Topic:2. I was given access to skills training to help me do my job better                                                                     613 non-null    object 
 20  InstituteViews. Topic:3. I was given adequate opportunities for personal development                                                                           610 non-null    object 
 21  InstituteViews. Topic:4. I was given adequate opportunities for promotion within %Institute]Q25LBL%                                                            608 non-null    object 
 22  InstituteViews. Topic:5. I felt the salary for the job was right for the responsibilities I had                                                                615 non-null    object 
 23  InstituteViews. Topic:6. The organisation recognised when staff did good work                                                                                  607 non-null    object 
 24  InstituteViews. Topic:7. Management was generally supportive of me                                                                                             614 non-null    object 
 25  InstituteViews. Topic:8. Management was generally supportive of my team                                                                                        608 non-null    object 
 26  InstituteViews. Topic:9. I was kept informed of the changes in the organisation which would affect me                                                          610 non-null    object 
 27  InstituteViews. Topic:10. Staff morale was positive within the Institute                                                                                       602 non-null    object 
 28  InstituteViews. Topic:11. If I had a workplace issue it was dealt with quickly                                                                                 601 non-null    object 
 29  InstituteViews. Topic:12. If I had a workplace issue it was dealt with efficiently                                                                             597 non-null    object 
 30  InstituteViews. Topic:13. If I had a workplace issue it was dealt with discreetly                                                                              601 non-null    object 
 31  WorkUnitViews. Topic:14. I was satisfied with the quality of the management and supervision within my work unit                                                609 non-null    object 
 32  WorkUnitViews. Topic:15. I worked well with my colleagues                                                                                                      605 non-null    object 
 33  WorkUnitViews. Topic:16. My job was challenging and interesting                                                                                                607 non-null    object 
 34  WorkUnitViews. Topic:17. I was encouraged to use my initiative in the course of my work                                                                        610 non-null    object 
 35  WorkUnitViews. Topic:18. I had sufficient contact with other people in my job                                                                                  613 non-null    object 
 36  WorkUnitViews. Topic:19. I was given adequate support and co-operation by my peers to enable me to do my job                                                   609 non-null    object 
 37  WorkUnitViews. Topic:20. I was able to use the full range of my skills in my job                                                                               609 non-null    object 
 38  WorkUnitViews. Topic:21. I was able to use the full range of my abilities in my job. ; Category:Level of Agreement; Question:YOUR VIEWS ABOUT YOUR WORK UNIT]  608 non-null    object 
 39  WorkUnitViews. Topic:22. I was able to use the full range of my knowledge in my job                                                                            608 non-null    object 
 40  WorkUnitViews. Topic:23. My job provided sufficient variety                                                                                                    611 non-null    object 
 41  WorkUnitViews. Topic:24. I was able to cope with the level of stress and pressure in my job                                                                    610 non-null    object 
 42  WorkUnitViews. Topic:25. My job allowed me to balance the demands of work and family to my satisfaction                                                        611 non-null    object 
 43  WorkUnitViews. Topic:26. My supervisor gave me adequate personal recognition and feedback on my performance                                                    606 non-null    object 
 44  WorkUnitViews. Topic:27. My working environment was satisfactory e.g. sufficient space, good lighting, suitable seating and working area                       610 non-null    object 
 45  WorkUnitViews. Topic:28. I was given the opportunity to mentor and coach others in order for me to pass on my skills and knowledge prior to my cessation date  609 non-null    object 
 46  WorkUnitViews. Topic:29. There was adequate communication between staff in my unit                                                                             603 non-null    object 
 47  WorkUnitViews. Topic:30. Staff morale was positive within my work unit                                                                                         606 non-null    object 
 48  Induction. Did you undertake Workplace Induction?                                                                                                              619 non-null    object 
 49  InductionInfo. Topic:Did you undertake a Corporate Induction?                                                                                                  432 non-null    object 
 50  InductionInfo. Topic:Did you undertake a Institute Induction?                                                                                                  483 non-null    object 
 51  InductionInfo. Topic: Did you undertake Team Induction?                                                                                                        440 non-null    object 
 52  InductionInfo. Face to Face Topic:Did you undertake a Corporate Induction; Category:How it was conducted?                                                      555 non-null    object 
 53  InductionInfo. On-line Topic:Did you undertake a Corporate Induction; Category:How it was conducted?                                                           555 non-null    object 
 54  InductionInfo. Induction Manual Topic:Did you undertake a Corporate Induction?                                                                                 555 non-null    object 
 55  InductionInfo. Face to Face Topic:Did you undertake a Institute Induction?                                                                                     530 non-null    object 
 56  InductionInfo. On-line Topic:Did you undertake a Institute Induction?                                                                                          555 non-null    object 
 57  InductionInfo. Induction Manual Topic:Did you undertake a Institute Induction?                                                                                 553 non-null    object 
 58  InductionInfo. Face to Face Topic: Did you undertake Team Induction; Category?                                                                                 555 non-null    object 
 59  InductionInfo. On-line Topic: Did you undertake Team Induction?process you undertook and how it was conducted.]                                                555 non-null    object 
 60  InductionInfo. Induction Manual Topic: Did you undertake Team Induction?                                                                                       555 non-null    object 
 61  Workplace. Topic:Did you and your Manager develop a Performance and Professional Development Plan (PPDP)?                                                      608 non-null    object 
 62  Workplace. Topic:Does your workplace promote a work culture free from all forms of unlawful discrimination?                                                    594 non-null    object 
 63  Workplace. Topic:Does your workplace promote and practice the principles of employment equity?                                                                 587 non-null    object 
 64  Workplace. Topic:Does your workplace value the diversity of its employees?                                                                                     586 non-null    object 
 65  Workplace. Topic:Would you recommend the Institute as an employer to others?                                                                                   581 non-null    object 
 66  Gender. What is your Gender?                                                                                                                                   596 non-null    object 
 67  CurrentAge. Current Age                                                                                                                                        596 non-null    object 
 68  Employment Type. Employment Type                                                                                                                               596 non-null    object 
 69  Classification. Classification                                                                                                                                 596 non-null    object 
 70  LengthofServiceOverall. Overall Length of Service at Institute (in years)                                                                                      596 non-null    object 
 71  LengthofServiceCurrent. Length of Service at current workplace (in years)                                                                                      596 non-null    object 
dtypes: float64(2), object(70)
memory usage: 395.0+ KB

The TAFE survey contains more columns than the DETE survey. The column names are also much longer and makes the results of DataFrame.info() harder to read on a small screen.

An advantage of this survey though is that the column names are more specific and make it easier to understand what each column means. The breakdown of survey questions is also more apparent given the headings such as Contributing Factors, InstituteViews, WorkUnitViews, and InductionInfo.

Overall though, the survey can also be broken down into the same three areas as the previous one with a slight difference in the arrangement of the questions.

OVERALL

Going back to the objective of this project, it can be seen that, for both datasets, there are multiple columns which are unnecessary to answer the questions stated at the beginning.

Also, for both datasets, there are similar columns which are named differently on each dataset.

These and other issues will be addressed in the succeeding cleaning process.

Cleaning the Datasets

Identifying Missing Values

As previously mentioned, the DETE Start Date and Role Start Date columns of the DETE survey have elements which contain Not Stated instead of dates. These entries will be converted to NaN by rereading the CSV file and passing Not Stated as an argument for the na_values parameter.

The reread CSV file is reassigned to dete_survey.

In [6]:
dete_survey = pd.read_csv('dete_survey.csv', na_values='Not Stated')
dete_survey.head()
Out[6]:
ID SeparationType Cease Date DETE Start Date Role Start Date Position Classification Region Business Unit Employment Status ... Kept informed Wellness programs Health & Safety Gender Age Aboriginal Torres Strait South Sea Disability NESB
0 1 Ill Health Retirement 08/2012 1984.0 2004.0 Public Servant A01-A04 Central Office Corporate Strategy and Peformance Permanent Full-time ... N N N Male 56-60 NaN NaN NaN NaN Yes
1 2 Voluntary Early Retirement (VER) 08/2012 NaN NaN Public Servant AO5-AO7 Central Office Corporate Strategy and Peformance Permanent Full-time ... N N N Male 56-60 NaN NaN NaN NaN NaN
2 3 Voluntary Early Retirement (VER) 05/2012 2011.0 2011.0 Schools Officer NaN Central Office Education Queensland Permanent Full-time ... N N N Male 61 or older NaN NaN NaN NaN NaN
3 4 Resignation-Other reasons 05/2012 2005.0 2006.0 Teacher Primary Central Queensland NaN Permanent Full-time ... A N A Female 36-40 NaN NaN NaN NaN NaN
4 5 Age Retirement 05/2012 1970.0 1989.0 Head of Curriculum/Head of Special Education NaN South East NaN Permanent Full-time ... N A M Female 61 or older NaN NaN NaN NaN NaN

5 rows × 56 columns

The cell above shows the successful conversion of Not Stated to NaN (index 1).

Dropping Unnecessary Columns

Columns in both datasets that will not be needed for this project's analysis will be dropped using the DataFrame.drop() method.

Initially, for the DETE survey dataset, the columns from Professional Development to Gender will be dropped. This is easily done by indexing the column names from the DataFrame.columns attribute and setting the axis parameter to 1.

The new dataset is assigned to dete_survey_updated.

In [7]:
dete_survey_updated = dete_survey.drop(dete_survey.columns[28:49], axis=1)
dete_survey_updated.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 822 entries, 0 to 821
Data columns (total 35 columns):
 #   Column                               Non-Null Count  Dtype  
---  ------                               --------------  -----  
 0   ID                                   822 non-null    int64  
 1   SeparationType                       822 non-null    object 
 2   Cease Date                           788 non-null    object 
 3   DETE Start Date                      749 non-null    float64
 4   Role Start Date                      724 non-null    float64
 5   Position                             817 non-null    object 
 6   Classification                       455 non-null    object 
 7   Region                               717 non-null    object 
 8   Business Unit                        126 non-null    object 
 9   Employment Status                    817 non-null    object 
 10  Career move to public sector         822 non-null    bool   
 11  Career move to private sector        822 non-null    bool   
 12  Interpersonal conflicts              822 non-null    bool   
 13  Job dissatisfaction                  822 non-null    bool   
 14  Dissatisfaction with the department  822 non-null    bool   
 15  Physical work environment            822 non-null    bool   
 16  Lack of recognition                  822 non-null    bool   
 17  Lack of job security                 822 non-null    bool   
 18  Work location                        822 non-null    bool   
 19  Employment conditions                822 non-null    bool   
 20  Maternity/family                     822 non-null    bool   
 21  Relocation                           822 non-null    bool   
 22  Study/Travel                         822 non-null    bool   
 23  Ill Health                           822 non-null    bool   
 24  Traumatic incident                   822 non-null    bool   
 25  Work life balance                    822 non-null    bool   
 26  Workload                             822 non-null    bool   
 27  None of the above                    822 non-null    bool   
 28  Gender                               798 non-null    object 
 29  Age                                  811 non-null    object 
 30  Aboriginal                           16 non-null     object 
 31  Torres Strait                        3 non-null      object 
 32  South Sea                            7 non-null      object 
 33  Disability                           23 non-null     object 
 34  NESB                                 32 non-null     object 
dtypes: bool(18), float64(2), int64(1), object(14)
memory usage: 123.7+ KB

From 56, there are 35 columns remaining for the DETE survey dataset.

The same dropping process is done on the TAFE survey dataset. Most of the survey questions are removed as they will not make any contribution to this project.

In [8]:
tafe_survey_updated = tafe_survey.drop(tafe_survey.columns[17:66], axis=1)
tafe_survey_updated.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 702 entries, 0 to 701
Data columns (total 23 columns):
 #   Column                                                                     Non-Null Count  Dtype  
---  ------                                                                     --------------  -----  
 0   Record ID                                                                  702 non-null    float64
 1   Institute                                                                  702 non-null    object 
 2   WorkArea                                                                   702 non-null    object 
 3   CESSATION YEAR                                                             695 non-null    float64
 4   Reason for ceasing employment                                              701 non-null    object 
 5   Contributing Factors. Career Move - Public Sector                          437 non-null    object 
 6   Contributing Factors. Career Move - Private Sector                         437 non-null    object 
 7   Contributing Factors. Career Move - Self-employment                        437 non-null    object 
 8   Contributing Factors. Ill Health                                           437 non-null    object 
 9   Contributing Factors. Maternity/Family                                     437 non-null    object 
 10  Contributing Factors. Dissatisfaction                                      437 non-null    object 
 11  Contributing Factors. Job Dissatisfaction                                  437 non-null    object 
 12  Contributing Factors. Interpersonal Conflict                               437 non-null    object 
 13  Contributing Factors. Study                                                437 non-null    object 
 14  Contributing Factors. Travel                                               437 non-null    object 
 15  Contributing Factors. Other                                                437 non-null    object 
 16  Contributing Factors. NONE                                                 437 non-null    object 
 17  Gender. What is your Gender?                                               596 non-null    object 
 18  CurrentAge. Current Age                                                    596 non-null    object 
 19  Employment Type. Employment Type                                           596 non-null    object 
 20  Classification. Classification                                             596 non-null    object 
 21  LengthofServiceOverall. Overall Length of Service at Institute (in years)  596 non-null    object 
 22  LengthofServiceCurrent. Length of Service at current workplace (in years)  596 non-null    object 
dtypes: float64(2), object(21)
memory usage: 126.3+ KB

The number of columns has dropped from 72 to 23.

Cleaning Column Names

All column names for both datasets will be cleaned. Since both datasets will be combined to a form a single dataset, similar columns which are present in both will then be renamed to have the same names. This will allow for the combination process to be done easily. Analysis can then be carried out on the resulting dataset.

First, the column names for dete_survey_updated are printed below to find out how the column names will be cleaned.

In [9]:
dete_survey_updated.columns
Out[9]:
Index(['ID', 'SeparationType', 'Cease Date', 'DETE Start Date',
       'Role Start Date', 'Position', 'Classification', 'Region',
       'Business Unit', 'Employment Status', 'Career move to public sector',
       'Career move to private sector', 'Interpersonal conflicts',
       'Job dissatisfaction', 'Dissatisfaction with the department',
       'Physical work environment', 'Lack of recognition',
       'Lack of job security', 'Work location', 'Employment conditions',
       'Maternity/family', 'Relocation', 'Study/Travel', 'Ill Health',
       'Traumatic incident', 'Work life balance', 'Workload',
       'None of the above', 'Gender', 'Age', 'Aboriginal', 'Torres Strait',
       'South Sea', 'Disability', 'NESB'],
      dtype='object')

Vectorized string methods are chained and called to accomplish the following:

  • Remove any leading and trailing white spaces with str.strip()
  • Replace spaces with underscores with str.replace()
  • Converted all characters to lowercase with str.lower()
In [10]:
dete_survey_updated.columns = dete_survey_updated.columns.str.strip().str.replace(' ', '_').str.lower()

The updated column names are reassigned above to the columns attribute and are printed below.

In [11]:
dete_survey_updated.columns
Out[11]:
Index(['id', 'separationtype', 'cease_date', 'dete_start_date',
       'role_start_date', 'position', 'classification', 'region',
       'business_unit', 'employment_status', 'career_move_to_public_sector',
       'career_move_to_private_sector', 'interpersonal_conflicts',
       'job_dissatisfaction', 'dissatisfaction_with_the_department',
       'physical_work_environment', 'lack_of_recognition',
       'lack_of_job_security', 'work_location', 'employment_conditions',
       'maternity/family', 'relocation', 'study/travel', 'ill_health',
       'traumatic_incident', 'work_life_balance', 'workload',
       'none_of_the_above', 'gender', 'age', 'aboriginal', 'torres_strait',
       'south_sea', 'disability', 'nesb'],
      dtype='object')
In [12]:
tafe_survey_updated.columns
Out[12]:
Index(['Record ID', 'Institute', 'WorkArea', 'CESSATION YEAR',
       'Reason for ceasing employment',
       'Contributing Factors. Career Move - Public Sector ',
       'Contributing Factors. Career Move - Private Sector ',
       'Contributing Factors. Career Move - Self-employment',
       'Contributing Factors. Ill Health',
       'Contributing Factors. Maternity/Family',
       'Contributing Factors. Dissatisfaction',
       'Contributing Factors. Job Dissatisfaction',
       'Contributing Factors. Interpersonal Conflict',
       'Contributing Factors. Study', 'Contributing Factors. Travel',
       'Contributing Factors. Other', 'Contributing Factors. NONE',
       'Gender. What is your Gender?', 'CurrentAge. Current Age',
       'Employment Type. Employment Type', 'Classification. Classification',
       'LengthofServiceOverall. Overall Length of Service at Institute (in years)',
       'LengthofServiceCurrent. Length of Service at current workplace (in years)'],
      dtype='object')

For the column names from the TAFE survey printed above, there are columns similar to those present in the DETE survey. These will simply be renamed to match the cleaned column names present in the DETE survey. The rest of the columns will be dealt with later.

The updated column names are printed after renaming some of them.

In [13]:
col_rename = {'Record ID':'id',
'CESSATION YEAR':'cease_date',
'Reason for ceasing employment':'separationtype',
'Gender. What is your Gender?':'gender',
'CurrentAge. Current Age':'age',
'Employment Type. Employment Type':'employment_status',
'Classification. Classification':'position',
'LengthofServiceOverall. Overall Length of Service at Institute (in years)':'institute_service',
'LengthofServiceCurrent. Length of Service at current workplace (in years)':'role_service'}

tafe_survey_updated = tafe_survey_updated.rename(columns=col_rename)
In [14]:
tafe_survey_updated.columns
Out[14]:
Index(['id', 'Institute', 'WorkArea', 'cease_date', 'separationtype',
       'Contributing Factors. Career Move - Public Sector ',
       'Contributing Factors. Career Move - Private Sector ',
       'Contributing Factors. Career Move - Self-employment',
       'Contributing Factors. Ill Health',
       'Contributing Factors. Maternity/Family',
       'Contributing Factors. Dissatisfaction',
       'Contributing Factors. Job Dissatisfaction',
       'Contributing Factors. Interpersonal Conflict',
       'Contributing Factors. Study', 'Contributing Factors. Travel',
       'Contributing Factors. Other', 'Contributing Factors. NONE', 'gender',
       'age', 'employment_status', 'position', 'institute_service',
       'role_service'],
      dtype='object')

Filtering the Data

More unnecessary data will be removed in this section.

As stated in the introduction, analysis will be conducted on those employees who resigned due to some kind of dissatisfaction. The separationtype column shows the reason for why a person's employment ended. Looking at the summaries of this column for both datasets, it can be observed that there are various reasons aside from the person resigning.

In [15]:
dete_survey_updated['separationtype'].value_counts(dropna=False)
Out[15]:
Age Retirement                          285
Resignation-Other reasons               150
Resignation-Other employer               91
Resignation-Move overseas/interstate     70
Voluntary Early Retirement (VER)         67
Ill Health Retirement                    61
Other                                    49
Contract Expired                         34
Termination                              15
Name: separationtype, dtype: int64
In [16]:
tafe_survey_updated['separationtype'].value_counts(dropna=False)
Out[16]:
Resignation                 340
Contract Expired            127
Retrenchment/ Redundancy    104
Retirement                   82
Transfer                     25
Termination                  23
NaN                           1
Name: separationtype, dtype: int64

All rows which have reasons OTHER than Resignation as the employee's separation type will be removed as they will not be needed. Note that there are three kinds of Resignation for the DETE survey. The separationtype column for this survey will be cleaned first to place all corresponding rows solely under Resignation, without the specifiers.

Below, the specifiers are removed by splitting the string by the dash (-) and getting only the first word, Resignation. The resulting column is then reassigned.

The counts of unique values is shown after and it can be seen that there are now 311 entries for Resignation.

In [17]:
dete_survey_updated['separationtype'] = dete_survey_updated['separationtype'].str.split('-').str[0]
dete_survey_updated['separationtype'].value_counts(dropna=False)
Out[17]:
Resignation                         311
Age Retirement                      285
Voluntary Early Retirement (VER)     67
Ill Health Retirement                61
Other                                49
Contract Expired                     34
Termination                          15
Name: separationtype, dtype: int64

Below, both datasets are filtered with boolean masks to contain only the needed rows. The resulting datasets are each assigned to new variables. The DataFrame.copy() is used to avoid the SettingWithCopyWarning warning. This will be used frequently in the succeding steps. Details on this warning and how to resolve it for various cases can be found here.

In [18]:
dete_resignations = dete_survey_updated[dete_survey_updated['separationtype'] == 'Resignation'].copy()
tafe_resignations = tafe_survey_updated[tafe_survey_updated['separationtype'] == 'Resignation'].copy()

The counts of unique values for the separationtype column for both datasets are printed to verify the filtering that was made.

In [19]:
dete_resignations['separationtype'].value_counts(dropna=False)
Out[19]:
Resignation    311
Name: separationtype, dtype: int64
In [20]:
tafe_resignations['separationtype'].value_counts(dropna=False)
Out[20]:
Resignation    340
Name: separationtype, dtype: int64

Verifying the Data

Since length of stay of an employee is a factor to be considered in this project's analysis, columns containing information on this will be checked if any of them have inconsistencies. This is to avoid coming up with an erroneous or a useless analysis. The length of stay can simply be computed by subtracting the employee's cease date from their start date.

Below, the counts of unique values of the cease_date columns of the DETE survey are shown.

In [21]:
dete_resignations['cease_date'].value_counts(dropna=False)
Out[21]:
2012       126
2013        74
01/2014     22
12/2013     17
06/2013     14
09/2013     11
NaN         11
07/2013      9
11/2013      9
10/2013      6
08/2013      4
05/2013      2
05/2012      2
07/2006      1
09/2010      1
2010         1
07/2012      1
Name: cease_date, dtype: int64

Since the formats of the dates are inconsistent, the column will be cleaned to contain only the years. This is done by splitting the strings by the slash (/) and indexing only the last string, which is the year. After reassigning the resulting columns, the Series.value_counts() method is again called to verify the changes made.

In [22]:
dete_resignations['cease_date'] = dete_resignations['cease_date'].str.split('/').str[-1].astype(float)
In [23]:
dete_resignations['cease_date'].value_counts(dropna=False)
Out[23]:
2013.0    146
2012.0    129
2014.0     22
NaN        11
2010.0      2
2006.0      1
Name: cease_date, dtype: int64

Next, the contents of the dete_start_date column are shown below.

In [24]:
dete_resignations['dete_start_date'].value_counts(dropna=False).sort_index()
Out[24]:
1963.0     1
1971.0     1
1972.0     1
1973.0     1
1974.0     2
1975.0     1
1976.0     2
1977.0     1
1980.0     5
1982.0     1
1983.0     2
1984.0     1
1985.0     3
1986.0     3
1987.0     1
1988.0     4
1989.0     4
1990.0     5
1991.0     4
1992.0     6
1993.0     5
1994.0     6
1995.0     4
1996.0     6
1997.0     5
1998.0     6
1999.0     8
2000.0     9
2001.0     3
2002.0     6
2003.0     6
2004.0    14
2005.0    15
2006.0    13
2007.0    21
2008.0    22
2009.0    13
2010.0    17
2011.0    24
2012.0    21
2013.0    10
NaN       28
Name: dete_start_date, dtype: int64

Given that most people in this field start working while in their 20s, the start date values above seem to be reasonable. Looking at the values below for the age columns of the DETE survey, it can be seen that 23 employees who answered the survey were 61 years old or older. Even if one started work at 1963 and assuming that that person ceased employment between 2006 - 2014, that person could have been in their late 60s to early 70s by the time. This is very possible and that person could be one of the 23.

In [25]:
dete_resignations['age'].value_counts()
Out[25]:
41-45            48
46-50            42
36-40            41
26-30            35
51-55            32
31-35            29
21-25            29
56-60            26
61 or older      23
20 or younger     1
Name: age, dtype: int64

Moving to the TAFE survey, the value counts for the cease_date column are shown below.

In [26]:
tafe_resignations['cease_date'].value_counts(dropna=False)
Out[26]:
2011.0    116
2012.0     94
2010.0     68
2013.0     55
NaN         5
2009.0      2
Name: cease_date, dtype: int64

When looking at the other columns, this survey does not contain one which states the start date of the employee. Luckily, this is not needed as the survey already has a column which states the length of service rendered by the employee to the institute. This is the institute_service column.

In [27]:
tafe_resignations['institute_service'].value_counts(dropna=False)
Out[27]:
Less than 1 year      73
1-2                   64
3-4                   63
NaN                   50
5-6                   33
11-20                 26
7-10                  21
More than 20 years    10
Name: institute_service, dtype: int64

Since the DETE survey dataset does not contain a similar column, a new column containing the same name, institute_service, will be created. The values for this column will be the difference between the cease_date and dete_start_date columns.

Once the new column is created, the value counts are printed and sorted by the years of rendered service.

It may be noticed that the columns for both datasets have different formats with the TAFE survey having categories and the DETE survey having individual values. This will be dealt with later when combining the datasets.

In [28]:
dete_resignations['institute_service'] = dete_resignations['cease_date'] - dete_resignations['dete_start_date']

dete_resignations['institute_service'].value_counts(dropna=False).sort_index()
Out[28]:
0.0     20
1.0     22
2.0     14
3.0     20
4.0     16
5.0     23
6.0     17
7.0     13
8.0      8
9.0     14
10.0     6
11.0     4
12.0     6
13.0     8
14.0     6
15.0     7
16.0     5
17.0     6
18.0     5
19.0     3
20.0     7
21.0     3
22.0     6
23.0     4
24.0     4
25.0     2
26.0     2
27.0     1
28.0     2
29.0     1
30.0     2
31.0     1
32.0     3
33.0     1
34.0     1
35.0     1
36.0     2
38.0     1
39.0     3
41.0     1
42.0     1
49.0     1
NaN     38
Name: institute_service, dtype: int64

Identify Dissatisfied Employees

After the separation type and the length of service, a third factor to be considered in the analysis is the contribution to an employee's resignation, namely, dissatisfaction.

The remaining column names are displayed below to help in selecting which ones can be considered as "dissatisfaction" columns.

In [29]:
tafe_resignations.columns
Out[29]:
Index(['id', 'Institute', 'WorkArea', 'cease_date', 'separationtype',
       'Contributing Factors. Career Move - Public Sector ',
       'Contributing Factors. Career Move - Private Sector ',
       'Contributing Factors. Career Move - Self-employment',
       'Contributing Factors. Ill Health',
       'Contributing Factors. Maternity/Family',
       'Contributing Factors. Dissatisfaction',
       'Contributing Factors. Job Dissatisfaction',
       'Contributing Factors. Interpersonal Conflict',
       'Contributing Factors. Study', 'Contributing Factors. Travel',
       'Contributing Factors. Other', 'Contributing Factors. NONE', 'gender',
       'age', 'employment_status', 'position', 'institute_service',
       'role_service'],
      dtype='object')

For the TAFE survey, the columns entitled Contributing Factors. Dissatisfaction and Contributing Factors. Job Dissatisfaction can easily be classified as "dissatisfaction" columns.

In [30]:
dete_resignations.columns
Out[30]:
Index(['id', 'separationtype', 'cease_date', 'dete_start_date',
       'role_start_date', 'position', 'classification', 'region',
       'business_unit', 'employment_status', 'career_move_to_public_sector',
       'career_move_to_private_sector', 'interpersonal_conflicts',
       'job_dissatisfaction', 'dissatisfaction_with_the_department',
       'physical_work_environment', 'lack_of_recognition',
       'lack_of_job_security', 'work_location', 'employment_conditions',
       'maternity/family', 'relocation', 'study/travel', 'ill_health',
       'traumatic_incident', 'work_life_balance', 'workload',
       'none_of_the_above', 'gender', 'age', 'aboriginal', 'torres_strait',
       'south_sea', 'disability', 'nesb', 'institute_service'],
      dtype='object')

For the DETE survey, the following columns can be considered to fall under the same category:

  • job_dissatisfaction
  • dissatisfaction_with_the_department
  • physical_work_environment
  • lack_of_recognition
  • lack_of_job_security
  • work_location
  • employment_conditions
  • work_life_balance
  • workload

For both surveys, the employees were asked to select from the various options, those situations which applied during their time in the institute or which contributed to the ceasing of their employment. An employee could have chosen more than one "dissatisfaction" option that contributed to their resignation.

To simplify analysis, an employee who resigned will be considered as "dissatisfied" if ANY of the relevant columns were selected and otherwise if none were selected. A new column will be created to contain True for "dissatisfied" employees. To do this, the DataFrame.any() method will be used. For the entire dataset, the method checks whether ANY element of a row contains a True value.

For this to work on the TAFE survey, the values for the relevant columns need to be converted first. The value counts for the two columns are shown below.

In [31]:
tafe_resignations['Contributing Factors. Dissatisfaction'].value_counts(dropna=False)
Out[31]:
-                                         277
Contributing Factors. Dissatisfaction      55
NaN                                         8
Name: Contributing Factors. Dissatisfaction, dtype: int64
In [32]:
tafe_resignations['Contributing Factors. Job Dissatisfaction'].value_counts(dropna=False)
Out[32]:
-                      270
Job Dissatisfaction     62
NaN                      8
Name: Contributing Factors. Job Dissatisfaction, dtype: int64

Below, a function update_vals is created to convert the values for the above columns:

  • - values will be converted to False
  • NaN values will be retained
  • The rest will be converted to True

The DataFrame.applymap() method applies the update_vals function to each element of the relevant columns. The DataFrame.any() method is then chained to check each row. The results are returned and assigned to a new columns, dissatisfied. The value counts of the column are shown after.

In [33]:
def update_vals(val):
    if val == '-':
        return False
    elif pd.isnull(val):
        return np.nan
    else:        
        return True
    
tafe_cols = ['Contributing Factors. Dissatisfaction', 'Contributing Factors. Job Dissatisfaction']
tafe_resignations['dissatisfied'] = tafe_resignations[tafe_cols].applymap(update_vals).any(axis=1, skipna=False)
tafe_resignations['dissatisfied'].value_counts(dropna=False)
Out[33]:
False    241
True      91
True       8
Name: dissatisfied, dtype: int64

For the DETE survey, there is no need to convert the values as they are already in the bool type. The DataFrame.any() method is called on the sliced dataset with the columns being selected by their indexes. The value counts of the new column, dissatisfied, are shown after.

In [34]:
dete_cols_indexes = [13,14,15,16,17,18,19,25,26]

dete_resignations['dissatisfied'] = dete_resignations.iloc[:,dete_cols_indexes].any(axis=1, skipna=False)

dete_resignations['dissatisfied'].value_counts(dropna=False)
Out[34]:
False    162
True     149
Name: dissatisfied, dtype: int64
In [35]:
dete_resignations_up = dete_resignations.copy()
tafe_resignations_up = tafe_resignations.copy()

The results show, per institute, the number of employees who resigned due to some form of dissatisfaction.

Combining the Data

Now that most of the cleaning process has been performed, both datasets can now be combined.

A new column, institute, containing the institute acronyms is created below to help distinguish the rows for each.

In [36]:
dete_resignations_up['institute'] = 'DETE'
tafe_resignations_up ['institute'] = 'TAFE'

The pd.concat() function is used to vertically combine the datasets. Summary information is displayed after for the resulting dataset, combined.

In [37]:
combined = pd.concat([dete_resignations_up,tafe_resignations_up ], ignore_index=True)

combined.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 651 entries, 0 to 650
Data columns (total 53 columns):
 #   Column                                               Non-Null Count  Dtype  
---  ------                                               --------------  -----  
 0   id                                                   651 non-null    float64
 1   separationtype                                       651 non-null    object 
 2   cease_date                                           635 non-null    float64
 3   dete_start_date                                      283 non-null    float64
 4   role_start_date                                      271 non-null    float64
 5   position                                             598 non-null    object 
 6   classification                                       161 non-null    object 
 7   region                                               265 non-null    object 
 8   business_unit                                        32 non-null     object 
 9   employment_status                                    597 non-null    object 
 10  career_move_to_public_sector                         311 non-null    object 
 11  career_move_to_private_sector                        311 non-null    object 
 12  interpersonal_conflicts                              311 non-null    object 
 13  job_dissatisfaction                                  311 non-null    object 
 14  dissatisfaction_with_the_department                  311 non-null    object 
 15  physical_work_environment                            311 non-null    object 
 16  lack_of_recognition                                  311 non-null    object 
 17  lack_of_job_security                                 311 non-null    object 
 18  work_location                                        311 non-null    object 
 19  employment_conditions                                311 non-null    object 
 20  maternity/family                                     311 non-null    object 
 21  relocation                                           311 non-null    object 
 22  study/travel                                         311 non-null    object 
 23  ill_health                                           311 non-null    object 
 24  traumatic_incident                                   311 non-null    object 
 25  work_life_balance                                    311 non-null    object 
 26  workload                                             311 non-null    object 
 27  none_of_the_above                                    311 non-null    object 
 28  gender                                               592 non-null    object 
 29  age                                                  596 non-null    object 
 30  aboriginal                                           7 non-null      object 
 31  torres_strait                                        0 non-null      object 
 32  south_sea                                            3 non-null      object 
 33  disability                                           8 non-null      object 
 34  nesb                                                 9 non-null      object 
 35  institute_service                                    563 non-null    object 
 36  dissatisfied                                         643 non-null    object 
 37  institute                                            651 non-null    object 
 38  Institute                                            340 non-null    object 
 39  WorkArea                                             340 non-null    object 
 40  Contributing Factors. Career Move - Public Sector    332 non-null    object 
 41  Contributing Factors. Career Move - Private Sector   332 non-null    object 
 42  Contributing Factors. Career Move - Self-employment  332 non-null    object 
 43  Contributing Factors. Ill Health                     332 non-null    object 
 44  Contributing Factors. Maternity/Family               332 non-null    object 
 45  Contributing Factors. Dissatisfaction                332 non-null    object 
 46  Contributing Factors. Job Dissatisfaction            332 non-null    object 
 47  Contributing Factors. Interpersonal Conflict         332 non-null    object 
 48  Contributing Factors. Study                          332 non-null    object 
 49  Contributing Factors. Travel                         332 non-null    object 
 50  Contributing Factors. Other                          332 non-null    object 
 51  Contributing Factors. NONE                           332 non-null    object 
 52  role_service                                         290 non-null    object 
dtypes: float64(4), object(49)
memory usage: 269.7+ KB

The columns with over 500 non-null values are those common to both datasets and which have resulted from the previous steps. Those with less than 500 non-null values will be dropped using the DataFrame.dropna() method with thresh set to 500 to indicate this threshold. Summary information on the resulting dataset is then shown after.

In [38]:
combined_updated = combined.dropna(axis=1, thresh=500).copy()
combined_updated.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 651 entries, 0 to 650
Data columns (total 10 columns):
 #   Column             Non-Null Count  Dtype  
---  ------             --------------  -----  
 0   id                 651 non-null    float64
 1   separationtype     651 non-null    object 
 2   cease_date         635 non-null    float64
 3   position           598 non-null    object 
 4   employment_status  597 non-null    object 
 5   gender             592 non-null    object 
 6   age                596 non-null    object 
 7   institute_service  563 non-null    object 
 8   dissatisfied       643 non-null    object 
 9   institute          651 non-null    object 
dtypes: float64(2), object(8)
memory usage: 51.0+ KB

Cleaning the institute_service Column

As observed in section 3.5, the institute_service column has different formats for each survey. The value counts for this column in the combined dataset are shown below.

To conduct meaningful analysis, the values will be converted into categories. This article will serve as the basis for this categorization. The article argues that career stage is more effective than age as a reference for understanding an employee's needs.

The following will be the categories:

  • New: Less than 3 years at a company
  • Experienced: 3-6 years at a company
  • Established: 7-10 years at a company
  • Veteran: 11 or more years at a company
In [39]:
combined_updated['institute_service'].value_counts(dropna=False)
Out[39]:
NaN                   88
Less than 1 year      73
1-2                   64
3-4                   63
5-6                   33
11-20                 26
5.0                   23
1.0                   22
7-10                  21
0.0                   20
3.0                   20
6.0                   17
4.0                   16
2.0                   14
9.0                   14
7.0                   13
More than 20 years    10
8.0                    8
13.0                   8
15.0                   7
20.0                   7
10.0                   6
12.0                   6
22.0                   6
14.0                   6
17.0                   6
16.0                   5
18.0                   5
11.0                   4
23.0                   4
24.0                   4
19.0                   3
39.0                   3
21.0                   3
32.0                   3
28.0                   2
26.0                   2
25.0                   2
30.0                   2
36.0                   2
35.0                   1
41.0                   1
38.0                   1
27.0                   1
34.0                   1
33.0                   1
31.0                   1
42.0                   1
49.0                   1
29.0                   1
Name: institute_service, dtype: int64

Below, the column values are first converted to the string type using the Series.astype() method. The Series.str.extract() method is then used to extract the first number (NOT digit) with the regular expression, r'(\d+)'. The numbers are converted to the float type and assigned to a new column, institute_service_up.

The value counts are printed after.

In [40]:
combined_updated['institute_service_up'] = combined_updated['institute_service'].astype(str).str.extract(r'(\d+)')
combined_updated['institute_service_up'] = combined_updated['institute_service_up'].astype(float)
combined_updated['institute_service_up'].value_counts(dropna=False)
Out[40]:
1.0     159
NaN      88
3.0      83
5.0      56
7.0      34
11.0     30
0.0      20
20.0     17
6.0      17
4.0      16
9.0      14
2.0      14
13.0      8
8.0       8
15.0      7
22.0      6
10.0      6
17.0      6
14.0      6
12.0      6
16.0      5
18.0      5
24.0      4
23.0      4
21.0      3
39.0      3
32.0      3
19.0      3
36.0      2
30.0      2
25.0      2
26.0      2
28.0      2
42.0      1
29.0      1
35.0      1
27.0      1
41.0      1
49.0      1
38.0      1
34.0      1
33.0      1
31.0      1
Name: institute_service_up, dtype: int64

Now that the necessary numbers have been extracted, a function below, categorize_service is created to assign them in their respective categories.

In [41]:
def categorize_service(val):
    if val < 3:
        return 'New'
    elif (val >= 3) & (val <= 6):
        return 'Experienced'
    elif (val >= 7) & (val <= 10):
        return 'Established'
    elif val >= 11:
        return 'Veteran'
    elif pd.isnull(val):
        return np.nan

The Series.apply() method applies the function above to each value in the institute_service_up column. The resulting values are assigned to the service_cat columns. The value counts are printed afterwards.

In [42]:
combined_updated['service_cat'] = combined_updated['institute_service_up'].apply(categorize_service)
combined_updated['service_cat'].value_counts(dropna=False)
Out[42]:
New            193
Experienced    172
Veteran        136
NaN             88
Established     62
Name: service_cat, dtype: int64

Cleaning the age column

The fourth and final factor needed to answer this project's questions is the age of the employee upon resignation. The age column of the combined dataset also shows varying formats, just like the institute_service column, as shown in the value counts below.

The age ranges will also be converted to categories. One way to do this is to categorize by generation (Baby Boomers, Generation X, Millenials, etc.). Unfortunately, the upper and lower ranges, 61 or older and 20 or younger, make it difficult to distinguish between two generations that may fall under a single range (i.e. Millenials and Generation Z in the 20 or younger range).

Instead, the following broadly defined categories will be used:

  • Young Adult: 18-35 years old
  • Middle-aged Adult: 36-55 years old
  • Older Adult: 55 years old and above
In [43]:
combined_updated['age'].value_counts(dropna=False)
Out[43]:
51-55            71
NaN              55
41-45            48
41  45           45
46-50            42
36-40            41
46  50           39
26-30            35
21  25           33
36  40           32
26  30           32
31  35           32
56 or older      29
31-35            29
21-25            29
56-60            26
61 or older      23
20 or younger    10
Name: age, dtype: int64

Below, the column values are first converted to the string type using the Series.astype() method. The Series.str.extract() method is then used to extract the first number (NOT digit) with the regular expression, r'(\d+)'. The numbers are converted to the float type and assigned to a new column, age_up.

The value counts are printed after.

In [44]:
combined_updated['age_up'] = combined_updated['age'].astype(str).str.extract(r'(\d+)')
combined_updated['age_up'] = combined_updated['age_up'].astype(float)
combined_updated['age_up'].value_counts(dropna=False)
Out[44]:
41.0    93
46.0    81
36.0    73
51.0    71
26.0    67
21.0    62
31.0    61
NaN     55
56.0    55
61.0    23
20.0    10
Name: age_up, dtype: int64

Now that the necessary numbers have been extracted, a function below, categorize_age is created to assign them in their respective categories.

In [45]:
def categorize_age(val):
    if (val >= 18) & (val <=35):
        return 'Young Adult'
    elif (val >= 36) & (val <= 55):
        return 'Middle-aged Adult'
    elif val >= 55: 
        return 'Older Adult'
    elif pd.isnull(val):
        return np.nan

The Series.apply() method applies the function above to each value in the age_up column. The resulting values are assigned to the age_cat columns. The value counts are printed afterwards.

In [46]:
combined_updated['age_cat'] = combined_updated['age_up'].apply(categorize_age)
combined_updated['age_cat'].value_counts(dropna=False)
Out[46]:
Middle-aged Adult    318
Young Adult          200
Older Adult           78
NaN                   55
Name: age_cat, dtype: int64

Dropping Non-Contributory Rows

It will be noticed that the service_cat and age_cat columns have NaN values of 88 and 55, respectively. Since it is possible to have NaN values on both columns for a certain row, the combined dataset is filtered below with a boolean mask to confirm this. The first few rows are displayed and the shape attribute is printed afterwards.

In [47]:
combined_updated[(combined_updated['institute_service_up'].isnull())&(combined_updated['age'].isnull())].head()
Out[47]:
id separationtype cease_date position employment_status gender age institute_service dissatisfied institute institute_service_up service_cat age_up age_cat
141 4.060000e+02 Resignation 2012.0 Teacher NaN NaN NaN NaN False DETE NaN NaN NaN NaN
301 8.040000e+02 Resignation 2013.0 Teacher Aide Permanent Part-time NaN NaN NaN False DETE NaN NaN NaN NaN
310 8.230000e+02 Resignation 2013.0 Teacher Aide NaN NaN NaN NaN False DETE NaN NaN NaN NaN
311 6.341399e+17 Resignation 2010.0 NaN NaN NaN NaN NaN False TAFE NaN NaN NaN NaN
322 6.341770e+17 Resignation 2010.0 NaN NaN NaN NaN NaN NaN TAFE NaN NaN NaN NaN
In [48]:
combined_updated[(combined_updated['institute_service_up'].isnull())&(combined_updated['age'].isnull())].shape
Out[48]:
(53, 14)

It can be observed that there are 53 employees who have missing data for both columns. This is about 8% of the total number of employees for the combined dataset. Since this project requires data for both columns, rows with NaN values for both will not be able to contribute to the analysis. Given the limited available data, imputing data would also not be a viable option. These 53 rows will be dropped instead.

Below, filtered dataset is assigned to to_drop. The indexes of the rows are taken using the index attribute and are converted to a list. The DataFrame.drop() method drops those rows corresponding to the taken indexes. The resulting dataset is then reassigned to combined_updated. Summary information is printed to confirm the changes made.

In [49]:
to_drop = combined_updated[(combined_updated['institute_service_up'].isnull())&(combined_updated['age'].isnull())]
combined_updated = combined_updated.drop(list(to_drop.index))
combined_updated.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 598 entries, 0 to 650
Data columns (total 14 columns):
 #   Column                Non-Null Count  Dtype  
---  ------                --------------  -----  
 0   id                    598 non-null    float64
 1   separationtype        598 non-null    object 
 2   cease_date            585 non-null    float64
 3   position              595 non-null    object 
 4   employment_status     596 non-null    object 
 5   gender                592 non-null    object 
 6   age                   596 non-null    object 
 7   institute_service     563 non-null    object 
 8   dissatisfied          598 non-null    object 
 9   institute             598 non-null    object 
 10  institute_service_up  563 non-null    float64
 11  service_cat           563 non-null    object 
 12  age_up                596 non-null    float64
 13  age_cat               596 non-null    object 
dtypes: float64(4), object(10)
memory usage: 70.1+ KB

From 651, there are now 598 entries remaining.

Analysis

With an acceptable cleaning process done, analysis can now be conducted to answer the questions posed in the Introduction.

Dissatisfaction Based on Length of Service

In [50]:
combined_updated['service_cat'].value_counts(dropna=False)
Out[50]:
New            193
Experienced    172
Veteran        136
Established     62
NaN             35
Name: service_cat, dtype: int64
In [51]:
combined_updated['dissatisfied'].value_counts(dropna=False)
Out[51]:
False    372
True     226
Name: dissatisfied, dtype: int64
In [53]:
combined_updated['dissatisfied'] = combined_updated['dissatisfied'].fillna(False)

Below, a pivot table is created to contain the percentages of dissatisfied employees by service category.

In [55]:
dissatisfied_service = combined_updated.pivot_table(values='dissatisfied',index='service_cat',margins=True)
dissatisfied_service.reset_index(inplace=True)
dissatisfied_service['dissatisfied_perc'] = dissatisfied_service['dissatisfied'] * 100
dissatisfied_service
Out[55]:
service_cat dissatisfied dissatisfied_perc
0 Established 0.516129 51.612903
1 Experienced 0.343023 34.302326
2 New 0.295337 29.533679
3 Veteran 0.485294 48.529412
4 All 0.380107 38.010657

A bar plot visualizing the percentages in the table above is created using Seaborn.

In [55]:
fig, ax = plt.subplots(figsize=(8,6))
sns.set_style('white')
sns.barplot(x=dissatisfied_service['service_cat'], y=dissatisfied_service['dissatisfied_perc'])
sns.despine(left=True, bottom=True)
plt.title('Dissatisfied Employess by Service Category', fontsize=14)
plt.tick_params(left=False, bottom=False)
plt.ylabel('Dissatisfied Employees, %', fontsize=12)
plt.xlabel('Service Category', fontsize=12)
plt.show()

It can be observed that there is a higher percentage of employees who have worked longer in the institutions who resigned due to some sort of dissatisfaction. This is shown by the categories Established and Veteran. This means that more tenured employees are likely to experience some sort of dissatisfaction that will lead to their resignation.

Dissatisfaction Based on Age

In [56]:
combined_updated['age_cat'].value_counts(dropna=False)
Out[56]:
Middle-aged Adult    318
Young Adult          200
Older Adult           78
NaN                    2
Name: age_cat, dtype: int64

Below, a pivot table is created to contain the percentages of dissatisfied employees by age category.

In [57]:
dissatisfied_age = combined_updated.pivot_table(values='dissatisfied',index='age_cat',margins=True)
dissatisfied_age.reset_index(inplace=True)
dissatisfied_age['dissatisfied_perc'] = dissatisfied_service['dissatisfied'] * 100
dissatisfied_age
Out[57]:
age_cat dissatisfied dissatisfied_perc
0 Middle-aged Adult 0.380503 51.612903
1 Older Adult 0.423077 34.302326
2 Young Adult 0.360000 29.533679
3 All 0.379195 48.529412

A bar plot visualizing the percentages in the table above is created using Seaborn.

In [58]:
fig, ax = plt.subplots(figsize=(7,6))
sns.set_style('white')
sns.barplot(x=dissatisfied_age['age_cat'], y=dissatisfied_age['dissatisfied_perc'])
sns.despine(left=True, bottom=True)
plt.title('Dissatisfied Employess by Age Category', fontsize=14)
plt.tick_params(left=False, bottom=False)
plt.ylabel('Dissatisfied Employees, %', fontsize=12)
plt.xlabel('Age Category', fontsize=12)
plt.show()

It can be observed that the Middle-aged Adult category has the highest percentage of employees that resigned due to some kind of dissatisfaction. This means that employees aged 36-55 are more likely to resign due to some kind of dissatisfaction. Younger adults and older adults are less likely to do this.

Conclusion

From the results of the analysis, it was observed that all categories for the employees' length of service show employees that resigned due to some sort of dissatisfaction. An employee resigning out of dissatisfaction can happen regardless of their length of service. However, this is more likely to happen to more tenured employees as compared to those who have served for a shorter number of years.

An employee can also resign out of dissatisfaction regardless of age group. However, there is a higher chance of this occurring to employees aged 36-55, those falling in the Middle-aged Adult category.