Job Dissatisfaction and Exit Surveys:

Analyzing age and length of job retention and their association with certain factors

Image

This project aims to first clean exit survey data from the Department of Education, Training, and Employement (DETE) as well as the Technical and Further Education (TAFE) institute. From there, the data will be combined and factors affecting job dissatisfaction will be assessed in order to draw conclusions on what the most important factors were according to the exit surveys.

In [1]:
import pandas as pd
import numpy as np
In [2]:
dete_survey = pd.read_csv('dete_survey.csv', na_values = 'Not Stated') ## this reads 'Not Stated'
                                                                       ## values in as NaN
In [3]:
tafe_survey = pd.read_csv('tafe_survey.csv')
In [4]:
pd.options.display.max_columns = 150
dete_survey.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 822 entries, 0 to 821
Data columns (total 56 columns):
 #   Column                               Non-Null Count  Dtype  
---  ------                               --------------  -----  
 0   ID                                   822 non-null    int64  
 1   SeparationType                       822 non-null    object 
 2   Cease Date                           788 non-null    object 
 3   DETE Start Date                      749 non-null    float64
 4   Role Start Date                      724 non-null    float64
 5   Position                             817 non-null    object 
 6   Classification                       455 non-null    object 
 7   Region                               717 non-null    object 
 8   Business Unit                        126 non-null    object 
 9   Employment Status                    817 non-null    object 
 10  Career move to public sector         822 non-null    bool   
 11  Career move to private sector        822 non-null    bool   
 12  Interpersonal conflicts              822 non-null    bool   
 13  Job dissatisfaction                  822 non-null    bool   
 14  Dissatisfaction with the department  822 non-null    bool   
 15  Physical work environment            822 non-null    bool   
 16  Lack of recognition                  822 non-null    bool   
 17  Lack of job security                 822 non-null    bool   
 18  Work location                        822 non-null    bool   
 19  Employment conditions                822 non-null    bool   
 20  Maternity/family                     822 non-null    bool   
 21  Relocation                           822 non-null    bool   
 22  Study/Travel                         822 non-null    bool   
 23  Ill Health                           822 non-null    bool   
 24  Traumatic incident                   822 non-null    bool   
 25  Work life balance                    822 non-null    bool   
 26  Workload                             822 non-null    bool   
 27  None of the above                    822 non-null    bool   
 28  Professional Development             808 non-null    object 
 29  Opportunities for promotion          735 non-null    object 
 30  Staff morale                         816 non-null    object 
 31  Workplace issue                      788 non-null    object 
 32  Physical environment                 817 non-null    object 
 33  Worklife balance                     815 non-null    object 
 34  Stress and pressure support          810 non-null    object 
 35  Performance of supervisor            813 non-null    object 
 36  Peer support                         812 non-null    object 
 37  Initiative                           813 non-null    object 
 38  Skills                               811 non-null    object 
 39  Coach                                767 non-null    object 
 40  Career Aspirations                   746 non-null    object 
 41  Feedback                             792 non-null    object 
 42  Further PD                           768 non-null    object 
 43  Communication                        814 non-null    object 
 44  My say                               812 non-null    object 
 45  Information                          816 non-null    object 
 46  Kept informed                        813 non-null    object 
 47  Wellness programs                    766 non-null    object 
 48  Health & Safety                      793 non-null    object 
 49  Gender                               798 non-null    object 
 50  Age                                  811 non-null    object 
 51  Aboriginal                           16 non-null     object 
 52  Torres Strait                        3 non-null      object 
 53  South Sea                            7 non-null      object 
 54  Disability                           23 non-null     object 
 55  NESB                                 32 non-null     object 
dtypes: bool(18), float64(2), int64(1), object(35)
memory usage: 258.6+ KB
In [5]:
dete_survey.head()
Out[5]:
ID SeparationType Cease Date DETE Start Date Role Start Date Position Classification Region Business Unit Employment Status Career move to public sector Career move to private sector Interpersonal conflicts Job dissatisfaction Dissatisfaction with the department Physical work environment Lack of recognition Lack of job security Work location Employment conditions Maternity/family Relocation Study/Travel Ill Health Traumatic incident Work life balance Workload None of the above Professional Development Opportunities for promotion Staff morale Workplace issue Physical environment Worklife balance Stress and pressure support Performance of supervisor Peer support Initiative Skills Coach Career Aspirations Feedback Further PD Communication My say Information Kept informed Wellness programs Health & Safety Gender Age Aboriginal Torres Strait South Sea Disability NESB
0 1 Ill Health Retirement 08/2012 1984.0 2004.0 Public Servant A01-A04 Central Office Corporate Strategy and Peformance Permanent Full-time True False False True False False True False False False False False False False False False False True A A N N N A A A A N N N A A A N A A N N N Male 56-60 NaN NaN NaN NaN Yes
1 2 Voluntary Early Retirement (VER) 08/2012 NaN NaN Public Servant AO5-AO7 Central Office Corporate Strategy and Peformance Permanent Full-time False False False False False False False False False False False False False False False False False False A A N N N N A A A N N N A A A N A A N N N Male 56-60 NaN NaN NaN NaN NaN
2 3 Voluntary Early Retirement (VER) 05/2012 2011.0 2011.0 Schools Officer NaN Central Office Education Queensland Permanent Full-time False False False False False False False False False False False False False False False False False True N N N N N N N N N N N N N N N A A N N N N Male 61 or older NaN NaN NaN NaN NaN
3 4 Resignation-Other reasons 05/2012 2005.0 2006.0 Teacher Primary Central Queensland NaN Permanent Full-time False True False False False False False False False False False False False False False False False False A N N N A A N N A A A A A A A A A A A N A Female 36-40 NaN NaN NaN NaN NaN
4 5 Age Retirement 05/2012 1970.0 1989.0 Head of Curriculum/Head of Special Education NaN South East NaN Permanent Full-time False False False False False False False False False False False False False False False True False False A A N N D D N A A A A A A SA SA D D A N A M Female 61 or older NaN NaN NaN NaN NaN
In [6]:
print(dete_survey['SeparationType'].value_counts())


## ↓↓↓↓ Other columns of interest:
# print(dete_survey['DETE Start Date'].value_counts())
# print('\n')
# print(dete_survey['Role Start Date'].value_counts())
# print('\n')
# print(dete_survey['Age'].value_counts())
# print('\n')
# print(dete_survey['Cease Date'].value_counts())
# print('\n')
Age Retirement                          285
Resignation-Other reasons               150
Resignation-Other employer               91
Resignation-Move overseas/interstate     70
Voluntary Early Retirement (VER)         67
Ill Health Retirement                    61
Other                                    49
Contract Expired                         34
Termination                              15
Name: SeparationType, dtype: int64
In [7]:
print(tafe_survey.info())

## ↓↓↓↓ Other columns of interest:
# print(tafe_survey['CurrentAge. Current Age'].value_counts())
# print('\n')
# print(tafe_survey['CESSATION YEAR'].value_counts())
# print('\n')
# print(tafe_survey['LengthofServiceCurrent. Length of Service at current workplace (in years)'].value_counts())
# print('\n')
# print(tafe_survey['LengthofServiceOverall. Overall Length of Service at Institute (in years)'].value_counts())
# print('\n')
# print(tafe_survey['Reason for ceasing employment'].value_counts())
# print('\n')

# tafe_survey.isnull()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 702 entries, 0 to 701
Data columns (total 72 columns):
 #   Column                                                                                                                                                         Non-Null Count  Dtype  
---  ------                                                                                                                                                         --------------  -----  
 0   Record ID                                                                                                                                                      702 non-null    float64
 1   Institute                                                                                                                                                      702 non-null    object 
 2   WorkArea                                                                                                                                                       702 non-null    object 
 3   CESSATION YEAR                                                                                                                                                 695 non-null    float64
 4   Reason for ceasing employment                                                                                                                                  701 non-null    object 
 5   Contributing Factors. Career Move - Public Sector                                                                                                              437 non-null    object 
 6   Contributing Factors. Career Move - Private Sector                                                                                                             437 non-null    object 
 7   Contributing Factors. Career Move - Self-employment                                                                                                            437 non-null    object 
 8   Contributing Factors. Ill Health                                                                                                                               437 non-null    object 
 9   Contributing Factors. Maternity/Family                                                                                                                         437 non-null    object 
 10  Contributing Factors. Dissatisfaction                                                                                                                          437 non-null    object 
 11  Contributing Factors. Job Dissatisfaction                                                                                                                      437 non-null    object 
 12  Contributing Factors. Interpersonal Conflict                                                                                                                   437 non-null    object 
 13  Contributing Factors. Study                                                                                                                                    437 non-null    object 
 14  Contributing Factors. Travel                                                                                                                                   437 non-null    object 
 15  Contributing Factors. Other                                                                                                                                    437 non-null    object 
 16  Contributing Factors. NONE                                                                                                                                     437 non-null    object 
 17  Main Factor. Which of these was the main factor for leaving?                                                                                                   113 non-null    object 
 18  InstituteViews. Topic:1. I feel the senior leadership had a clear vision and direction                                                                         608 non-null    object 
 19  InstituteViews. Topic:2. I was given access to skills training to help me do my job better                                                                     613 non-null    object 
 20  InstituteViews. Topic:3. I was given adequate opportunities for personal development                                                                           610 non-null    object 
 21  InstituteViews. Topic:4. I was given adequate opportunities for promotion within %Institute]Q25LBL%                                                            608 non-null    object 
 22  InstituteViews. Topic:5. I felt the salary for the job was right for the responsibilities I had                                                                615 non-null    object 
 23  InstituteViews. Topic:6. The organisation recognised when staff did good work                                                                                  607 non-null    object 
 24  InstituteViews. Topic:7. Management was generally supportive of me                                                                                             614 non-null    object 
 25  InstituteViews. Topic:8. Management was generally supportive of my team                                                                                        608 non-null    object 
 26  InstituteViews. Topic:9. I was kept informed of the changes in the organisation which would affect me                                                          610 non-null    object 
 27  InstituteViews. Topic:10. Staff morale was positive within the Institute                                                                                       602 non-null    object 
 28  InstituteViews. Topic:11. If I had a workplace issue it was dealt with quickly                                                                                 601 non-null    object 
 29  InstituteViews. Topic:12. If I had a workplace issue it was dealt with efficiently                                                                             597 non-null    object 
 30  InstituteViews. Topic:13. If I had a workplace issue it was dealt with discreetly                                                                              601 non-null    object 
 31  WorkUnitViews. Topic:14. I was satisfied with the quality of the management and supervision within my work unit                                                609 non-null    object 
 32  WorkUnitViews. Topic:15. I worked well with my colleagues                                                                                                      605 non-null    object 
 33  WorkUnitViews. Topic:16. My job was challenging and interesting                                                                                                607 non-null    object 
 34  WorkUnitViews. Topic:17. I was encouraged to use my initiative in the course of my work                                                                        610 non-null    object 
 35  WorkUnitViews. Topic:18. I had sufficient contact with other people in my job                                                                                  613 non-null    object 
 36  WorkUnitViews. Topic:19. I was given adequate support and co-operation by my peers to enable me to do my job                                                   609 non-null    object 
 37  WorkUnitViews. Topic:20. I was able to use the full range of my skills in my job                                                                               609 non-null    object 
 38  WorkUnitViews. Topic:21. I was able to use the full range of my abilities in my job. ; Category:Level of Agreement; Question:YOUR VIEWS ABOUT YOUR WORK UNIT]  608 non-null    object 
 39  WorkUnitViews. Topic:22. I was able to use the full range of my knowledge in my job                                                                            608 non-null    object 
 40  WorkUnitViews. Topic:23. My job provided sufficient variety                                                                                                    611 non-null    object 
 41  WorkUnitViews. Topic:24. I was able to cope with the level of stress and pressure in my job                                                                    610 non-null    object 
 42  WorkUnitViews. Topic:25. My job allowed me to balance the demands of work and family to my satisfaction                                                        611 non-null    object 
 43  WorkUnitViews. Topic:26. My supervisor gave me adequate personal recognition and feedback on my performance                                                    606 non-null    object 
 44  WorkUnitViews. Topic:27. My working environment was satisfactory e.g. sufficient space, good lighting, suitable seating and working area                       610 non-null    object 
 45  WorkUnitViews. Topic:28. I was given the opportunity to mentor and coach others in order for me to pass on my skills and knowledge prior to my cessation date  609 non-null    object 
 46  WorkUnitViews. Topic:29. There was adequate communication between staff in my unit                                                                             603 non-null    object 
 47  WorkUnitViews. Topic:30. Staff morale was positive within my work unit                                                                                         606 non-null    object 
 48  Induction. Did you undertake Workplace Induction?                                                                                                              619 non-null    object 
 49  InductionInfo. Topic:Did you undertake a Corporate Induction?                                                                                                  432 non-null    object 
 50  InductionInfo. Topic:Did you undertake a Institute Induction?                                                                                                  483 non-null    object 
 51  InductionInfo. Topic: Did you undertake Team Induction?                                                                                                        440 non-null    object 
 52  InductionInfo. Face to Face Topic:Did you undertake a Corporate Induction; Category:How it was conducted?                                                      555 non-null    object 
 53  InductionInfo. On-line Topic:Did you undertake a Corporate Induction; Category:How it was conducted?                                                           555 non-null    object 
 54  InductionInfo. Induction Manual Topic:Did you undertake a Corporate Induction?                                                                                 555 non-null    object 
 55  InductionInfo. Face to Face Topic:Did you undertake a Institute Induction?                                                                                     530 non-null    object 
 56  InductionInfo. On-line Topic:Did you undertake a Institute Induction?                                                                                          555 non-null    object 
 57  InductionInfo. Induction Manual Topic:Did you undertake a Institute Induction?                                                                                 553 non-null    object 
 58  InductionInfo. Face to Face Topic: Did you undertake Team Induction; Category?                                                                                 555 non-null    object 
 59  InductionInfo. On-line Topic: Did you undertake Team Induction?process you undertook and how it was conducted.]                                                555 non-null    object 
 60  InductionInfo. Induction Manual Topic: Did you undertake Team Induction?                                                                                       555 non-null    object 
 61  Workplace. Topic:Did you and your Manager develop a Performance and Professional Development Plan (PPDP)?                                                      608 non-null    object 
 62  Workplace. Topic:Does your workplace promote a work culture free from all forms of unlawful discrimination?                                                    594 non-null    object 
 63  Workplace. Topic:Does your workplace promote and practice the principles of employment equity?                                                                 587 non-null    object 
 64  Workplace. Topic:Does your workplace value the diversity of its employees?                                                                                     586 non-null    object 
 65  Workplace. Topic:Would you recommend the Institute as an employer to others?                                                                                   581 non-null    object 
 66  Gender. What is your Gender?                                                                                                                                   596 non-null    object 
 67  CurrentAge. Current Age                                                                                                                                        596 non-null    object 
 68  Employment Type. Employment Type                                                                                                                               596 non-null    object 
 69  Classification. Classification                                                                                                                                 596 non-null    object 
 70  LengthofServiceOverall. Overall Length of Service at Institute (in years)                                                                                      596 non-null    object 
 71  LengthofServiceCurrent. Length of Service at current workplace (in years)                                                                                      596 non-null    object 
dtypes: float64(2), object(70)
memory usage: 395.0+ KB
None
In [8]:
tafe_survey.head()
Out[8]:
Record ID Institute WorkArea CESSATION YEAR Reason for ceasing employment Contributing Factors. Career Move - Public Sector Contributing Factors. Career Move - Private Sector Contributing Factors. Career Move - Self-employment Contributing Factors. Ill Health Contributing Factors. Maternity/Family Contributing Factors. Dissatisfaction Contributing Factors. Job Dissatisfaction Contributing Factors. Interpersonal Conflict Contributing Factors. Study Contributing Factors. Travel Contributing Factors. Other Contributing Factors. NONE Main Factor. Which of these was the main factor for leaving? InstituteViews. Topic:1. I feel the senior leadership had a clear vision and direction InstituteViews. Topic:2. I was given access to skills training to help me do my job better InstituteViews. Topic:3. I was given adequate opportunities for personal development InstituteViews. Topic:4. I was given adequate opportunities for promotion within %Institute]Q25LBL% InstituteViews. Topic:5. I felt the salary for the job was right for the responsibilities I had InstituteViews. Topic:6. The organisation recognised when staff did good work InstituteViews. Topic:7. Management was generally supportive of me InstituteViews. Topic:8. Management was generally supportive of my team InstituteViews. Topic:9. I was kept informed of the changes in the organisation which would affect me InstituteViews. Topic:10. Staff morale was positive within the Institute InstituteViews. Topic:11. If I had a workplace issue it was dealt with quickly InstituteViews. Topic:12. If I had a workplace issue it was dealt with efficiently InstituteViews. Topic:13. If I had a workplace issue it was dealt with discreetly WorkUnitViews. Topic:14. I was satisfied with the quality of the management and supervision within my work unit WorkUnitViews. Topic:15. I worked well with my colleagues WorkUnitViews. Topic:16. My job was challenging and interesting WorkUnitViews. Topic:17. I was encouraged to use my initiative in the course of my work WorkUnitViews. Topic:18. I had sufficient contact with other people in my job WorkUnitViews. Topic:19. I was given adequate support and co-operation by my peers to enable me to do my job WorkUnitViews. Topic:20. I was able to use the full range of my skills in my job WorkUnitViews. Topic:21. I was able to use the full range of my abilities in my job. ; Category:Level of Agreement; Question:YOUR VIEWS ABOUT YOUR WORK UNIT] WorkUnitViews. Topic:22. I was able to use the full range of my knowledge in my job WorkUnitViews. Topic:23. My job provided sufficient variety WorkUnitViews. Topic:24. I was able to cope with the level of stress and pressure in my job WorkUnitViews. Topic:25. My job allowed me to balance the demands of work and family to my satisfaction WorkUnitViews. Topic:26. My supervisor gave me adequate personal recognition and feedback on my performance WorkUnitViews. Topic:27. My working environment was satisfactory e.g. sufficient space, good lighting, suitable seating and working area WorkUnitViews. Topic:28. I was given the opportunity to mentor and coach others in order for me to pass on my skills and knowledge prior to my cessation date WorkUnitViews. Topic:29. There was adequate communication between staff in my unit WorkUnitViews. Topic:30. Staff morale was positive within my work unit Induction. Did you undertake Workplace Induction? InductionInfo. Topic:Did you undertake a Corporate Induction? InductionInfo. Topic:Did you undertake a Institute Induction? InductionInfo. Topic: Did you undertake Team Induction? InductionInfo. Face to Face Topic:Did you undertake a Corporate Induction; Category:How it was conducted? InductionInfo. On-line Topic:Did you undertake a Corporate Induction; Category:How it was conducted? InductionInfo. Induction Manual Topic:Did you undertake a Corporate Induction? InductionInfo. Face to Face Topic:Did you undertake a Institute Induction? InductionInfo. On-line Topic:Did you undertake a Institute Induction? InductionInfo. Induction Manual Topic:Did you undertake a Institute Induction? InductionInfo. Face to Face Topic: Did you undertake Team Induction; Category? InductionInfo. On-line Topic: Did you undertake Team Induction?process you undertook and how it was conducted.] InductionInfo. Induction Manual Topic: Did you undertake Team Induction? Workplace. Topic:Did you and your Manager develop a Performance and Professional Development Plan (PPDP)? Workplace. Topic:Does your workplace promote a work culture free from all forms of unlawful discrimination? Workplace. Topic:Does your workplace promote and practice the principles of employment equity? Workplace. Topic:Does your workplace value the diversity of its employees? Workplace. Topic:Would you recommend the Institute as an employer to others? Gender. What is your Gender? CurrentAge. Current Age Employment Type. Employment Type Classification. Classification LengthofServiceOverall. Overall Length of Service at Institute (in years) LengthofServiceCurrent. Length of Service at current workplace (in years)
0 6.341330e+17 Southern Queensland Institute of TAFE Non-Delivery (corporate) 2010.0 Contract Expired NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN Agree Agree Agree Neutral Agree Agree Agree Agree Agree Agree Agree Agree Agree Agree Agree Agree Strongly Agree Agree Agree Agree Agree Agree Agree Agree Agree Agree Agree Neutral Agree Agree Yes Yes Yes Yes Face to Face - - Face to Face - - Face to Face - - Yes Yes Yes Yes Yes Female 26 30 Temporary Full-time Administration (AO) 1-2 1-2
1 6.341337e+17 Mount Isa Institute of TAFE Non-Delivery (corporate) 2010.0 Retirement - - - - - - - - - Travel - - NaN Agree Agree Agree Agree Agree Strongly Agree Strongly Agree Agree Strongly Agree Agree Agree Agree Disagree Strongly Agree Strongly Agree Strongly Agree Agree Agree Agree Strongly Agree Agree Agree Agree Strongly Agree Agree Strongly Agree Strongly Agree Agree Agree Strongly Agree No NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN Yes Yes Yes Yes Yes NaN NaN NaN NaN NaN NaN
2 6.341388e+17 Mount Isa Institute of TAFE Delivery (teaching) 2010.0 Retirement - - - - - - - - - - - NONE NaN Agree Agree Agree Agree Agree Agree Strongly Agree Agree Agree Agree Agree Neutral Neutral Strongly Agree Strongly Agree Agree Agree Agree Agree Agree Agree Agree Agree Agree Agree Agree Agree Agree Agree Agree No NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN Yes Yes Yes Yes Yes NaN NaN NaN NaN NaN NaN
3 6.341399e+17 Mount Isa Institute of TAFE Non-Delivery (corporate) 2010.0 Resignation - - - - - - - - - Travel - - NaN Agree Agree Agree Agree Agree Agree Agree Agree Agree Agree Agree Agree Agree Strongly Agree Strongly Agree Strongly Agree Strongly Agree Strongly Agree Strongly Agree Strongly Agree Strongly Agree Strongly Agree Strongly Agree Strongly Agree Strongly Agree Strongly Agree Strongly Agree Strongly Agree Strongly Agree Strongly Agree Yes No Yes Yes - - - NaN - - - - - Yes Yes Yes Yes Yes NaN NaN NaN NaN NaN NaN
4 6.341466e+17 Southern Queensland Institute of TAFE Delivery (teaching) 2010.0 Resignation - Career Move - Private Sector - - - - - - - - - - NaN Agree Agree Strongly Agree Agree Strongly Agree Strongly Agree Strongly Agree Strongly Agree Agree Strongly Agree Strongly Agree Agree Strongly Agree Strongly Agree Strongly Agree Strongly Agree Strongly Agree Strongly Agree Strongly Agree Strongly Agree Strongly Agree Strongly Agree Strongly Agree Strongly Agree Strongly Agree Strongly Agree Strongly Agree Strongly Agree Strongly Agree Strongly Agree Yes Yes Yes Yes - - Induction Manual Face to Face - - Face to Face - - Yes Yes Yes Yes Yes Male 41 45 Permanent Full-time Teacher (including LVT) 3-4 3-4

Basic information gathered:

  • Information from both exit surveys include and create an idea for:
    • The age groups of the individuals
    • Amount of time employed
    • When employees started
    • Reasons for leaving

Some important takeaways from above:

Within the DETE Survey:

  • In reason for leaving, some interesting categories to evaluate further are:
    • Resignation-Other reasons
    • Resignation-Other employer
    • Resignation-Move overseas/interstate
  • More individuals left in 2012 than 2013
  • Individuals older than the age of 50 comprise well over half of the individuals leaving
  • Length of time is not calculated explicitly

Within the TAFE Survey:

  • In reason for leaving, some interesting categories to evaluate further are:
    • The categories for TAFE give a more general term 'Resignation' in 'Reason for ceasing employment'
    • The number of Resignations in TAFE relative to all other reasons in Resignations as proportion is much higher when looking at this same metric in DETE.
  • Length of time is explicitly calculated with its own column
  • Like with the DETE survey, Individuals older than the age of 50 comprise well over half of the individuals leaving.
In [9]:
dete_drop = dete_survey.iloc[:,28:49]
dete_survey_updated = dete_survey.drop(dete_drop,axis=1)
dete_survey_updated.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 822 entries, 0 to 821
Data columns (total 35 columns):
 #   Column                               Non-Null Count  Dtype  
---  ------                               --------------  -----  
 0   ID                                   822 non-null    int64  
 1   SeparationType                       822 non-null    object 
 2   Cease Date                           788 non-null    object 
 3   DETE Start Date                      749 non-null    float64
 4   Role Start Date                      724 non-null    float64
 5   Position                             817 non-null    object 
 6   Classification                       455 non-null    object 
 7   Region                               717 non-null    object 
 8   Business Unit                        126 non-null    object 
 9   Employment Status                    817 non-null    object 
 10  Career move to public sector         822 non-null    bool   
 11  Career move to private sector        822 non-null    bool   
 12  Interpersonal conflicts              822 non-null    bool   
 13  Job dissatisfaction                  822 non-null    bool   
 14  Dissatisfaction with the department  822 non-null    bool   
 15  Physical work environment            822 non-null    bool   
 16  Lack of recognition                  822 non-null    bool   
 17  Lack of job security                 822 non-null    bool   
 18  Work location                        822 non-null    bool   
 19  Employment conditions                822 non-null    bool   
 20  Maternity/family                     822 non-null    bool   
 21  Relocation                           822 non-null    bool   
 22  Study/Travel                         822 non-null    bool   
 23  Ill Health                           822 non-null    bool   
 24  Traumatic incident                   822 non-null    bool   
 25  Work life balance                    822 non-null    bool   
 26  Workload                             822 non-null    bool   
 27  None of the above                    822 non-null    bool   
 28  Gender                               798 non-null    object 
 29  Age                                  811 non-null    object 
 30  Aboriginal                           16 non-null     object 
 31  Torres Strait                        3 non-null      object 
 32  South Sea                            7 non-null      object 
 33  Disability                           23 non-null     object 
 34  NESB                                 32 non-null     object 
dtypes: bool(18), float64(2), int64(1), object(14)
memory usage: 123.7+ KB
In [10]:
tafe_drop = tafe_survey.iloc[:,17:66]
tafe_survey_updated = tafe_survey.drop(tafe_drop,axis=1)
tafe_drop.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 702 entries, 0 to 701
Data columns (total 49 columns):
 #   Column                                                                                                                                                         Non-Null Count  Dtype 
---  ------                                                                                                                                                         --------------  ----- 
 0   Main Factor. Which of these was the main factor for leaving?                                                                                                   113 non-null    object
 1   InstituteViews. Topic:1. I feel the senior leadership had a clear vision and direction                                                                         608 non-null    object
 2   InstituteViews. Topic:2. I was given access to skills training to help me do my job better                                                                     613 non-null    object
 3   InstituteViews. Topic:3. I was given adequate opportunities for personal development                                                                           610 non-null    object
 4   InstituteViews. Topic:4. I was given adequate opportunities for promotion within %Institute]Q25LBL%                                                            608 non-null    object
 5   InstituteViews. Topic:5. I felt the salary for the job was right for the responsibilities I had                                                                615 non-null    object
 6   InstituteViews. Topic:6. The organisation recognised when staff did good work                                                                                  607 non-null    object
 7   InstituteViews. Topic:7. Management was generally supportive of me                                                                                             614 non-null    object
 8   InstituteViews. Topic:8. Management was generally supportive of my team                                                                                        608 non-null    object
 9   InstituteViews. Topic:9. I was kept informed of the changes in the organisation which would affect me                                                          610 non-null    object
 10  InstituteViews. Topic:10. Staff morale was positive within the Institute                                                                                       602 non-null    object
 11  InstituteViews. Topic:11. If I had a workplace issue it was dealt with quickly                                                                                 601 non-null    object
 12  InstituteViews. Topic:12. If I had a workplace issue it was dealt with efficiently                                                                             597 non-null    object
 13  InstituteViews. Topic:13. If I had a workplace issue it was dealt with discreetly                                                                              601 non-null    object
 14  WorkUnitViews. Topic:14. I was satisfied with the quality of the management and supervision within my work unit                                                609 non-null    object
 15  WorkUnitViews. Topic:15. I worked well with my colleagues                                                                                                      605 non-null    object
 16  WorkUnitViews. Topic:16. My job was challenging and interesting                                                                                                607 non-null    object
 17  WorkUnitViews. Topic:17. I was encouraged to use my initiative in the course of my work                                                                        610 non-null    object
 18  WorkUnitViews. Topic:18. I had sufficient contact with other people in my job                                                                                  613 non-null    object
 19  WorkUnitViews. Topic:19. I was given adequate support and co-operation by my peers to enable me to do my job                                                   609 non-null    object
 20  WorkUnitViews. Topic:20. I was able to use the full range of my skills in my job                                                                               609 non-null    object
 21  WorkUnitViews. Topic:21. I was able to use the full range of my abilities in my job. ; Category:Level of Agreement; Question:YOUR VIEWS ABOUT YOUR WORK UNIT]  608 non-null    object
 22  WorkUnitViews. Topic:22. I was able to use the full range of my knowledge in my job                                                                            608 non-null    object
 23  WorkUnitViews. Topic:23. My job provided sufficient variety                                                                                                    611 non-null    object
 24  WorkUnitViews. Topic:24. I was able to cope with the level of stress and pressure in my job                                                                    610 non-null    object
 25  WorkUnitViews. Topic:25. My job allowed me to balance the demands of work and family to my satisfaction                                                        611 non-null    object
 26  WorkUnitViews. Topic:26. My supervisor gave me adequate personal recognition and feedback on my performance                                                    606 non-null    object
 27  WorkUnitViews. Topic:27. My working environment was satisfactory e.g. sufficient space, good lighting, suitable seating and working area                       610 non-null    object
 28  WorkUnitViews. Topic:28. I was given the opportunity to mentor and coach others in order for me to pass on my skills and knowledge prior to my cessation date  609 non-null    object
 29  WorkUnitViews. Topic:29. There was adequate communication between staff in my unit                                                                             603 non-null    object
 30  WorkUnitViews. Topic:30. Staff morale was positive within my work unit                                                                                         606 non-null    object
 31  Induction. Did you undertake Workplace Induction?                                                                                                              619 non-null    object
 32  InductionInfo. Topic:Did you undertake a Corporate Induction?                                                                                                  432 non-null    object
 33  InductionInfo. Topic:Did you undertake a Institute Induction?                                                                                                  483 non-null    object
 34  InductionInfo. Topic: Did you undertake Team Induction?                                                                                                        440 non-null    object
 35  InductionInfo. Face to Face Topic:Did you undertake a Corporate Induction; Category:How it was conducted?                                                      555 non-null    object
 36  InductionInfo. On-line Topic:Did you undertake a Corporate Induction; Category:How it was conducted?                                                           555 non-null    object
 37  InductionInfo. Induction Manual Topic:Did you undertake a Corporate Induction?                                                                                 555 non-null    object
 38  InductionInfo. Face to Face Topic:Did you undertake a Institute Induction?                                                                                     530 non-null    object
 39  InductionInfo. On-line Topic:Did you undertake a Institute Induction?                                                                                          555 non-null    object
 40  InductionInfo. Induction Manual Topic:Did you undertake a Institute Induction?                                                                                 553 non-null    object
 41  InductionInfo. Face to Face Topic: Did you undertake Team Induction; Category?                                                                                 555 non-null    object
 42  InductionInfo. On-line Topic: Did you undertake Team Induction?process you undertook and how it was conducted.]                                                555 non-null    object
 43  InductionInfo. Induction Manual Topic: Did you undertake Team Induction?                                                                                       555 non-null    object
 44  Workplace. Topic:Did you and your Manager develop a Performance and Professional Development Plan (PPDP)?                                                      608 non-null    object
 45  Workplace. Topic:Does your workplace promote a work culture free from all forms of unlawful discrimination?                                                    594 non-null    object
 46  Workplace. Topic:Does your workplace promote and practice the principles of employment equity?                                                                 587 non-null    object
 47  Workplace. Topic:Does your workplace value the diversity of its employees?                                                                                     586 non-null    object
 48  Workplace. Topic:Would you recommend the Institute as an employer to others?                                                                                   581 non-null    object
dtypes: object(49)
memory usage: 268.9+ KB

↑↑↑↑ The above code drops unnecessary columns for both TAFE and DETE exit surveys. Only columns with commonalities across both surveys will be further analyzed.

In [11]:
dete_survey_updated.columns
Out[11]:
Index(['ID', 'SeparationType', 'Cease Date', 'DETE Start Date',
       'Role Start Date', 'Position', 'Classification', 'Region',
       'Business Unit', 'Employment Status', 'Career move to public sector',
       'Career move to private sector', 'Interpersonal conflicts',
       'Job dissatisfaction', 'Dissatisfaction with the department',
       'Physical work environment', 'Lack of recognition',
       'Lack of job security', 'Work location', 'Employment conditions',
       'Maternity/family', 'Relocation', 'Study/Travel', 'Ill Health',
       'Traumatic incident', 'Work life balance', 'Workload',
       'None of the above', 'Gender', 'Age', 'Aboriginal', 'Torres Strait',
       'South Sea', 'Disability', 'NESB'],
      dtype='object')
In [12]:
tafe_survey_updated.head()
Out[12]:
Record ID Institute WorkArea CESSATION YEAR Reason for ceasing employment Contributing Factors. Career Move - Public Sector Contributing Factors. Career Move - Private Sector Contributing Factors. Career Move - Self-employment Contributing Factors. Ill Health Contributing Factors. Maternity/Family Contributing Factors. Dissatisfaction Contributing Factors. Job Dissatisfaction Contributing Factors. Interpersonal Conflict Contributing Factors. Study Contributing Factors. Travel Contributing Factors. Other Contributing Factors. NONE Gender. What is your Gender? CurrentAge. Current Age Employment Type. Employment Type Classification. Classification LengthofServiceOverall. Overall Length of Service at Institute (in years) LengthofServiceCurrent. Length of Service at current workplace (in years)
0 6.341330e+17 Southern Queensland Institute of TAFE Non-Delivery (corporate) 2010.0 Contract Expired NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN Female 26 30 Temporary Full-time Administration (AO) 1-2 1-2
1 6.341337e+17 Mount Isa Institute of TAFE Non-Delivery (corporate) 2010.0 Retirement - - - - - - - - - Travel - - NaN NaN NaN NaN NaN NaN
2 6.341388e+17 Mount Isa Institute of TAFE Delivery (teaching) 2010.0 Retirement - - - - - - - - - - - NONE NaN NaN NaN NaN NaN NaN
3 6.341399e+17 Mount Isa Institute of TAFE Non-Delivery (corporate) 2010.0 Resignation - - - - - - - - - Travel - - NaN NaN NaN NaN NaN NaN
4 6.341466e+17 Southern Queensland Institute of TAFE Delivery (teaching) 2010.0 Resignation - Career Move - Private Sector - - - - - - - - - - Male 41 45 Permanent Full-time Teacher (including LVT) 3-4 3-4
In [13]:
dete_survey_updated.columns = dete_survey_updated.columns.str.replace('/','_').str.replace(' ','_').str.lower()
dete_survey_updated.head()
Out[13]:
id separationtype cease_date dete_start_date role_start_date position classification region business_unit employment_status career_move_to_public_sector career_move_to_private_sector interpersonal_conflicts job_dissatisfaction dissatisfaction_with_the_department physical_work_environment lack_of_recognition lack_of_job_security work_location employment_conditions maternity_family relocation study_travel ill_health traumatic_incident work_life_balance workload none_of_the_above gender age aboriginal torres_strait south_sea disability nesb
0 1 Ill Health Retirement 08/2012 1984.0 2004.0 Public Servant A01-A04 Central Office Corporate Strategy and Peformance Permanent Full-time True False False True False False True False False False False False False False False False False True Male 56-60 NaN NaN NaN NaN Yes
1 2 Voluntary Early Retirement (VER) 08/2012 NaN NaN Public Servant AO5-AO7 Central Office Corporate Strategy and Peformance Permanent Full-time False False False False False False False False False False False False False False False False False False Male 56-60 NaN NaN NaN NaN NaN
2 3 Voluntary Early Retirement (VER) 05/2012 2011.0 2011.0 Schools Officer NaN Central Office Education Queensland Permanent Full-time False False False False False False False False False False False False False False False False False True Male 61 or older NaN NaN NaN NaN NaN
3 4 Resignation-Other reasons 05/2012 2005.0 2006.0 Teacher Primary Central Queensland NaN Permanent Full-time False True False False False False False False False False False False False False False False False False Female 36-40 NaN NaN NaN NaN NaN
4 5 Age Retirement 05/2012 1970.0 1989.0 Head of Curriculum/Head of Special Education NaN South East NaN Permanent Full-time False False False False False False False False False False False False False False False True False False Female 61 or older NaN NaN NaN NaN NaN
In [14]:
mapping_info = {'Record ID': 'id',
               'CESSATION YEAR': 'cease_date',
               'Reason for ceasing employment': 'separation_type',
               'Gender. What is your Gender?': 'gender',
               'CurrentAge. Current Age': 'age',
               'Employment Type. Employment Type': 'employment_status',
               'Classification. Classification': 'position',
               'LengthofServiceOverall. Overall Length of Service at Institute (in years)': 'institute_service',
               'LengthofServiceCurrent. Length of Service at current workplace (in years)': 'role_service'}

tafe_survey_updated = tafe_survey_updated.rename(mapping_info,axis=1)
In [15]:
tafe_survey_updated.head()
Out[15]:
id Institute WorkArea cease_date separation_type Contributing Factors. Career Move - Public Sector Contributing Factors. Career Move - Private Sector Contributing Factors. Career Move - Self-employment Contributing Factors. Ill Health Contributing Factors. Maternity/Family Contributing Factors. Dissatisfaction Contributing Factors. Job Dissatisfaction Contributing Factors. Interpersonal Conflict Contributing Factors. Study Contributing Factors. Travel Contributing Factors. Other Contributing Factors. NONE gender age employment_status position institute_service role_service
0 6.341330e+17 Southern Queensland Institute of TAFE Non-Delivery (corporate) 2010.0 Contract Expired NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN Female 26 30 Temporary Full-time Administration (AO) 1-2 1-2
1 6.341337e+17 Mount Isa Institute of TAFE Non-Delivery (corporate) 2010.0 Retirement - - - - - - - - - Travel - - NaN NaN NaN NaN NaN NaN
2 6.341388e+17 Mount Isa Institute of TAFE Delivery (teaching) 2010.0 Retirement - - - - - - - - - - - NONE NaN NaN NaN NaN NaN NaN
3 6.341399e+17 Mount Isa Institute of TAFE Non-Delivery (corporate) 2010.0 Resignation - - - - - - - - - Travel - - NaN NaN NaN NaN NaN NaN
4 6.341466e+17 Southern Queensland Institute of TAFE Delivery (teaching) 2010.0 Resignation - Career Move - Private Sector - - - - - - - - - - Male 41 45 Permanent Full-time Teacher (including LVT) 3-4 3-4
In [16]:
dete_survey_updated['separationtype'].value_counts()
Out[16]:
Age Retirement                          285
Resignation-Other reasons               150
Resignation-Other employer               91
Resignation-Move overseas/interstate     70
Voluntary Early Retirement (VER)         67
Ill Health Retirement                    61
Other                                    49
Contract Expired                         34
Termination                              15
Name: separationtype, dtype: int64
In [17]:
tafe_survey_updated['separation_type'].value_counts()
Out[17]:
Resignation                 340
Contract Expired            127
Retrenchment/ Redundancy    104
Retirement                   82
Transfer                     25
Termination                  23
Name: separation_type, dtype: int64
In [18]:
dete_survey_updated['separationtype'] = dete_survey_updated['separationtype'].str.split('-').str[0]
dete_survey_updated['separationtype'].value_counts()
Out[18]:
Resignation                         311
Age Retirement                      285
Voluntary Early Retirement (VER)     67
Ill Health Retirement                61
Other                                49
Contract Expired                     34
Termination                          15
Name: separationtype, dtype: int64
In [19]:
dete_resignation = dete_survey_updated[dete_survey_updated['separationtype'] == 'Resignation'].copy()

tafe_resignation = tafe_survey_updated[tafe_survey_updated['separation_type']=='Resignation'].copy
In [20]:
tafe_survey_updated = tafe_survey_updated.copy()[tafe_survey_updated['separation_type'] == 'Resignation']
tafe_resignations = tafe_survey_updated

Creating New Dataframes for TAFE and DETE:

For the separation type category (or column) in both surveys, only Resignation can be analyzed as a direct reason for dissatisfaction. Therefore, new data frames are created using only Resignation as the separation type. This should provide a clearer view of job dissatisfaction moving forward.

In [21]:
dete_resignation['cease_date'].value_counts()
Out[21]:
2012       126
2013        74
01/2014     22
12/2013     17
06/2013     14
09/2013     11
11/2013      9
07/2013      9
10/2013      6
08/2013      4
05/2013      2
05/2012      2
2010         1
07/2012      1
07/2006      1
09/2010      1
Name: cease_date, dtype: int64
In [22]:
dete_resignation['cease_date'] = dete_resignation['cease_date'].str.split('/').str[-1]
dete_resignation['cease_date'].value_counts()
Out[22]:
2013    146
2012    129
2014     22
2010      2
2006      1
Name: cease_date, dtype: int64
In [23]:
value_counts_dete = dete_resignation["cease_date"].astype(float).value_counts()
dete_resignation['cease_date'] = dete_resignation['cease_date'].astype(float)
print(value_counts_dete)
2013.0    146
2012.0    129
2014.0     22
2010.0      2
2006.0      1
Name: cease_date, dtype: int64
In [24]:
value_counts_tafe = tafe_resignations["cease_date"].value_counts()
print(value_counts_tafe)
2011.0    116
2012.0     94
2010.0     68
2013.0     55
2009.0      2
Name: cease_date, dtype: int64
In [25]:
value_counts_dete.sort_index(ascending = False)
Out[25]:
2014.0     22
2013.0    146
2012.0    129
2010.0      2
2006.0      1
Name: cease_date, dtype: int64
In [26]:
value_counts_tafe.sort_index(ascending = False)
Out[26]:
2013.0     55
2012.0     94
2011.0    116
2010.0     68
2009.0      2
Name: cease_date, dtype: int64
In [27]:
dete_resignation.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 311 entries, 3 to 821
Data columns (total 35 columns):
 #   Column                               Non-Null Count  Dtype  
---  ------                               --------------  -----  
 0   id                                   311 non-null    int64  
 1   separationtype                       311 non-null    object 
 2   cease_date                           300 non-null    float64
 3   dete_start_date                      283 non-null    float64
 4   role_start_date                      271 non-null    float64
 5   position                             308 non-null    object 
 6   classification                       161 non-null    object 
 7   region                               265 non-null    object 
 8   business_unit                        32 non-null     object 
 9   employment_status                    307 non-null    object 
 10  career_move_to_public_sector         311 non-null    bool   
 11  career_move_to_private_sector        311 non-null    bool   
 12  interpersonal_conflicts              311 non-null    bool   
 13  job_dissatisfaction                  311 non-null    bool   
 14  dissatisfaction_with_the_department  311 non-null    bool   
 15  physical_work_environment            311 non-null    bool   
 16  lack_of_recognition                  311 non-null    bool   
 17  lack_of_job_security                 311 non-null    bool   
 18  work_location                        311 non-null    bool   
 19  employment_conditions                311 non-null    bool   
 20  maternity_family                     311 non-null    bool   
 21  relocation                           311 non-null    bool   
 22  study_travel                         311 non-null    bool   
 23  ill_health                           311 non-null    bool   
 24  traumatic_incident                   311 non-null    bool   
 25  work_life_balance                    311 non-null    bool   
 26  workload                             311 non-null    bool   
 27  none_of_the_above                    311 non-null    bool   
 28  gender                               302 non-null    object 
 29  age                                  306 non-null    object 
 30  aboriginal                           7 non-null      object 
 31  torres_strait                        0 non-null      object 
 32  south_sea                            3 non-null      object 
 33  disability                           8 non-null      object 
 34  nesb                                 9 non-null      object 
dtypes: bool(18), float64(3), int64(1), object(13)
memory usage: 49.2+ KB
In [28]:
%matplotlib inline
dete_resignation.boxplot(column = ['dete_start_date','role_start_date','cease_date']) 
Out[28]:
<matplotlib.axes._subplots.AxesSubplot at 0x7fba5673d3a0>

The boxplot is unusually small because of the outlier at around 250. Clearly no one started in the first millenium so this data point needs to be removed. This type of visualization also helps understand where there may be obvious outliers—instead of a function like value counts where several indices may make it more difficult to pick out an index which is out of place. Below, this single outlier is removed and a more accurate range provides a more interpretable graph.

In [29]:
dete_resignation = dete_resignation[dete_resignation['role_start_date'] != 200.0]

dete_resignation['role_start_date'].value_counts()

dete_resignation.boxplot(column = ['dete_start_date','role_start_date','cease_date']) 
                                                       
Out[29]:
<matplotlib.axes._subplots.AxesSubplot at 0x7fba54923f70>
In [30]:
dete_resignation = dete_resignation[(dete_resignation['cease_date'] != 2006.0) & (dete_resignation['cease_date'] != 2010.0)]

tafe_resignations = tafe_resignations[tafe_resignations['cease_date'] != 2009.0]
In [31]:
dete_resignation['cease_date'].value_counts()
Out[31]:
2013.0    145
2012.0    129
2014.0     22
Name: cease_date, dtype: int64
In [32]:
tafe_resignations['cease_date'].value_counts()
Out[32]:
2011.0    116
2012.0     94
2010.0     68
2013.0     55
Name: cease_date, dtype: int64
In [33]:
dete_resignation['institute_service'] = dete_resignation['cease_date'] - dete_resignation['dete_start_date'] 

In the boxplots above, the interquartile range is in the late 90's to 2010 for DETE start date and mid 2000's to just after 2010 for the role start date. The interquartile range for the role start date occurs on a slightly more delayed time frame. Cease date also has a very tight interquartile range comparatively and so 2006 as well 2009 years were removed.

In [34]:
tafe_resignations['Contributing Factors. Dissatisfaction'].value_counts()
Out[34]:
-                                         275
Contributing Factors. Dissatisfaction      55
Name: Contributing Factors. Dissatisfaction, dtype: int64
In [35]:
tafe_resignations['Contributing Factors. Job Dissatisfaction'].value_counts()
Out[35]:
-                      268
Job Dissatisfaction     62
Name: Contributing Factors. Job Dissatisfaction, dtype: int64
In [36]:
tafe_resignations.head()
Out[36]:
id Institute WorkArea cease_date separation_type Contributing Factors. Career Move - Public Sector Contributing Factors. Career Move - Private Sector Contributing Factors. Career Move - Self-employment Contributing Factors. Ill Health Contributing Factors. Maternity/Family Contributing Factors. Dissatisfaction Contributing Factors. Job Dissatisfaction Contributing Factors. Interpersonal Conflict Contributing Factors. Study Contributing Factors. Travel Contributing Factors. Other Contributing Factors. NONE gender age employment_status position institute_service role_service
3 6.341399e+17 Mount Isa Institute of TAFE Non-Delivery (corporate) 2010.0 Resignation - - - - - - - - - Travel - - NaN NaN NaN NaN NaN NaN
4 6.341466e+17 Southern Queensland Institute of TAFE Delivery (teaching) 2010.0 Resignation - Career Move - Private Sector - - - - - - - - - - Male 41 45 Permanent Full-time Teacher (including LVT) 3-4 3-4
5 6.341475e+17 Southern Queensland Institute of TAFE Delivery (teaching) 2010.0 Resignation - - - - - - - - - - Other - Female 56 or older Contract/casual Teacher (including LVT) 7-10 7-10
6 6.341520e+17 Barrier Reef Institute of TAFE Non-Delivery (corporate) 2010.0 Resignation - Career Move - Private Sector - - Maternity/Family - - - - - Other - Male 20 or younger Temporary Full-time Administration (AO) 3-4 3-4
7 6.341537e+17 Southern Queensland Institute of TAFE Delivery (teaching) 2010.0 Resignation - - - - - - - - - - Other - Male 46 50 Permanent Full-time Teacher (including LVT) 3-4 3-4
In [37]:
dete_resignation.head()
Out[37]:
id separationtype cease_date dete_start_date role_start_date position classification region business_unit employment_status career_move_to_public_sector career_move_to_private_sector interpersonal_conflicts job_dissatisfaction dissatisfaction_with_the_department physical_work_environment lack_of_recognition lack_of_job_security work_location employment_conditions maternity_family relocation study_travel ill_health traumatic_incident work_life_balance workload none_of_the_above gender age aboriginal torres_strait south_sea disability nesb institute_service
3 4 Resignation 2012.0 2005.0 2006.0 Teacher Primary Central Queensland NaN Permanent Full-time False True False False False False False False False False False False False False False False False False Female 36-40 NaN NaN NaN NaN NaN 7.0
5 6 Resignation 2012.0 1994.0 1997.0 Guidance Officer NaN Central Office Education Queensland Permanent Full-time False True False False False False False False False True True False False False False False False False Female 41-45 NaN NaN NaN NaN NaN 18.0
8 9 Resignation 2012.0 2009.0 2009.0 Teacher Secondary North Queensland NaN Permanent Full-time False True False False False False False False False False False False False False False False False False Female 31-35 NaN NaN NaN NaN NaN 3.0
9 10 Resignation 2012.0 1997.0 2008.0 Teacher Aide NaN NaN NaN Permanent Part-time False False True True True False False False False False False False False False False False False False Female 46-50 NaN NaN NaN NaN NaN 15.0
11 12 Resignation 2012.0 2009.0 2009.0 Teacher Secondary Far North Queensland NaN Permanent Full-time False False False False False False False False False False True True False False False False False False Male 31-35 NaN NaN NaN NaN NaN 3.0
In [40]:
def update_vals(val):
    if pd.isnull(val):
        return np.nan
    elif val == '-':
        return False
    else:
        return True

tafe_resignations.copy()['dissatisfied'] = tafe_resignations[['Contributing Factors. Dissatisfaction','Contributing Factors. Job Dissatisfaction']].applymap(update_vals).any(axis=1, skipna=False)    
tafe_resignations_up = tafe_resignations

tafe_resignations['dissatisfied'].value_counts(dropna = False)
Out[40]:
False    239
True      91
NaN        8
Name: dissatisfied, dtype: int64
In [41]:
dete_resignation['dissatisfied'] = dete_resignation[['job_dissatisfaction', 'dissatisfaction_with_the_department','lack_of_recognition','lack_of_job_security','work_location','employment_conditions','work_life_balance','workload']].any(axis=1,skipna=False)

dete_resignation['dissatisfied'] = dete_resignation[['job_dissatisfaction', 'dissatisfaction_with_the_department','lack_of_recognition','lack_of_job_security','work_location','employment_conditions','work_life_balance','workload']]
dete_resignation['dissatisfied'].isnull().sum() ###applymap is not necessary in this circumstance
Out[41]:
0
In [42]:
dete_resignations_up = dete_resignation.copy()

dete_resignations_up['dissatisfied'].value_counts(dropna=False)
Out[42]:
False    266
True      41
Name: dissatisfied, dtype: int64

Using relevant columns to understand dissatisfaction:

  • created 'dissatisfied' column which determined whether or not a client left because some of the original data set's determined columns.
  • Those columns in DETE are: job_dissatisfaction,dissatisfaction_with_the_department, lack_of_recognition,lack_of_job_security,work_location, employment_conditions,work_life_balance, and workload. For the TAFE survey, those columns are: Contributing Factors. Dissatisfaction and Contributing Factors. Job Dissatisfaction
  • defined a function to transform elements in columns to either False, True or NaN
  • dete_resignation df did not have NaN values so code looks different!
  • These columns are chosen to focus and understand on reasons for job dissatisfaction in terms of True and False.
In [43]:
dete_resignations_up.copy()['institute'] = 'DETE'
tafe_resignations_up.copy()['institute'] = 'TAFE'
In [44]:
combined = pd.concat([tafe_resignations_up,dete_resignations_up], ignore_index=True)
In [45]:
combined_updated = combined.dropna(axis=1,thresh=500)
In [46]:
combined_updated.head()
Out[46]:
id cease_date gender age employment_status position institute_service dissatisfied
0 6.341399e+17 2010.0 NaN NaN NaN NaN NaN False
1 6.341466e+17 2010.0 Male 41 45 Permanent Full-time Teacher (including LVT) 3-4 False
2 6.341475e+17 2010.0 Female 56 or older Contract/casual Teacher (including LVT) 7-10 False
3 6.341520e+17 2010.0 Male 20 or younger Temporary Full-time Administration (AO) 3-4 False
4 6.341537e+17 2010.0 Male 46 50 Permanent Full-time Teacher (including LVT) 3-4 False

Dropping null values:

  • Both dataframes are combined keeping all columns from each data frame.
  • For non-null values in a column which are less than 500, this columns is dropped
In [47]:
combined_updated['institute_service'].value_counts(dropna=False)
Out[47]:
NaN                   88
Less than 1 year      73
1-2                   64
3-4                   61
5-6                   33
11-20                 26
5.0                   23
1.0                   22
7-10                  21
0.0                   19
3.0                   19
6.0                   17
4.0                   16
9.0                   14
2.0                   14
7.0                   13
More than 20 years    10
8.0                    8
13.0                   7
15.0                   7
20.0                   7
10.0                   6
12.0                   6
14.0                   6
17.0                   6
22.0                   6
18.0                   5
16.0                   5
24.0                   4
23.0                   4
19.0                   3
39.0                   3
11.0                   3
21.0                   3
32.0                   3
25.0                   2
28.0                   2
30.0                   2
26.0                   2
36.0                   2
34.0                   1
49.0                   1
42.0                   1
41.0                   1
33.0                   1
38.0                   1
35.0                   1
29.0                   1
27.0                   1
31.0                   1
Name: institute_service, dtype: int64
In [50]:
combined_updated.copy()['institute_service_up'] = combined_updated['institute_service'].astype('str').str.extract(r"(\d+)")
combined_updated.copy()['institute_service_up'] = combined_updated['institute_service_up'].astype('float')                                                                                     

combined_updated.copy()['institute_service_up'].value_counts()
Out[50]:
1.0     159
3.0      80
5.0      56
7.0      34
11.0     29
0.0      19
6.0      17
20.0     17
4.0      16
9.0      14
2.0      14
8.0       8
13.0      7
15.0      7
22.0      6
14.0      6
17.0      6
12.0      6
10.0      6
18.0      5
16.0      5
23.0      4
24.0      4
21.0      3
19.0      3
32.0      3
39.0      3
30.0      2
26.0      2
36.0      2
28.0      2
25.0      2
35.0      1
38.0      1
34.0      1
33.0      1
49.0      1
41.0      1
27.0      1
42.0      1
29.0      1
31.0      1
Name: institute_service_up, dtype: int64

Extracting singular year values:

After looking at the frequency for the amount of years using value_counts, the numbers needed to be extracted—either from strings or from ranges. Once those values were extracted, value_counts was used again to evaluate frequencies. While the ranges aren't averaged and the phrases before extraction don't give exact values, it doesn't necessarily mean the data will be misrepresented. For instance, "5-6" years changes to 5 years, however since 5-6 years fits into the 3-6 years category which will be created below, it makes no difference if the code written counts more values towards 5 years than 6 years. Likewise, with an expression like 'More than 20 years', if the threshold for "Veteran" is anyone who's served at the company longer than 11 years, than the code extracting "More than 20 years" as just 20 years, will ultimately be categorized correctly as more than 11 years.

In [53]:
def exp_range(val):
    ## if pd.isnull(val):
        ## return 'NaN'
    
    if val < 3:
        return "New: Less than 3 years"
    elif 3 < val < 6: 
        return "Experienced: 3-6 years"
    elif 7 < val < 10: 
        return "Established: 7-10 years"
    else: 
        return "Veteran 11 or more years"
        
combined_updated.copy()['service_cat'] = combined_updated['institute_service_up'].apply(exp_range)
combined_updated.dropna(axis=0,subset=['service_cat'])
Out[53]:
id cease_date gender age employment_status position institute_service dissatisfied institute_service_up service_cat
0 6.341399e+17 2010.0 NaN NaN NaN NaN NaN False NaN Veteran 11 or more years
1 6.341466e+17 2010.0 Male 41 45 Permanent Full-time Teacher (including LVT) 3-4 False 3.0 Veteran 11 or more years
2 6.341475e+17 2010.0 Female 56 or older Contract/casual Teacher (including LVT) 7-10 False 7.0 Veteran 11 or more years
3 6.341520e+17 2010.0 Male 20 or younger Temporary Full-time Administration (AO) 3-4 False 3.0 Veteran 11 or more years
4 6.341537e+17 2010.0 Male 46 50 Permanent Full-time Teacher (including LVT) 3-4 False 3.0 Veteran 11 or more years
... ... ... ... ... ... ... ... ... ... ...
640 8.100000e+02 2013.0 Female 26-30 Permanent Part-time Teacher Aide 3 False 3.0 Veteran 11 or more years
641 8.170000e+02 2014.0 Male 21-25 Permanent Full-time Teacher 2 False 2.0 New: Less than 3 years
642 8.180000e+02 2014.0 Female 21-25 Permanent Full-time Teacher 2 False 2.0 New: Less than 3 years
643 8.210000e+02 2014.0 Female 31-35 Permanent Full-time Public Servant 5 False 5.0 Experienced: 3-6 years
644 8.230000e+02 2013.0 NaN NaN NaN Teacher Aide NaN False NaN Veteran 11 or more years

645 rows × 10 columns

In [54]:
combined_updated.copy()['dissatisfied'].value_counts(dropna = False)
Out[54]:
False    505
True     132
NaN        8
Name: dissatisfied, dtype: int64
In [55]:
combined_updated.copy()['dissatisfied'] = combined_updated['dissatisfied'].fillna(False)
In [60]:
combined_updated.copy()['dissatisfied'].value_counts()
combined_updated.copy()['dissatisfied'] = combined_updated['dissatisfied'].astype(float)
In [61]:
service_cat_pv = combined_updated.pivot_table(values = 'dissatisfied',index = 'service_cat',margins=False)
print(service_cat_pv)
                          dissatisfied
service_cat                           
Established: 7-10 years       0.227273
Experienced: 3-6 years        0.180556
New: Less than 3 years        0.208333
Veteran 11 or more years      0.210826
In [62]:
import matplotlib.pyplot as plt
%matplotlib inline
service_cat_pv.plot(kind = 'bar', title = 'Average Dissatisfaction Based on time with company', xlim=(0,1), legend = False)

plt.ylabel('percentage dissatisfied')
Out[62]:
Text(0, 0.5, 'percentage dissatisfied')

Final Observations:

The following bar plot indicates all experience levels are dissatisfied within the range of about 18-23%. The most dissatisfied group—based on service time is the Established group (about 23%). The Veteran group (followed very closely by the New group) are second most dissatisfied (~21%). The least dissatisfied was the Experienced group (about 18%). This is perhaps because those who are established are looking to expand their skillset and didn't think they could do that at their current company.