Notebook

Troubleshoot HDFS kerberos connections¶

Dependencies¶

This Notebook requires below installations.

Application	Required	Description
java	Yes	Java 7 or higher is required to run this tool. Please install it from (Java installation)
java classpath setup	Yes	If java is not setup in classpath please follow instruction Here to setup.

Use this tool to test if the Polybase kerberos connection to your kerberos secure HDFS is successfull. This notebook requires about 40 MB of disk space. Learn more about HDFS and polybase kerberos troubleshooting.

Important Instructions¶

Before you begin, you will need:¶

HDFS cluster name
Kerberos username
Kerberos password
Kerberos realm
kerberos KDC name or IP address

This independent tool provide complete details of secure connections which helps administrator to trouble shoot the kerberos connection issue. (Note: This tool does not require Polybase feature and it does not depend on any SQL server components.)

Function definition¶

(no user input required)¶

In [11]:

def PromptForInfo(promptMsg, errorMsg):
    
    promptResponse = input(promptMsg)
    if promptResponse == "":
        raise SystemExit(errorMsg + '\n')
    return promptResponse

def executeCommand(cmd, successMsgs, printMsg):
    print(printMsg)
    cmdOutput = !{cmd}
    cmdOutput = '\n'.join(cmdOutput)
    if any(msg in cmdOutput for msg in successMsgs):
        print(f"\nKerberos Success ")
    else:
        if any(msg in cmdOutput for msg in ['java.net.SocketTimeoutException: Receive timed out']):
            raise SystemExit(f'\nFailed due to wrong kdc hostname: Please check the kdc name and rerun the notebook after changing kdc hostname.\n' + cmd + '\n output:\n' + cmdOutput)
        if  any(msg in cmdOutput for msg in ['Client not found in Kerberos database (6) - CLIENT_NOT_FOUND']):
            raise SystemExit(f'\nThe admin Service Principal supplied does not exist in the realm. Please check if user exist in kerberos realm.\n' + cmd + '\n output:\n' + cmdOutput)
        if  any(msg in cmdOutput for msg in ['Checksum failed']):
            raise SystemExit(f'\n Bad password. Please check the password and re run the test with right password.\n' + cmd + '\n output:\n' + cmdOutput)
        if  any(msg in cmdOutput for msg in ['java.io.FileNotFoundException File does not exist']):
            raise SystemExit(f'\n specified file location does not exist on hdfs. change the location.\n' + cmd + '\n output:\n' + cmdOutput)
        if any(msg in cmdOutput for msg in ['not recongnized']):
            raise SystemExit(f'\n Java was not installed. Please check pre requisits and rerun this cell.\n' + cmd + '\n output:\n' + cmdOutput)
        if any(msg in cmdOutput for msg in ['java.net.ConnectException: Connection refused']):
            raise SystemExit(f'\n Port number is not correct. Please check hdfs connection port number is active.\n' + cmd + '\n output:\n' + cmdOutput)
        raise SystemExit(f'\nFailed during:\n\n\t{cmd}\n\nreturned: \n' + ''.join(cmdOutput) + '.\n')

Check if Java installed.¶

In [12]:

cmd = f'java -version'
executeCommand(cmd, ['java version', 'build'], 'Checking whether java installed or not.')

Checking whether java installed or not.

Success >> java -version output:java version "1.8.0_221"
Java(TM) SE Runtime Environment (build 1.8.0_221-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.221-b11, mixed mode)

Enter hdfs hostname: (or IP address)

In [13]:

# Prompt user inputs:
hdfs_node_name = PromptForInfo('Please provide your cluster hdfs node IP address or host name ', 'Cluster Name node name is required!')

enter HDFS port number: Usually its 8020.

In [14]:

port_number = PromptForInfo('Please provide your Cluster hdfs listening port number', 'port number is required!')

Enter Kebreros realm name: upper case only (example: EXAMPLE.COM)

In [15]:

kerberos_realm = PromptForInfo('Please provide your Kerberos realm(UPPER CASE) ', 'Kerberos realm is required!')

Enter KDC server name/IP address (You can find this information in Ambari or hortonworks )

In [16]:

kerberos_kdc = PromptForInfo('Please provide your Kerberos KDC host name or IP ', 'kdc is required!')

Enter kerberos username: (Example: pdw_user)

In [21]:

kerberos_username = PromptForInfo('Please provide your Kerberos Username for login: ', 'Kerberos Username is required!')

Enter the file name which contains kerberos password. (You can edit the myKerbPass.txt in current working directory with password and provide myKerbPass.txt as input or enter complete path to the text file where password is stored.)

In [18]:

kerberos_password_file = PromptForInfo('Kerberos Password file name: ', 'Kerberos password file name is required!')

Run the test

In [22]:

cmd = f'java -classpath hdfs-kerberos-tester-0.0.1-jar-with-dependencies.jar com.microsoft.kerberos.client.HdfsBridge ' + hdfs_node_name + ' ' + port_number + ' ' + kerberos_realm + ' ' + kerberos_kdc+ ' '+ kerberos_username + ' ' + kerberos_password_file

executeCommand(cmd, ['CHECKPOINT 2 COMPLETED'], 'Testing Kebreros Connection')

Testing Kebreros Connection

An exception has occurred, use %tb to see the full traceback.

SystemExit: 
The admin Service Principal supplied does not exist in the realm. Please check if user exisit in kerberos realm.
java -classpath hdfs-kerberos-tester-0.0.1-jar-with-dependencies.jar com.microsoft.kerberos.client.HdfsBridge 10.193.27.232 8020 APSHDPMS.COM 10.193.27.236 pdw_user1 myKerbPass.txt
 output:
Current working directory : f:\azure-data-studio\notebooks-python
0 [main] INFO com.microsoft.kerberos.client.HdfsBridge  - Hostname: 10.193.27.232
2 [main] INFO com.microsoft.kerberos.client.HdfsBridge  - Port: 8020
3 [main] INFO com.microsoft.kerberos.client.HdfsBridge  - Kerberos Principal: pdw_user1
3 [main] INFO com.microsoft.kerberos.client.HdfsBridge  - Kerberos Password File: myKerbPass.txt
3 [main] INFO com.microsoft.kerberos.client.HdfsBridge  - CHECKPOINT 1 STARTING: Authentication against KDC server
3 [main] INFO com.microsoft.kerberos.client.HdfsBridge  - Filepath to test: /
805 [main] INFO org.apache.hadoop.conf.Configuration.deprecation  - fs.default.name is deprecated. Instead, use fs.defaultFS
806 [main] INFO com.microsoft.kerberos.client.HdfsBridge  - Starting to connect to external storage...
Java config name: null
Native config name: C:\windows\krb5.ini
>>> KdcAccessibility: reset
Using builtin default etypes for default_tkt_enctypes
default etypes for default_tkt_enctypes: 18 17 16 23.
>>> KrbAsReq creating message
>>> KrbKdcReq send: kdc=10.193.27.236 UDP:88, timeout=30000, number of retries =3, #bytes=148
>>> KDCCommunication: kdc=10.193.27.236 UDP:88, timeout=30000,Attempt =1, #bytes=148
>>> KrbKdcReq send: #bytes read=177
>>> KdcAccessibility: remove 10.193.27.236
>>> KDCRep: init() encoding tag is 126 req type is 11
>>>KRBError:
	 cTime is Sat Apr 11 16:40:47 PDT 2015 1428795647000
	 sTime is Thu Sep 05 11:14:55 PDT 2019 1567707295000
	 suSec is 863876
	 error code is 6
	 error Message is Client not found in Kerberos database
	 cname is pdw_user1@APSHDPMS.COM
	 sname is krbtgt/APSHDPMS.COM@APSHDPMS.COM
	 msgType is 30
KRBError received: CLIENT_NOT_FOUND
javax.security.auth.login.LoginException: Client not found in Kerberos database (6) - CLIENT_NOT_FOUND
	at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Unknown Source)
	at com.sun.security.auth.module.Krb5LoginModule.login(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at javax.security.auth.login.LoginContext.invoke(Unknown Source)
	at javax.security.auth.login.LoginContext.access$000(Unknown Source)
	at javax.security.auth.login.LoginContext$4.run(Unknown Source)
	at javax.security.auth.login.LoginContext$4.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.login.LoginContext.invokePriv(Unknown Source)
	at javax.security.auth.login.LoginContext.login(Unknown Source)
	at com.microsoft.kerberos.client.KerberosSecureLogin.secureLogin(KerberosSecureLogin.java:89)
	at com.microsoft.kerberos.client.HdfsBridge.kerberosLoginHelper(HdfsBridge.java:159)
	at com.microsoft.kerberos.client.HdfsBridge.connect(HdfsBridge.java:305)
	at com.microsoft.kerberos.client.HdfsBridge.main(HdfsBridge.java:497)
Caused by: KrbException: Client not found in Kerberos database (6) - CLIENT_NOT_FOUND
	at sun.security.krb5.KrbAsRep.<init>(Unknown Source)
	at sun.security.krb5.KrbAsReqBuilder.send(Unknown Source)
	at sun.security.krb5.KrbAsReqBuilder.action(Unknown Source)
	... 17 more
Caused by: KrbException: Identifier doesn't match expected value (906)
	at sun.security.krb5.internal.KDCRep.init(Unknown Source)
	at sun.security.krb5.internal.ASRep.init(Unknown Source)
	at sun.security.krb5.internal.ASRep.<init>(Unknown Source)
	... 20 more
2056 [main] WARN com.microsoft.kerberos.client.HdfsBridgeException  - Kerberos login failed (LoginContext.login) with provided principal/password.
javax.security.auth.login.LoginException: Client not found in Kerberos database (6) - CLIENT_NOT_FOUND
	at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Unknown Source)
	at com.sun.security.auth.module.Krb5LoginModule.login(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at javax.security.auth.login.LoginContext.invoke(Unknown Source)
	at javax.security.auth.login.LoginContext.access$000(Unknown Source)
	at javax.security.auth.login.LoginContext$4.run(Unknown Source)
	at javax.security.auth.login.LoginContext$4.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.login.LoginContext.invokePriv(Unknown Source)
	at javax.security.auth.login.LoginContext.login(Unknown Source)
	at com.microsoft.kerberos.client.KerberosSecureLogin.secureLogin(KerberosSecureLogin.java:89)
	at com.microsoft.kerberos.client.HdfsBridge.kerberosLoginHelper(HdfsBridge.java:159)
	at com.microsoft.kerberos.client.HdfsBridge.connect(HdfsBridge.java:305)
	at com.microsoft.kerberos.client.HdfsBridge.main(HdfsBridge.java:497)
Caused by: KrbException: Client not found in Kerberos database (6) - CLIENT_NOT_FOUND
	at sun.security.krb5.KrbAsRep.<init>(Unknown Source)
	at sun.security.krb5.KrbAsReqBuilder.send(Unknown Source)
	at sun.security.krb5.KrbAsReqBuilder.action(Unknown Source)
	... 17 more
Caused by: KrbException: Identifier doesn't match expected value (906)
	at sun.security.krb5.internal.KDCRep.init(Unknown Source)
	at sun.security.krb5.internal.ASRep.init(Unknown Source)
	at sun.security.krb5.internal.ASRep.<init>(Unknown Source)
	... 20 more
2062 [main] WARN com.microsoft.kerberos.client.HdfsBridge  - Exception at connect
com.microsoft.kerberos.client.HdfsBridgeException: Kerberos login failed (LoginContext.login) with provided principal/password.
	at com.microsoft.kerberos.client.HdfsBridge.kerberosLoginHelper(HdfsBridge.java:164)
	at com.microsoft.kerberos.client.HdfsBridge.connect(HdfsBridge.java:305)
	at com.microsoft.kerberos.client.HdfsBridge.main(HdfsBridge.java:497)
Caused by: javax.security.auth.login.LoginException: Client not found in Kerberos database (6) - CLIENT_NOT_FOUND
	at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Unknown Source)
	at com.sun.security.auth.module.Krb5LoginModule.login(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at javax.security.auth.login.LoginContext.invoke(Unknown Source)
	at javax.security.auth.login.LoginContext.access$000(Unknown Source)
	at javax.security.auth.login.LoginContext$4.run(Unknown Source)
	at javax.security.auth.login.LoginContext$4.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.login.LoginContext.invokePriv(Unknown Source)
	at javax.security.auth.login.LoginContext.login(Unknown Source)
	at com.microsoft.kerberos.client.KerberosSecureLogin.secureLogin(KerberosSecureLogin.java:89)
	at com.microsoft.kerberos.client.HdfsBridge.kerberosLoginHelper(HdfsBridge.java:159)
	... 2 more
Caused by: KrbException: Client not found in Kerberos database (6) - CLIENT_NOT_FOUND
	at sun.security.krb5.KrbAsRep.<init>(Unknown Source)
	at sun.security.krb5.KrbAsReqBuilder.send(Unknown Source)
	at sun.security.krb5.KrbAsReqBuilder.action(Unknown Source)
	... 17 more
Caused by: KrbException: Identifier doesn't match expected value (906)
	at sun.security.krb5.internal.KDCRep.init(Unknown Source)
	at sun.security.krb5.internal.ASRep.init(Unknown Source)
	at sun.security.krb5.internal.ASRep.<init>(Unknown Source)
	... 20 more
2064 [main] WARN com.microsoft.kerberos.client.HdfsBridgeException  - Exception at connect
com.microsoft.kerberos.client.HdfsBridgeException: Kerberos login failed (LoginContext.login) with provided principal/password.
	at com.microsoft.kerberos.client.HdfsBridge.kerberosLoginHelper(HdfsBridge.java:164)
	at com.microsoft.kerberos.client.HdfsBridge.connect(HdfsBridge.java:305)
	at com.microsoft.kerberos.client.HdfsBridge.main(HdfsBridge.java:497)
Caused by: javax.security.auth.login.LoginException: Client not found in Kerberos database (6) - CLIENT_NOT_FOUND
	at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Unknown Source)
	at com.sun.security.auth.module.Krb5LoginModule.login(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at javax.security.auth.login.LoginContext.invoke(Unknown Source)
	at javax.security.auth.login.LoginContext.access$000(Unknown Source)
	at javax.security.auth.login.LoginContext$4.run(Unknown Source)
	at javax.security.auth.login.LoginContext$4.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.login.LoginContext.invokePriv(Unknown Source)
	at javax.security.auth.login.LoginContext.login(Unknown Source)
	at com.microsoft.kerberos.client.KerberosSecureLogin.secureLogin(KerberosSecureLogin.java:89)
	at com.microsoft.kerberos.client.HdfsBridge.kerberosLoginHelper(HdfsBridge.java:159)
	... 2 more
Caused by: KrbException: Client not found in Kerberos database (6) - CLIENT_NOT_FOUND
	at sun.security.krb5.KrbAsRep.<init>(Unknown Source)
	at sun.security.krb5.KrbAsReqBuilder.send(Unknown Source)
	at sun.security.krb5.KrbAsReqBuilder.action(Unknown Source)
	... 17 more
Caused by: KrbException: Identifier doesn't match expected value (906)
	at sun.security.krb5.internal.KDCRep.init(Unknown Source)
	at sun.security.krb5.internal.ASRep.init(Unknown Source)
	at sun.security.krb5.internal.ASRep.<init>(Unknown Source)
	... 20 more
2066 [main] ERROR com.microsoft.kerberos.client.HdfsBridge  - exception
com.microsoft.kerberos.client.HdfsBridgeException: Exception at connect
	at com.microsoft.kerberos.client.HdfsBridge.connect(HdfsBridge.java:326)
	at com.microsoft.kerberos.client.HdfsBridge.main(HdfsBridge.java:497)
Caused by: com.microsoft.kerberos.client.HdfsBridgeException: Kerberos login failed (LoginContext.login) with provided principal/password.
	at com.microsoft.kerberos.client.HdfsBridge.kerberosLoginHelper(HdfsBridge.java:164)
	at com.microsoft.kerberos.client.HdfsBridge.connect(HdfsBridge.java:305)
	... 1 more
Caused by: javax.security.auth.login.LoginException: Client not found in Kerberos database (6) - CLIENT_NOT_FOUND
	at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Unknown Source)
	at com.sun.security.auth.module.Krb5LoginModule.login(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at javax.security.auth.login.LoginContext.invoke(Unknown Source)
	at javax.security.auth.login.LoginContext.access$000(Unknown Source)
	at javax.security.auth.login.LoginContext$4.run(Unknown Source)
	at javax.security.auth.login.LoginContext$4.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.login.LoginContext.invokePriv(Unknown Source)
	at javax.security.auth.login.LoginContext.login(Unknown Source)
	at com.microsoft.kerberos.client.KerberosSecureLogin.secureLogin(KerberosSecureLogin.java:89)
	at com.microsoft.kerberos.client.HdfsBridge.kerberosLoginHelper(HdfsBridge.java:159)
	... 2 more
Caused by: KrbException: Client not found in Kerberos database (6) - CLIENT_NOT_FOUND
	at sun.security.krb5.KrbAsRep.<init>(Unknown Source)
	at sun.security.krb5.KrbAsReqBuilder.send(Unknown Source)
	at sun.security.krb5.KrbAsReqBuilder.action(Unknown Source)
	... 17 more
Caused by: KrbException: Identifier doesn't match expected value (906)
	at sun.security.krb5.internal.KDCRep.init(Unknown Source)
	at sun.security.krb5.internal.ASRep.init(Unknown Source)
	at sun.security.krb5.internal.ASRep.<init>(Unknown Source)
	... 20 more
2068 [main] ERROR com.microsoft.kerberos.client.HdfsBridge  - KDC authentication has failed. Unable to connect to KDC server with provided information. Please contact hadoop admin to resolve this issue
LSA: Found Ticket
LSA: Made NewWeakGlobalRef
LSA: Found PrincipalName
LSA: Made NewWeakGlobalRef
LSA: Found DerValue
LSA: Made NewWeakGlobalRef
LSA: Found EncryptionKey
LSA: Made NewWeakGlobalRef
LSA: Found TicketFlags
LSA: Made NewWeakGlobalRef
LSA: Found KerberosTime
LSA: Made NewWeakGlobalRef
LSA: Found String
LSA: Made NewWeakGlobalRef
LSA: Found DerValue constructor
LSA: Found Ticket constructor
LSA: Found PrincipalName constructor
LSA: Found EncryptionKey constructor
LSA: Found TicketFlags constructor
LSA: Found KerberosTime constructor
LSA: Finished OnLoad processing

f:\azure-data-studio\notebooks-python\0.0.1\lib\site-packages\IPython\core\interactiveshell.py:2969: UserWarning: To exit: use 'exit', 'quit', or Ctrl-D.
  warn("To exit: use 'exit', 'quit', or Ctrl-D.", stacklevel=1)

If you have installed Polybase feature, you can test Polybase configuration settings with this tool.

You can run below command and see if you can successfully connect to the kerberos HDFS server using polybase configuration.

Copy 'Hadoop' folder from Polybase installation folder to the current working directory of this notebook
Run below cell.

In [20]:

#cmd = f'java -classpath "hdfs-kerberos-tester-0.0.1.jar;.\Hadoop\HDP2_2\*;.\Hadoop\*;.\Hadoop\conf" com.microsoft.kerberos.client.HdfsBridge ' + hdfs_node_name + ' ' + port_number + ' ' + kerberos_realm + ' ' + kerberos_kdc+ ' '+ kerberos_username + ' ' + kerberos_password_file + ' ' 

# executeCommand(cmd, ['CHECKPOINT 2 COMPLETED'], 'Testing Kebreros Connection')