Wilson HPC Computing Facility

Troubleshooting Kerberos and SSH problems

Contents

Additional notes specific to Mac and Windows client software are found on the User Authenticaion page


Part 1. Kerberos kinit Problems

Error message: kinit(v5): Cannot find KDC for requested realm while getting initial credentials

Problem:

/etc/krb5.conf file doesn't contain .FNAL.GOV information.

Solutions:

  1. Replace /etc/krb5.conf with Fermilab-supplied version of krb5.conf
  2. Modify /etc/krb5.conf, adding Fermi-specific stanzas
  3. If user does not have permission to modify /etc/krb5.conf, copy Fermilab-supplied version into home area, and do
         export KRB5_CONFIG=$HOME/krb5.conf
    		  		
    to tell all Kerberos commands to use the user's copy of krb5.conf

Related Problem

On Macintosh computers (OS-X operating system), Kerberos is installed on all recent versions. However, there are two locations and names for the krb5.conf file:

  • /etc/krb5.conf
  • /Library/Preferences/edu.mit.Kerberos

(Note: the file in /Library is named edu.mit.Kerberos, not krb5.conf.) Either will work, but you should only have one of these two files in place.

Error message: kinit: Preauthentication failed while getting initial credentials

Problem:

kinit fails with preauthentication error

Solutions:

  1. Usually the problem is simply that you have typed in your kerberos password incorrectly. Try again.
  2. If you have lost your kerberos password, call the Fermilab service desk at 630-840-2345, during business hours, to have your password reset.
  3. Occassionally this is not a password problem, but a problem with your system's clock. Make sure that the "date" command returns a time correct to within 5 minutes.

Error message: kinit: krb5_get_init_creds: Too large time skew

Problem:

kinit fails with time skew message

Solution:

  1. Your system clock must be within 5 minutes of the correct value. Make sure that the "date" command returns a time correct to within 5 minutes.

Error message: kinit: KDC has no support for encryption type while getting initial credentials

Problem:

kinit fails with complaint about encryption type

Solution:

  1. This problem appears on recent Ubuntu and related Linux distributions. To fix, edit /etc/krb5.conf file, and in the [libdefaults] section add
           allow_weak_crypto = true		  
    		  		

Error message: kinit: Client not found in Kerberos database while getting initial credentials

Problem:

kinit fails with database complaint

Solutions:

  1. Your kerberos principal may differ from your username on your local system. Use "kinit username@FNAL.GOV" where username is your Fermilab kerberos principle. Using "kinit" by itself will use your local username by default.
  2. Your Fermilab ID or visitor ID has expired. Please see the renewal instructions at Accounts and Passwords webpage.

Error message: kinit: Client's entry in database has expired

Problem:

kinit fails because of an expired password

Solutions:

  1. You must change your kerberos password once a year with the "kpasswd" command (the same command you used to change your initial kerberos password). You can try to change your password, even if it is expired, by using "kpasswd" on your local machine.
  2. You can call the Fermilab service desk, 630-840-2345, and request that they reset your kerberos password.

Part 2. ssh problems

General Note

In general, ssh login failures will be indicated by either "permission denied" messages, or by a cryptocard prompt. If none of the solutions below fixes your problem please email the output of the command "klist -f; ssh -vvv tev.fnal.gov" to tev-admin@fnal.gov for further assistance.

Problem:

Not having a kerberos ticket granting ticket (TGT), or having an expired TGT

Solution:

Verify with the "klist -f" command that you have a ticket. If you don't have a ticket, or have an expired ticket, get a new ticket with kinit.

	hostname:~$ kinit -rf 7d johndoe@FNAL.GOV
	hostname:~$ klist -f
	Ticket cache: /tmp/krb5cc_1234
	Default principal: johndoe@FNAL.GOV

	Valid starting     Expires            Service principal
	08/17/12 09:31:16  08/18/12 11:31:16  krbtgt/FNAL.GOV@FNAL.GOV
	        renew until 08/24/12 09:31:09, Flags: FRIA
    	

This is the normal output, indicating that a forwardable, renewable, ticket exists. Check the expiration time - if the current time is past the expiration, login attempts will fail.

	hostname:~$ klist
	klist: No credentials cache file found (ticket cache /tmp/krb5cc_6789)
      

This klist output indicates that you do not have a Kerberos ticket. Use kinit to get a ticket before attempting to login.

Kerberos tickets expire after 24 hours. If you include the "-r 7d" switch on your kinit command line, you will receive a renewable ticket. Renewable tickets may be renewed by using "kinit -R" before they expire at the end of any 24hour period. Tickets are renewable for up to the period specified in the "-r" switch, to a maximum of 7 days.

Another useful switch to kinit is "-f", which asks for a forwardable ticket. If you have a forwardable ticket, once you login to a Fermilab machine, say tev.fnal.gov, you can then login from tev to another machine, say mylaptop.fnal.gov, without doing a new kinit. It is in general a bad idea to use "kinit" on any machine but your local system, as your password may be captured as it traverses the internet. The only time typing a kinit password is safe on a remote machine is when you a using an encrypted connection, such as ssh.

Problem:

Not having an account on the target machine, or having an account on the target machine under a different username

Solution:

A "permission denied" error will occur if you do not have an account on the target machine, or if your username on the target machine differs from your username on your local machine. Try

      ssh username@tev.fnal.gov

         or

      ssh -l username tev.fnal.gov
      

where username is your Fermilab username (the same name that you used in your kinit command). If this fails, send e-mail to tev-admin@fnal.gov and ask that the administrators verify that you have a valid account on the Wilson Cluster systems.

Problem:

Using an internet connection which has a "NAT" (network address translation), such as on a home wireless router.

Solution:

Nearly all home routers, wired or wireless, have a "NAT" function, which results in your local system having a different local network address than what is presented to remote machines. This allows you to have multiple local machines and only one external IP address. Your local addresses will generally be something like 192.168.x.y, or 10.x.y.z, when a NAT is present.

With a NAT, your ssh logins will fail with "Incorrect net address". To fix this, use "addressless" tickets. First, use "kdestroy" to delete your current ticket. Then, use kinit with "-a", "-A", or "-n" to request an addressless ticket. The switch required varies with the version of kerberos installed on your system. Use "man kinit" on your local system to determine which of these three switches to use.

For Mac OS users, please be aware that between versions 10.5 and 10.6 Apple changed the switch for addressless tickets from -A to -a. So if you have recently upgraded from Leopard (10.5) to Snow Leopard (10.6) and are still using -A you will need to change to -a. Also, the default behaviour on Mac OS is to supply addressless tickets, so you should also be able to simply drop the -A or -a switch entirely.

Problem:

Using an ssh client which does not have Kerberos authentication enabled.

Solution:

Some versions of ssh will not attempt to perform kerberos authentication. In this case, you will either receive a "permission denied" error, or a cryptocard prompt. To enable kerberos authentication, try the following "-o" switch:

     ssh -o "GSSAPIAuthentication yes" username@tev.fnal.gov
      

The quotation marks are required. If this form of ssh succeeds, you can configure your local system to always attempt to use kerberos authentication by editing either $HOME/.ssh/config or /etc/ssh/ssh_config and adding these lines:

    Host *.fnal.gov
       GSSAPIAuthentication yes
       GSSAPIDelegateCredentials yes
      

The "GSSAPIDelegateCredentials" line is necessary if you want to want to use X-windows clients on the remote (Fermilab) system. Note that you may need a "-X" or "-Y" switch on your ssh command to enable X forwarding.

Contact: Amitoj Singh
Last modified: Nov 6, 2017