Potential for Kerberos Issues When Using a Cisco VPN/ASA with Win2003 or later DC’s

[LEFT][CODE]http://blogs.technet.com/askds/archive/2009/06/18/potential-for-kerberos-issues-when-using-a-cisco-vpn-asa-with-win2003-or-later-dc-s.aspx[/CODE]
Hey everyone, Rob Greene here back after a long hiatus from blogging. I had an interesting case come through that I thought many of you IT pros would be interested in.
[B]Background[/B]
The customer had an issue with using [URL="http://www.cisco.com/en/US/products/hw/vpndevc/ps2284/index.html"]Cisco VPN[/URL] and [URL="http://www.cisco.com/en/US/products/ps6120/index.html"]Cisco ASA[/URL] concentrators and authenticating the user with Kerberos authentication. After they upgraded all their Windows 2000 domain controllers to Windows Server 2003 domain controllers, users could not authenticate through the VPN.
After doing this they gave Cisco a call and they were told to start [B]DSA.MSC [/B]and check the box under [I]Account Options[/I] to “Do not require Kerberos preauthentication” for all user accounts that need to connect through the appliances. When they did this as a test it did allow the user to start authenticating through the devices. However the customer did not like this option since:[INDENT] 1. This change lowers the security of Kerberos authentication by disabling password replay attack protection.
2. Everything had worked previously with Windows 2000.[/INDENT]If you want to learn more about preauthentication you can click [URL="http://technet.microsoft.com/en-us/library/bb742516.aspx"]here[/URL] and look under the section:
[B]Example AS Administration[/B].
If you search Cisco’s website you will find that they have older documentation stating that you must disable this feature on the user’s account before the VPN will start working.
I don’t have real network traces (as I can’t show you a real customer’s network!), but I can tell you what we saw in the traces that led us to a potential workaround.[INDENT] VPN Appliance --> UDP Port 88 / KRB5 AS-REQ --> Windows DC
VPN Appliance <-- UDP Port 88 / KRB5KDC_ERR_REAUTH_REQUIRED <--Windows DC
VPN Appliance --> UDP Port 88 / AS-REQ (with Pre-Auth Data) --> Windows DC
VPN Appliance <-- UDP Port 88 / [COLOR=#ff0000]KRB5KDC_ERR_RESPONSE_TOO_BIG[/COLOR] <-- Windows DC[/INDENT]A [B]KRB5KDC_ERR_RESPONSE_TOO_BIG[/B] means that the requested Kerberos data was larger than the maximum default datagram size of 1465 bytes. The next step here usually would be for the client to start the Kerberos ticketing process over using TCP rather than UDP. If this was a Windows XP client instead of the Cisco VPN appliance we would have the customer implement the registry key change outlined in the below KB article.[INDENT] 244474 How to force Kerberos to use TCP instead of UDP in Windows
[URL="http://support.microsoft.com/default.aspx?scid=kb;EN-US;244474"]How to force Kerberos to use TCP instead of UDP in Windows[/URL][/INDENT]So basically the real issue is that the Cisco appliance is being asked by the KDC to use TCP instead of UDP to do the AS-REQ with pre-authentication data present. However, the Cisco appliance does not continue at this point. I spent the past few weeks working with Cisco directly (big shoutout - they have been [I]great![/I]), and they have published the following new articles:[INDENT] Cisco VPN 3000: [URL="http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCed27444"]CSCed27444[/URL] - VPN3000 does not switch to TCP upon receiving Kerberos error code 52
Cisco ASA platform: [URL="http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCsi32224"]CSCsi32224[/URL] - ASA does not switch to TCP upon receiving Kerberos error code 52
[URL="http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCtd92673"]CSCtd92673[/URL] -- "Kerberos authentication fails with pre-auth enabled"[/INDENT]Unfortunately, the Cisco VPN 3000 concentrator has been discontinued so there is no update available for the Cisco VPN 3000 to add this functionality. Also, you will need a support contract from Cisco to logon to those two articles and read them.
Please contact Cisco directly to find out more information about the issue, and with what IOS version it will be included in; we have provided this information to make it easier to find the content.
[B]In the Meantime[/B]
So now the question becomes this: If that is what is going on, then how did it work when a Windows 2000 domain controller was used? And what can you do about it until an IOS update is released?
There was a change in the default datagram reply packet size used by the Kerberos Key Distribution Center (KDC) between Windows 2000 and Windows Server 2003. In Windows 2000 the default size is 2000 bytes, where it is 1465 bytes in Windows Server 2003. For those interested you can review the below KB article and find the registry key of [B]MaxDatagramReplySize[/B].[INDENT] 837361 Kerberos protocol registry entries and KDC configuration keys in Windows Server 2003
[URL="http://support.microsoft.com/default.aspx?scid=kb;EN-US;837361"]Kerberos protocol registry entries and KDC configuration keys in Windows Server 2003[/URL][/INDENT]So that means this customer’s Kerberos ticket size for the user accounts is less than 2000 bytes, but more than 1465 bytes. Keep in mind that if you are reading this blog, your environment may still not work with the registry key setting at 2000. This is because the Kerberos ticket size is going to vary based on the number of groups that the user belongs to. You can review the KB below to learn more:[INDENT] 327825 New resolution for problems with Kerberos authentication when users belong to many groups
[URL="http://support.microsoft.com/default.aspx?scid=kb;EN-US;327825"]New resolution for problems with Kerberos authentication when users belong to many groups[/URL][/INDENT]The only workaround that we can give a customer is to [URL="http://support.microsoft.com/kb/837361"]modify the [B]MaxDatagramReplySize[/B] registry value[/URL] to the size of your user’s largest Kerberos ticket size. It really depends on the group membership size of the user attempting to authenticate because of the PAC data that is included in the TGT. For those customers that have used the [B]MaxTokenSize[/B] registry value should equal that value and restart the KDC service. If none of these acronyms are making sense you might want to check out my older blog post [URL="http://blogs.technet.com/askds/archive/2008/03/06/kerberos-for-the-busy-admin.aspx"]Kerberos for the Busy Admin[/URL].
Hey wow, this sounds like a pretty easy fix right? Well, you can actually have other problems because of this change. If your networking gear (switches, routers, firewalls) cannot handle large UDP packets without fragmenting them then [I]all [/I]Kerberos authentication could start to fail in your environment. See [URL="http://support.microsoft.com/default.aspx?scid=kb;EN-US;244474"]KB244474[/URL] under more information to learn more about this.
Please keep in mind what you are doing when you implement the [B]MaxDatagramReplySize[/B] registry value. You are in fact telling the KDC to never request the client send [B]KRB5KDC_ERR_RESPONSE_TOO_BIG[/B] message back to a Kerberos client, and you should consider implementing the custom group policy setting included in [URL="http://support.microsoft.com/default.aspx?scid=kb;EN-US;244474"]KB244474[/URL] on all domain-joined machines to use TCP for Kerberos authentication always. If you are running purely Vista/2008 or later though, this is on by default.
Lastly, if you have to set this value very high, and you know that your networking gear will not support the packet sizes that had to be implemented, you may have to put a domain controller on the same IP subnet as the Cisco appliance to successfully get Kerberos tickets.
- Rob ‘I mean, you know’ Greene

[/LEFT]

[LEFT][CODE]http://blogs.technet.com/askds/archive/2008/03/06/kerberos-for-the-busy-admin.aspx[/CODE]
Hi Rob here, I am a Support Escalation Engineer in Directory Services out of Charlotte, NC, USA. We work a lot of Kerberos authentication failure issues. Since Kerberos is typically the first authentication method attempted, it ends up having authentication failures more often. One of the great things about Windows is that the product seems to just work without too much customization that is needed by the customer. However I wanted to create a blog to try and demystify how Kerberos authentication works. I plan on writing several Kerberos blogs in the near future to include Kerberos Delegation aka Double-hop, how to troubleshoot Kerberos authentication.
There are some general terms that you might not be familiar, so let's run through them quickly.
[B][I][U]Principal Names:[/U][/I][/B]
Kerberos defines two different types of accounts (or Principals). The two different names given to these types of accounts are User Principal Name (UPN), and Service Principal Name (SPN). We would typically relate these two types of principals to Active Directory users and computers.
Only user accounts have a UPN defined on their account. When looking at a user account if you click on the Account tab, the UPN is derived from the combining of the two fields listed for “User logon name”. A User Principal Name must be unique across the entire forest otherwise when the KDC goes to look up the Users Account via UPN it will get back more than one account and cause authentication failures for all users that have the same UPN. The UPN of an Active Directory object is an attribute of the object, and can only hold a single value. The attribute name is userPrincipalName. An example of a UPN is: [EMAIL="rob@contoso.com"]rob@contoso.com[/EMAIL].
Service Principal Names MUST be unique across the entire Active Directory forest, and can be assigned to either User accounts or Computer accounts. Only computer accounts automatically have Service Principal Names defined.
Service Principal Names define what services run under the accounts security context. For example some of the services that a computer might have are File server / CIFS (Common Internet File System), if it is a domain controller it is going to have an LDAP SPN, and Active Directory Replication SPN, and FRS SPN. Service Principal Names can be defined on user accounts when a Service or application is running under that users Security context. Typically these types of user accounts are known as “Service Accounts”. It is very import that you understand that Service Principal Names MUST be unique throughout the entire Active Directory forest.
Some typical scenarios when a user account has a Service Principal Name defined are:

[LIST][*]When SQL Server Service is using a user account or “Service Account” instead of the default of “LocalSystem”. An example is: MSSQLSvc/sqlsrvr.contoso.com:1433[*]When an IIS Web Application Pool is running as a specified user account rather than as the default of “Network Service”. An example is: http/websrvr.contoso.com[/LIST]
Some tools that can be used to list the Service Principal names on an Active Directory object are: LDP, LDIFDE (These two are great utilities if you want to query LDAP for all objects that have the SPN defined on them.), next is ADSIEdit, or SetSPN. The last two are great utilities if you want to see what SPNs are registered on a given object.
[B][I][U]Kerberos tickets:[/U][/I][/B]
[U]KDC (Key Distribution Center)[/U]: The KDC is a service that should only be running on a domain controller. The service name is “Kerberos Key Distribution Center”. Basically the KDC is the service that is responsible for authenticating users when Kerberos is used. The KDC implements two server components. Authentication Server (AS), and Ticket Granting Server (TGS).[INDENT] [U]Authentication Server (AS)[/U]: The KDC role that verifies the identity of the principal and issues the Ticket Granting Ticket (TGT) to the principal upon successful authentication. Holding a valid TGT allows the principal to request a Service ticket from the Ticket Granting Server (TGS). There will be a TGT in the Credentials Cache for each domain the principal has accessed resources in. An example of this would be: a user in contoso.com domain wanted access to a file server in emea.contoso.com the user would have a TGT for contoso.com, and emea.contoso.com[/INDENT][INDENT] [U]Ticket Granting Server (TGS)[/U]: The KDC role that issues a service ticket when a principal requests connection to a Kerberos service. You must have a Ticket Granting Ticket (TGT) for the Active Directory domain before you can be issued a service ticket in that Active Directory domain. Although the KDC issues the service ticket it does not talk directly to the service that the principal is requesting the ticket for. Once a service ticket has been issued by the Ticket Granting Server, the service ticket is put into the principal’s credentials cache for later use. When the principal needs to connect to the requested service the service ticket is used from the credentials cache and sent to the service it is attempting to connect to.[/INDENT]There are only two different types for tickets that the KDC issues.

[LIST][*]Ticket Granting Ticket (TGT)[*]Service tickets from the Ticket Granting Server.[/LIST]
[B][I][U]Kerberos Ticketing Process:[/U][/I][/B]
How Kerberos works can be very difficult to keep straight. There is a lot of decrypting and encrypting of authentication data. I have laid out the entire ticketing process here in two formats. If you are just trying to understand at a high level of how Kerberos authentication works I would suggest that you keep to the number lists below. If you already know the high level Kerberos ticketing process and are looking for more detail on how Kerberos authentication works I would suggest that you look at the bulleted list under each numbered list below.[INDENT] [IMG]http://blogs.technet.com/blogfiles/askds/WindowsLiveWriter/KerberosfortheBusyAdmin_8472/clip_image002_thumb.gif[/IMG][/INDENT][I]Image is taken from the [URL="http://technet2.microsoft.com/WindowsServer/en/library/4a1daa3e-b45c-44ea-a0b6-fe8910f92f281033.mspx"]Kerberos TechNet article[/URL][/I]
1. The client sends a KRB_AS_REQ to the KDC and more specifically the Authentication Server to request a Ticket Granting Ticket (TGT).

[LIST][*]The AS_REQ is built on the client machine using the current computer time and encrypting it with the users Password hash. There is some other information within the AS_REQ packet that includes the UPN of the Principal. This information is called “Authentication Data”[/LIST]
2. Once the KDC verifies the users Authentication Data, it responds back to the client with a KRB_AS_REP to the client with a TGT and session key for the TGT.

[LIST][*]The KDC can decrypt the AS_REQ because the principal’s password hash is stored within Active Directory; it then verifies that the timestamp in the AS_REQ to make sure that the systems are within 5 minutes of time skew. This process validates that the principal authenticating knows the users account and password.[*]The TGT session key is used in subsequent service ticket requests.[/LIST]
3. The client is then able to request service tickets since it has a valid TGT for the Active Directory domain. The client then sends a KRB_TGS_REQ to the KDC and more specifically the Ticket Granting Server to request a Service Ticket. Keep in mind that it not only sends the Service Ticket Request, but also a copy of the TGT that it was given earlier.

[LIST][*]The principal is going to build an authenticator that is encrypted with the Session key of the TGT. This authenticator data is the User Principal Name, as well as the current system timestamp. The TGS_REQ has the following information: The Service Principal name that they want access to, and the TGT from the previous step.[/LIST]
4. Once the KDC has verified the validity of the TGT that is included with the Service Ticket request it responds back to the client with a KRB_TGS_REP with the Service Ticket and service session key.

[LIST][*]After successful decryption of the TGT, the Ticket Granting Server has access to the TGS Session key that the authenticator data is encrypted with. Decrypting the Authenticator data helps to prove the user principal who was issued the TGT is who sent the Service Ticket Request. Also with decrypting the Authenticator data, the KDC is able to verify that system time is within 5 minutes of time skew and is not a replay attack.[*]An LDAP query is done for the Service Principal Name (SPN) that the ticket is being requested for from the Ticket Information. The LDAP Search is doing a query on the servicePrincipalName attribute and is done against a domain partition.[*]The TGS generates a Unique Service Session Key. This session key is going to be used by the principal and service.[*]The KDC then creates the service ticket with the following information within it:
[LIST][*]A copy of the unique session key[*]v Authorization data that is copied from the principals TGT that was given in the request. The Authorization data is going to have the Principals SID, all group information, again this information is known as the Privileged Attribute Certificate or PAC.[*]Once the service ticket information is compiled the entire service ticket is encrypted with the Services User Key (password hash).[*]The ticket information and the principal’s copy of the service session key are encrypted with the Ticket Granting Server session key.[/LIST]
[/LIST]
5. Next the client sends the Service Ticket to the Service/Application as a KRB_AP_REQ. You will typically see this embedded in the type of packet that the service uses. Like file shares use SMB, for example.

[LIST][*]The information in the KRB_AP_REQ is the service ticket obtained from the TGS, and an authenticator encrypted with the session key for the service.[/LIST]
6. After authentication succeeds The Service responds back to the client with a KRB_AP_REP.

[LIST][*]When the client receives KRB_AP_REP, it decrypts the service’s authenticator with the service session key it shares with the service and compares the time returned by the service with the time in the client's original authenticator. If the times match, the client knows that the service is genuine.[*]After the server authenticates the client using Kerberos authentication, the Privilege Attribute Certificate or PAC is taken from the service ticket and used to create the user's access token. This PAC is verified against a domain controller through a NetLogon call to verify the PAC Signature.[/LIST]
As you can see, the KDC does not participate directly in the authentication of users to the end service with Kerberos. The KDC is known as the trusted 3rd party in this type of authentication. It is known this way because it is the only service that knows the passwords of the user and the service.
[B][I][U]Kerberos Delegation:[/U][/I][/B]
Kerberos delegation is the act of principal (Service) impersonating another principal (user) to gain access to a 3rd principal (service). In other words, User connecting to an IIS Server that queries a SQL Server as the user who is requesting some data from the web server.
There are two different types of delegation.

[LIST][*]Unconstrained (Windows 2000/2003/2008 Servers can do this type of delegation.)[*]Constrained (Windows 2003/2008 Servers in 2003 Domain Functional Level (or higher) can do this type of delegation.)[/LIST]
We are going to cover Kerberos Delegation in detail in another blog entry.
[B][I][U]Tools used to view and troubleshoot Kerberos:[/U][/I][/B]
[U]KerbTray[/U]: This is a great utility GUI based utility that shows up in the system tray that allows you to view all your Kerberos tickets as well as being able to purge them. The purge feature is done by right clicking the green ticket in the system tray and selecting “Purge Tickets”. The Purge Tickets options delete all Kerberos tickets.
[url=http://www.microsoft.com/downloads/details.aspx?FamilyID=4e3a58be-29f6-49f6-85be-e866af8e7a88&DisplayLang=en]Download details: Windows 2000 Resource Kit Tool: Kerbtray.exe[/url]
[U]KList[/U]: This is a great command line tool that lists Kerberos tickets as well as being able to purge Kerberos tickets. The nice thing about this tool is that you can selectively purge Kerberos tickets rather than deleting all tickets like the KerbTray utility does.
[URL="http://www.microsoft.com/downloads/details.aspx?FamilyID=1581e6e7-7e64-4a2d-8aba-73e909d2a7dc&DisplayLang=en"]Download details: Windows 2000 Resource Kit Tool: Klist.exe[/URL]
[U]Network Captures[/U]: Network capturing utilities can be indispensable when troubleshooting a Kerberos authentication issue. As we say here “the truth is on the wire”. Most network capture utilities have very good Kerberos parsers included. Some of our favorites here are Network Monitor 3, WireShark, and Ethereal.
[U]Kerberos Event logging[/U]: The operating system by default does not create event log entries for Kerberos authentication events. You can however turn this feature by reviewing the following KB article:
[I]262177 How to enable Kerberos event logging - [/I][URL="http://support.microsoft.com/default.aspx?scid=kb;EN-US;262177"]How to enable Kerberos event logging[/URL]
You would enable this feature on the client machines and any other machines participating in Kerberos delegation.
[I][COLOR=#ff0000]Note: I would caution you on enabling this feature. There are some events that you will see that are really not Kerberos errors – such as 0x12 KDC_ERR_CLIENT_REVOKED, 0xD KDC_ERR_BADOPTION, or 0x34 KRB_ERR_RESPONSE_TOO_BIG. We have had cases where the customer enabled this from a previous case and never turned it back off. Since they were now sensitive to all Kerberos errors they have opened up a new case just to be asked to turn off the logging because the events were not really errors.[/COLOR][/I]
[B][I][U]Kerberos Dependencies:[/U][/I][/B]
There are some basic dependencies that need to be in place for Kerberos Authentication to succeed.
For Kerberos to function correctly, the supporting infrastructure must be sound. Since Passwords are used to encrypt data within Tickets it is imperative that when a user or computer changes their password that Active Directory replication is able to send these changes throughout the environment.
Proper name resolution is required. For Kerberos to function it has to be able to resolve the proper IP Addresses for the KDC as well as the Service Principal you are attempting to access. Both DNS and NetBIOS name resolution must be setup correctly. Many cases identified as Kerberos issues were caused by bad records in DNS, broken WINS replication, or HOSTS/LMHOSTS files with invalid data. DNS SRV records for _kerberos will need to be in place, for both the _tcp and _udp DNS sub-domains. Check the configured DNS suffixes and search order as well. A typical problem that we find is that the DNS Zone has been configured for WINS Lookup. When this happens the wrong DNS FQDN is found for the service the user is attempting to connect to, which then causes the application to ask for a service principal for the wrong FQDN.
All machines participating in Kerberos authentication need to be within 5 minutes of time. By default Windows will use the Windows Time (w32time) service, but can use a third party NTP client. We don’t care what time it is as long as all computers in the forest agree to within 5 minutes of one another.
We need to ensure that we have good connectivity. TCP and UDP port 88 must be open from clients to domain controllers. A common problem is that routers will arbitrarily fragment UDP packets; when this happens the Kerberos ticket request packets are discarded by the KDC. Windows Vista and Windows Server 2008 now default to using TCP for kerberos ticket requests. Typically you work around this issue by implementing the following KB article:
[I]244474 How to force Kerberos to use TCP instead of UDP in Windows Server 2003, in Windows XP, and in Windows 2000[/I] - [URL="http://support.microsoft.com/default.aspx?scid=kb;EN-US;244474"]How to force Kerberos to use TCP instead of UDP in Windows[/URL]
LDAP queries will be made by the DC / KDC for Service Principal Name records. Duplicate computer names, usernames, etc, or manually registered duplicate SPN's anywhere in the forest can cause Kerberos errors. There is an event that is created when this happens, but it is only logged on the domain controller that attempted to find the service principal. It is a Kerberos Event 7.
[U]Other Kerberos information:[/U]
How the Kerberos Version 5 Authentication Protocol Works [URL="http://technet2.microsoft.com/WindowsServer/en/library/4a1daa3e-b45c-44ea-a0b6-fe8910f92f281033.mspx"]How the Kerberos Version 5 Authentication Protocol Works: Logon and Authentication[/URL]
Kerberos Authentication in Windows Server 2003 [URL="http://technet2.microsoft.com/windowsserver/en/technologies/featured/kerberos/default.mspx"]Kerberos Protocol Transition and Constrained Delegation[/URL]
MIT’s references for Kerberos
[URL="http://web.mit.edu/kerberos/"]Kerberos: The Network Authentication Protocol[/URL]
The Kerberos Consortium
[URL="http://www.kerberos.org/"]http://www.kerberos.org[/URL]
Kerberos RFC
[URL]http://www.ietf.org/rfc/rfc1510.txt[/URL]
[URL]http://www.ietf.org/rfc/rfc4120.txt[/URL]




- Rob Greene

[/LEFT]