AD Slow Authentication and prompting for credentials again and again

AD Slow Authentication and prompting for credentials again and again (Active directory Troubleshooting - Part 2)

It’s a most common issue in a complicated Active Directory environment, before am going to discuss about the authentication issues, I would like to discuss about the Active Directory basics like Pass through authentication, AD secure channel, NTLM and Kerberos

Pass through authentication

If you are worked on multi Domain/Forest environment or environment designed with user forest and resource forest, an Exchange Server resource forest topology has two forests. One forest contains the all the user accounts for your organization. This forest is called the user forest (accounts forest). The other forest does not contain any user accounts. It only contains the Exchange Server and disabled user accounts, in simple you have one Active Directory forest where your user accounts live and another Active Directory forest where your application are lives (Exchange server, File server)

In the above scenario Domain controller receiving the request from Exchange/file server to verify the user access, this must pass the request to Domain controller in the user forest, we should have the trust between domain of the server (called the resource domain/forest) and the domain of the user account (called the account domain/forest)

User from Domain A try to access application on server from Domain B, application server in Domain B doesn’t have user detail, it will check the local Domain controller on Domain B through workstation secure channel, and Domain controller on Domain B check the Domain controller on Domain A through trusted domain secure channel and Domain A return back the authentication to Domain B it’s called the pass through authentication since the user authentication request been passed to user domain.

Secure Channel

I have discussed about the secure channel, what is secure channel? It’s a communication channel provides more secure communication path between the domain controller and the workstations or member servers. It can also be used to retrieve domain-specific information, handling NTLM authentication pass-through to the domain controller or from DC to DC for the same.

Two Forest or Domain connected through Forest trust / Domain trust, trust establishment is a shared secret (called a trust password) that domain controller use in the two domains for computing the session key that is used for protecting the secure channel traffic. By using this secure channel, the DC in the resource domain can pass logon requests securely to the DC in the account domain, in the same way that the server passed the logon request to the former DC. The secure channel between DCs in two domains that are connected via a trust relationship is called a trusted domain secure channel. In contrast, the secure channel between the member server and the DC in the resource domain is called a workstation secure channel

While adding a computer to domain, computer account has been created in Activity directory and password been generated for computer account, computer account password been changed every 30 day’s and stored in computer and domain controller, while power on the computer, Netlogon service on computer use the computer account password from the computer is authenticated against the password on the Domain Controller and establishes a secure channel with that DC, same way server creates a secure channel with that DC it get authenticated

For Domain controller, Netlogon service sets up secure channels with all the trusted domains (one Domain controller in each trusted domains) you can check this using Nltest command

To check the current secure channel with a particular Domain

nltest /sc_query:Domain Name

To reset secure channel to different Domain Controller (this will randomly select the Domain Controller)

nltest /SC_RESET: Domain Name

To reset secure channel to a particular Domain Controller

nltest /SC_RESET: Domain Name\ Domain Controller Name

You can change the Domain and Domain Controller Name as per your requirement

For remote server you can add

/server: server name

NTLM and Kerberos

NTLM and Kerberos are the protocols used for authentication, we all know NTLM is outdated and Kerberos is the new protocols used for authentication, Kerberos can impersonate a user when trusted, so no need to contact Domain controller every time in order to authenticate access to a resource, If the client is logged on to a domain, the browser never prompts the user for credentials; it simply uses the user's default logon credentials.

We are not using NTLM any more? No still some areas where we have to use NTLM for the sake of compatibility, RPC over HTTP to connect to an Exchange mailbox. ISA for web proxy servers, let’s go in deep how it’s works

Client opens a URL through browser

Client browser sent a request to proxy server with integrated authentication credentials

The proxy server needs to verify the user credentials, by sending the authentication request to the domain controller is has a secure channel

That domain controller responds to the proxy server

The proxy server answers the client with the requested internet page

Seems to be simple however for each web connection from a client, the proxy server needs to verify the user credentials by sending authentication request to the DC, this will increase the high volume of NTLM authentication

Think the similar scenario in multi Domain/Forest environment, user in one Forest and proxy server in other Forest, this will increase the NTLM Pass through authentication traffic

NTLM authentication handled by Netlogon service, passing NTLM authentication requests to a domain controller that can handle them, and receiving them on that domain controller to be handled, you can enable debug logging for the Net Logon service to see what happen on the proxy server / domain controller, like which user getting authenticated to which domain controller

We have number of threads which will authentication request, like number of concurrent NTLM authentications processed by the server, the defaults are typically 1 for this, meaning that there is one thread to hand off, receive and process these requests, we can re-configure this through MaxConcurrentApi

The MaxConcurrentApi thread can only deal with one authentication at a time, in normal scenario it is very quick.  So the high volume of authentication transactions must be handled by one or two threads (by default) and this will be a bottleneck, which resulting the delay and the authentication request have to wait longer than a remote client can tolerate

Due to this delay, client browser would ask for a credential prompt rather than the web page he want to open, so this might be the issue for slow Authentication and prompted for credentials again and again

We can resolve this by increasing the MaxConcurrentApi value on proxy server, if it’s a multi Domain/Forest environment, then we needs to increase the MaxConcurrentApi value on resource forest Domain Controller, it depends on the configuration

First we have to understand current issue

Enable the Netlogon logs on proxy server and Domain Controller authenticating proxy server or the proxy server secure channelled Domain Controller

Analysis the log for authentication failure and delay

We can monitor the current secure channel traffic through the perfmon by adding the counters, Semaphore Waiters, Semaphore Holders, Semaphore Acquires, Semaphore Timeouts, Average Semaphore Hold Time

Logon to Domain Controller authenticating proxy server

Open perfmon, add counters – select Netlogon on performance object

And select Average Semaphore Hold Time

If you don’t find the counters, need to install http://support.microsoft.com/kb/928576 hot fix, this adds New performance counters for Windows Server 2003

If the Average Semaphore Hold Time is greater than normal, Average Semaphore Hold Time should normally be very quick. Longer hold times mean that a potential bottleneck is occurring, it will delay authentication process and the authentication request has to wait longer than a remote client can tolerate, it will slowdown the authentication and prompted for credentials to re-initiate the authentication process

To calculate MaxConcurrentApi for your environment

http://support.microsoft.com/kb/2688798

It depends on the outcome, you should add more servers to service the legacy authentication load or increase the MaxConcurrentApi registry value

Warning rules:

Average Semaphore Hold Time > 0.2 should be a yellow warning.

Average Semaphore Hold Time > 0 should be a red warning.

Semaphore waiters > 1 should be a yellow warning.

 Semaphore waiters > 4 should be a red warning.

 Any Semaphore Timeouts is a red warning.

Conclusion:

Slow Authentication might be due to many issues, like client to DC connectivity, network, subnet and site configuration, DNS SRV configuration, Profile, logon script and GPO, so you have to analysis properly to find the root case, this is the one of the scenarios which I have faced many times in a complex environment, hope this will help you to understand the Active Directory Authentication and troubleshooting procedures see you soon in an another article.

Active directory Troubleshooting (Part1 - Diagnostics Logging)

Troubleshoot Active Directory Server Replication

Share this . . .