Users are facing delay in logging in to a Terminal server via RDP client. After entering the credentials, they are seeing a long delay before they get the desktop

Problem:

In Windows 2003 SP2 x64 bit R2 server, faced issue with Terminal services.
Users are facing delay in logging in to a Terminal server via RDP client. After entering the credentials, they are seeing a long delay before they get the desktop

Also apart from above, amazing part has been observed that few initial users were able to access desktop and taskbar after credential in terminal session but from next user onwards, noone will get desktop, just a blank screen and almost to wait for 30 to 40 minutes.

More info: While troubleshooting, we found that few users who RDP to the server will get a desktop and few of them actually never get a desktop. The users who get the 3^rd party application launched when the login will never get a desktop. Rather they get the application which will close the explorer.exe as soon as it starts off. The application has an option to logoff within the tool bar of the application itself and that will trigger the log off process and close the RDP session.

So, for users who gets desktop, we see delay in getting the desktop. For the users who gets the application interface, we see delay in getting the application interface.

Cause: We found two different issues in this case. The first  one is due to couple of 3^rd party HP services running, namely “Net Driver HPZ12” and “PML Driver HPZ12” which has bloated the registry entries under following registry locations.

[ HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Terminal Server\Install\Software\Hewlett-Packard\ ]

[ HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Terminal Server\Install\RefHive\Hewlett-Packard\ ]

[ HKU\.DEFAULT\Software\Hewlett-Packard ]

When the user logs into RDP session, due to the above bloated registry, we were waiting for the enumeration of the subkeys / entries under the above keys. Once we enumerate all entries here, the login process will then complete and eventually user will get desktop or application interface.

Due to the large number of entries here, it was taking a long time to complete this process. However, these entries are not required for regular working for the users as they got created by the above mentioned HP services.

We also found another issue being caused by the 3^rd party application init.exe which will be initiated for those users who get the application interface. Randomly the init.exe stops responding at the startup when the users login. In this instance, we see that userinit.exe which will start the explorer.exe is done with its job and exited. But the init.exe which is supposed to close the explorer.exe and initiate the Application Interface is not completing and is hung. We are not sure what exactly it is doing and this has to be investigated by the application owner / vendor. But we see this issue very randomly and during our testing we found this init.exe getting stuck only once.

Solution:

For the registry bloat issue by HP service, we followed the action plan outlined below.

1.      Stop and disable HP services “Net Driver HPZ12” and “PML Driver HPZ12”

2.      Rename the associated .msi file (cioum64.msi) of the above service under C:\windows\system32\spool\drivers\x64\3\ to cioum64.old and created a blank dummy cioum64.msi (created blank cioum64.txt and renamed it as cioum64.msi)

3.      Deleted the following bloated registry entries which are actually not required for server operation and deleting them won’t cause any harm to the server.

HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Terminal Server\Install\Software\Hewlett-Packard

HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Terminal Server\Install\RefHive\Hewlett-Packard

HKU\.DEFAULT\Software\Hewlett-Packard

After doing this, we successfully logged in with couple of users who gets the desktop (not the application interface when they login) and did not see anymore delay in getting desktop. We logged them in within 1 min which is good enough.

We tried logging in around 4 – 5 users who get the application interface, at one instance we found that “init.exe” hung and did not get interface. Killing that hung init.exe will close the session for the user.

For all other attempts to login the same user and other users who gets application interface, we did get the application launched without any problem.

So if you see similar issue of such users not getting application when they login and their session getting stuck on “Init.exe” which spikes CPU to 25%, we have to engage application owner / vendor.