By default, autoenrollment logs errors/failures and successful enrollments in the Application event log on the client machine.

Simultaneous Netmon Trace from both the client and the CA.

  • Filter the trace on RPC traffic (at client and CA levels) with netmon (use the Windows netmon parsers and not the default netmon parsers).
  • The client queries Active Directory for a list of available CAs and certificate templates that they are granted read and enroll permissions to.
  • The client then makes an RPC bind to the ICertRequest DCOM Interface on the CA using Kerberos authentication.

To enable enhanced logging of autoenrollment processes to include warning and informational messages, the following registry values  must be created.

User Autoenrollment: create the key below,
HKEY_CURRENT_USER\Software\Microsoft\Cryptography\Autoenrollment:
Create a new DWORD value named AEEventLogLevel”; set value to 0.

Machine Autoenrollment: create the keys below,

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Cryptography\AutoEnrollment\AEExpress]

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\System]

“AutoEnrollmentRefreshTime”=dword:00000000

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Cryptography\AutoEnrollment]

“AEEventLogLevel”=dword:00000000

[HKEY_CURRENT_USER\Software\Microsoft\Cryptography\AutoEnrollment]

“AEEventLogLevel”=dword:00000000

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Cryptography\AutoEnrollment]

“LogLevel”=dword:4

For DCOM tracing:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Ole]

“CallFailureLogginLevel”=dword:00000001

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Ole]

“ActivationFailureLogginLevel”=dword:00000001

For more information, please refer to the following articles:
Troubleshooting (Certificate Autoenrollment in Windows Server 2003):
http://technet.microsoft.com/en-us/library/cc755801(WS.10).aspx

Certification Authority Settings

  • Enable Certificate Services Debug Logging by running the following commands on the CA:
    certutil.exe -f -setreg ca\debug 0xffffffff
    Net Stop Certsvc && Net Start Certsvc
  • The following log files will be created:
    %SystemRoot%\certsrv.log (Certsrv.exe) Certificate Services
    %SystemRoot%\certutil.log (Certutil.exe)
    %SystemRoot%\certreq.log (Certreq.exe)
    %SystemRoot%\certmmc.log (Certmmc.dll) Certificate Services MMC snap-in
    %SystemRoot%\certocm.log (Certocm.dll) Certificate Services Setup

Troubleshooting Certificate Autoenrollment in Active Directory Certificate Services (AD CS):
http://social.technet.microsoft.com/wiki/contents/articles/3048.aspx

 Troubleshooting autoenrollment:

http://blogs.msdn.com/b/windowsvistanow/archive/2008/04/08/troubleshooting-certificate-enrollment.aspx
http://blogs.technet.com/b/instan/archive/2009/12/07/troubleshooting-autoenrollment.aspx

Hope this helps.

============================= Use Cases ====================================

#Tip: Error: target computer’s event log application contains Event ID: 6 ((0x800706ba) The RPC server is unavailable.)

Investigation:

A1) Check if GPO has been setup correctly: Computer,Security settings,Public … autorenrollment …

check if AEPolicy = 0x0000007 under hklm\software\microsoft\cryptography

A2) enable logging using certutil -setreg debugxffffffe3

and start trace capture: netsh trace start capture=yes

restart cryptsvc service

gpupdate /force

and stop netsh trace stop

send the c:\windows\autoenroll.log

and c:\users\<username>\appdata\temp\nettraces\Nettraces.cab and .etl

A3) In Certificate Template snap-in, right click the certificate template “Windows Computer Authentication” and ensure that Domain computers  has the Enroll and Autoenroll permissions, Authenticated Users has Read permission.

B)  Verify that Authenticated Users is member of the Certificate Service DCOM Access group.

C) Verify that CERTSVC_DCOM_ACCESS has been added to the DCOM Security Limits on the issuing CA: a. Click on Start, then Programs, then Administrative Tools, the Component Services. b. Expand the Component Services node. c. Expand the Computers node. d. Right-click on My Computer and select Properties from the context menu. e. Click on the COM Security tab. f. Under Access Permissions, click Edit Limits. g. Verify that the CERTSVC_DCOM_ACCESS group has been granted Allow Local Access and Allow Remote Access permissions. h. Click Cancel. i. Under Launch and Activation Permissions, click Edit Limits. j. Verify that the CERTSVC_DCOM_ACCESS group has been granted All Local Activation and Allow Remote Activation permissions. k. Click Cancel. l. Click Cancel. m. Close Component Services. I checked the component services and both “Edit Limits” and “Access permissions” have certificate dcom access -group listed with correct rights.

D) Define read and execute permissions for Authenticated users on C:\windows\system32\certsr. Yes, autheticated users has read and execute for certsrv folder.

E) Define read and execute permissions for Authenticated users on C:\windows\system32\certsr. Yes, autheticated users has read and execute for certsrv folder.

For some reason buildin\users group was missing two groups. Sometimes event 13 with “Server RPC is unavailable” means “access is denied”. A possible cause of this issue is that one of the following objects is not
added to the Builtin\Users group:

NT AUTHORITY\Authenticated Users
NT AUTHORITY\INTERACTIVE

  • This error occurs when attempting to bind to the Certification Authority to generate the Certificate request.  Troubleshooting includes:
    • Verify that the client can get a certificate using the Manual Enrollment via the MMC Certificate Wizard.
    • Check network connectivity to all of the available Certification Authorities listed in the Enrollment services object listed in the AD:
      CN=Enrollment Services,CN=Public Key Services,CN=Services,CN=Configuration,DC=Domain,DC=com
    • Verify that the Certificate Services service is running on the Certification Authority.
    • Verify that you can ping the Certificate Request Interface by running the following command:
      Certutil –Ping –Config CAMachineName\CAName
      Note that you can run the following command to get the Config string of the available Certification Authorities:
      Certutil –Dump
  • The Certutil –Ping command runs under the context of the user.  If the command works for the user but the AutoEnrollment failure errors for the computer account, then open a command prompt under the machine account and then re-run the ping command.
  • If the ping command fails for either the user or the computer:
    • Verify that Dcom is Enabled on the Certification Authority.
      For more information, see the following:
      How to disable DCOM support in Windows
      http://support.microsoft.com/default.aspx?scid=kb;en-us;825750
    • Check for firewalls and proxy settings.
    • Check for Checkpoint Firewalls which have issues with RPC traffic from a Windows Server 2003 SP1 server.  For more information, see the following:
      Some firewalls may reject network traffic that originates from Windows Server 2003 Service Pack 1-based computer
      http://support.microsoft.com/kb/899148
    • Use Portqry to verify that the appropriate RPC ports are opened.  For more information, please see the following:
      PortQryUI – User Interface for the PortQry Command Line Port Scanner
      http://www.microsoft.com/downloads/details.aspx?FamilyID=8355e537-1ea6-4569-aabb-f248f4bd91d0&DisplayLang=en
    • Starting portqry.exe -n 192.168.1.200 -e 135 -p TCP ...
      Querying target system called:
       192.168.1.200
      Attempting to resolve IP address to a name...
      IP address resolved to dc1.contoso.com
      querying...
      
      TCP port 135 (epmap service): LISTENING
      The last line shows that port 135 is open. Otherwise, the last line would have indicated NOT LISTENING.
      To check for a range of ports, enter the range of port numbers separated by commas, such as “135,1024-5000”.

     

Solution and workaround:

Based on investigation phase above, the Client configuration is OK, the CA configuration is ok. We suspect something strange between the client and the CA.

Verify if a firewall/router is not blocking the RPC/DCOM traffic,

1)      Make a trace on a computer client using netmon

2)      Make a trace on the CA

3)      Analyze the netmon traces

The root cause of the auto-enrollment issue has been identified :

  • The client machine tries to submit the certificate enrollment request to the CA by initiating a DCOM traffic (RemoteCreateInstance request) .
  • The CA server responds to the Request, but the answer RemoteCreateInstance Response never gets to the client computer…

Here is additional information :

The CA server responds to the Request, but the answer RemoteCreateInstance Response never gets to the client computer.

On the CA side we see two RPC conversations

5284 19:13:30.9027070 169.2637070  XXX.XX.XX.XXX ContosoCA1   MSRPC MSRPC:c/o Bind: IRemoteSCMActivator(DCOM) UUID{000001A0-0000-0000-C000-000000000046}  Call=0x3  Assoc Grp=0x7F73  Xmit=0x16D0  Recv=0x16D0  {MSRPC:1745, TCP:1744, IPv4:1684}
5285 19:13:30.9027100 169.2637100  XXX.XX.XX.XXX ContosoCA1   MSRPC MSRPC: Invalid RPC Header {TCP:1744, IPv4:1684}
5286 19:13:30.9027610 169.2637610  ContosoCA1   XXX.XX.XX.XXX TCP TCP:Flags=…A…., SrcPort=DCE endpoint resolution(135), DstPort=49207, PayloadLen=0, Seq=3247340487, Ack=3015246715, Win=64860 (scale factor 0x0) = 64860 {TCP:1744, IPv4:1684}
5287 19:13:30.9039890 169.2649890  ContosoCA1   XXX.XX.XX.XXX MSRPC MSRPC:c/o Bind Ack:  Call=0x3  Assoc Grp=0x7F73  Xmit=0x16D0  Recv=0x16D0  {MSRPC:1745, TCP:1744, IPv4:1684}

And

5289 19:13:30.9047800 169.2657800  XXX.XX.XX.XXX ContosoCA1   MSRPC MSRPC:c/o Alter Cont: IRemoteSCMActivator(DCOM)  UUID{000001A0-0000-0000-C000-000000000046}  Call=0x3  {MSRPC:1745, TCP:1744, IPv4:1684}
5290 19:13:30.9051990 169.2661990  ContosoCA1   XXX.XX.XX.XXX MSRPC MSRPC:c/o Alter Cont Resp:  Call=0x3  Assoc Grp=0x7F73  Xmit=0x16D0  Recv=0x16D0  {MSRPC:1745, TCP:1744, IPv4:1684}
5291 19:13:30.9054610 169.2664610  XXX.XX.XX.XXX ContosoCA1   TCP TCP:Flags=…A…., SrcPort=49207, DstPort=DCE endpoint resolution(135), PayloadLen=0, Seq=3015246935, Ack=3247340828, Win=64519 (scale factor 0x0) = 64519 {TCP:1744, IPv4:1684}
5292 19:13:30.9079460 169.2689460  XXX.XX.XX.XXX ContosoCA1   DCOM DCOM:RemoteCreateInstance Request {MSRPC:1745, TCP:1744, IPv4:1684}
5293 19:13:30.9090550 169.2700550  ContosoCA1   XXX.XX.XX.XXX DCOM DCOM:RemoteCreateInstance Response {MSRPC:1745, TCP:1744, IPv4:1684}

But, on the client side we have the following
The first RPS session is ok as verified on the CA side above –

1160 7:13:31 PM 10/30/2013 17.9158188 WINDOWS7PC.Contoso.net ContosoCA1.Contoso.net MSRPC MSRPC:c/o Bind: IRemoteSCMActivator(DCOM) UUID{000001A0-0000-0000-C000-000000000046}  Call=0x3  Assoc Grp=0x7F73  Xmit=0x16D0  Recv=0x16D0  {MSRPC:236, TCP:234, IPv4:39}
1161 7:13:31 PM 10/30/2013 17.9158188 WINDOWS7PC.Contoso.net ContosoCA1.Contoso.net TCP TCP:[Continuation to #1160]Flags=…AP…, SrcPort=49207, DstPort=DCE endpoint resolution(135), PayloadLen=345, Seq=2993966407 – 2993966752, Ack=927754047, Win=64860 (scale factor 0x0) = 64860 {TCP:234, IPv4:39}
1162 7:13:31 PM 10/30/2013 17.9163141 ContosoCA1.Contoso.net WINDOWS7PC.Contoso.net TCP TCP:Flags=…A…., SrcPort=DCE endpoint resolution(135), DstPort=49207, PayloadLen=0, Seq=927754047, Ack=2993966407, Win=6812 (scale factor 0x0) = 6812 {TCP:234, IPv4:39}
1163 7:13:31 PM 10/30/2013 17.9163353 ContosoCA1.Contoso.net WINDOWS7PC.Contoso.net TCP TCP:Flags=…A…., SrcPort=DCE endpoint resolution(135), DstPort=49207, PayloadLen=0, Seq=927754047, Ack=2993966752, Win=6467 (scale factor 0x0) = 6467 {TCP:234, IPv4:39}
1164 7:13:31 PM 10/30/2013 17.9179916 ContosoCA1.Contoso.net WINDOWS7PC.Contoso.net MSRPC MSRPC:c/o Bind Ack:  Call=0x3  Assoc Grp=0x7F73  Xmit=0x16D0  Recv=0x16D0  {MSRPC:236, TCP:234, IPv4:39}
On the 2nd RPC session the answer from the CA never reaches the client machine –

1165 7:13:31 PM 10/30/2013 17.9182928 WINDOWS7PC.Contoso.net ContosoCA1.Contoso.net MSRPC MSRPC:c/o Alter Cont: IRemoteSCMActivator(DCOM)  UUID{000001A0-0000-0000-C000-000000000046}  Call=0x3  {MSRPC:236, TCP:234, IPv4:39}
1166 7:13:31 PM 10/30/2013 17.9185225 ContosoCA1.Contoso.net WINDOWS7PC.Contoso.net TCP TCP:Flags=…A…., SrcPort=DCE endpoint resolution(135), DstPort=49207, PayloadLen=0, Seq=927754283, Ack=2993966972, Win=64640 (scale factor 0x0) = 64640 {TCP:234, IPv4:39}
1167 7:13:31 PM 10/30/2013 17.9191750 ContosoCA1.Contoso.net WINDOWS7PC.Contoso.net MSRPC MSRPC:c/o Alter Cont Resp:  Call=0x3  Assoc Grp=0x7F73  Xmit=0x16D0  Recv=0x16D0  {MSRPC:236, TCP:234, IPv4:39}
1168 7:13:31 PM 10/30/2013 17.9211960 WINDOWS7PC.Contoso.net ContosoCA1.Contoso.net DCOM DCOM:RemoteCreateInstance Request {MSRPC:236, TCP:234, IPv4:39}  <–THIS IS NOT ANSWERED

The very next frame is an acknowledgement that the CA received the RemoteCreateInstance Request

1169 7:13:31 PM 10/30/2013 17.9215874 ContosoCA1.Contoso.net WINDOWS7PC.Contoso.net TCP TCP:Flags=…A…., SrcPort=DCE endpoint resolution(135), DstPort=49207, PayloadLen=0, Seq=927754388, Ack=2993967960, Win=63652 (scale factor 0x0) = 63652 {TCP:234, IPv4:39}

Resolution

We have the CISCO ASA device in the Middle, which is more than just routing packets. The fact is that this device IS modifying and / or dropping packets sent to those 2 IP ‘s.
Packets departing from client and server with TTL 128 (Standard) and end up with 254, when in fact this TTL should be decremented by every router between the machine.

The customer had CISCO ASA device between all the client computers and the Enterprise Issuing CA.  On the CISCO ASA they had a rule, setup to inspect all RPC traffic and to drop anything not compliant with RPC based strictly on the RFC for the protocol.

More Information

A double sided trace from the CA and the client computer showed the behavior and the necessary packets leaving the CA and not reaching the client computer.

 

 

Main troubleshoot/debug actions which have been done :

  • Check/Validation of the autoenrollment properties that are stored under the key : « HKLM\Software\Microsoft\Cryptography\Autoenrollment”
  • Purge all the  autoenrollment directory cache information for the “computer”  : To manually force a new download, delete the below registry key and all subordinate keys on the test machine : HKLM\SOFTWARE\Microsoft\Cryptography\AutoEnrollment\AEDirectoryCache
  •  Identification of a new loglevel registry key :

                    [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Cryptography\AutoEnrollment]

                    “LogLevel”=dword:4

  •  Check/Validate of certificate template properties that are cached by each machine (specially the “msPKI-Enrollment-Flag” setting related to the template “WorkstationAuthentication”)
  • Purge all the certificate template directory cache information for “computer”  : To manually force a new download, delete the below registry key and all subordinate keys on the test machine : HKLM\SOFTWARE\Microsoft\Cryptography\CertificateTemplateCache
  • Check/validate the properties of the system task related to the “certificate services client” (that is responsible for  the autoenrollmennt)
  • Debug the mechanism of launching the task schedule by collecting TTTRACER dumps of :
  • The process SVCHOST where the “schedule” service runs:   Illustration : tttracer -dumpfull -attach <pid-schedule>
  • The process TASKHOST where the system task related to auto-enrollment is hosted …Illustration of the used command line : tttracer -dumpfull -out c:\ms -onlaunch taskhost.exe
  • Identification of the inter-process communication used by these two processes which is ALPC on the “ubpmtaskhostchannel” interface
  • Recall of the new Unified Background Processing Model (UBPM):
      The UBPM is hosted in-process by the clients that utilize its services

The UBPM infrastructure has two user-mode components:

ubpm.dll – loaded in Services.exe and the Svchost.exe instance that hosts the Task Scheduler service

Taskhost.exe (Host Process for Windows Tasks) – used to host UBPM-managed scheduled tasks

The UBPM is an RPC server that provides two sets of APIs

One set for trigger providers, such as the PnP Manager

One set for trigger consumers, or clients, such as the Service Control Manager and Task Scheduler service

The UBPM starts an Event Tracing for Windows (ETW) session

Enables each registered provider GUID in the session

Implements an ETW real-time consumer that listens for the pre-registered trigger provider events and notifies the trigger consumers when they occur

UBPM is not designed to be present in minwin

UBPM cannot be directly loaded by Services.exe

For non-minwin platforms, Services.exe loads scext.dll which then loads ubpm.dll

See http://blogs.technet.com/b/askperf/archive/2009/10/04/windows-7-windows-server-2008-r2-unified-background-process-manager-ubpm.aspx

 

 

 

  • Delete the Volatile- keys… the keys are removed automatically at the end of autoenrollment process (it means, at the end of the DCOM/RPC transaction between the client computer and the issuing CA).

To delete the temporary keys, use the reg.exe utility,

reg delete “HKLM\Software\Microsoft\Volatile-KeyRoam-EXCLUSIVE” /f

                      reg delete “HKLM\Software\Microsoft\Volatile-AutoEnroll-EXCLUSIVE” /f

 

To solve this problem finally we have two solutions:

– modify the behavior of the Firewall and the way to filter  DCE (MSRPC) inspection filter,

– Or use Web enrollment service (CEP/CES) (“requires two dedicated Web servers for CEP and CES”)

– Or use a script to replace manual certificate request using the MMC snap-in (Certificates\Local computer)

 Note: important, use the certreq.exe from Windows 8.1 – v6.3.9600.16384

1)      Script for a client workstation – to request a client computer certificate (typically used by a VPN software):

certreq -enroll -machine -policyserver “ldap: ”  -q “WorkstationAuthentication”

2)     Script for a domain controller – to request the KerberosAuthentication template (for LDAPS):

certreq -enroll -machine -policyserver “ldap: ”  -q “KerberosAuthentication”

 

ANNEX – Useful tools:

    • Tool Description
      DCDiag Analyzes the state of domain controllers.
      Event Viewer Displays logged events.
      Gpotool Determines whether the policies are valid and consistent.
      NetDiag Used to test network connectivity.
      Network Monitor Monitors and captures network traffic.
      Ntdsutil Provides management facilities for Active Directory.
      PortQry, PortQueryUI Used for TCP/IP connectivity testing.
      Repadmin Diagnoses replication problems between Windows DCs.
      RPCDump Typically used together with Network Monitor.
      RPCPing Used to confirm RPC connectivity.