Category: System and Network Admins


  • Microsoft Security Response Center: Protection, detection, and response.
  • Malware Protection Center: The Microsoft Malware Protection Center (MMPC) provides world class antimalware research and response capabilities that support Microsoft’s range of security products and services. With laboratories in multiple locations around the globe the MMPC is able to respond quickly and effectively to new malicious and potentially unwanted software threats wherever and whenever they arise.
  • My Bulletins: Is an online tool that provides you with a personalized list of the Microsoft security bulletins that matter most to you. Support for My Bulletins is being deprecated as Microsoft moves to simplify the update terminology and process. Going forward please use the Security Update Guide (SUG) for a more relevant and customizable way to track updates.
  • Security Update Guide (SUG): The Security Update Guide is the authoritative source of information on our security updates. It provides guidance, response, bulletin and advisory information and you can even get Security Update information through the API.
  • CSS YouTube Channel for customers: Now customers get a seamless self-service experience across our support sites and social channels. Check out the new CSS channel on YouTube – view it, like it, subscribe to it. The more activity, the easier it is for customers to find us when they need our help.
  • Blog MS Monthly: Blog MS consolidates a large number of highly relevant and up to date information sources across the Microsoft product and online services portfolio. You can expect to find important announcements and details of Microsoft news, product releases, service packs and important support issues.
  • Microsoft Premier Support: The Microsoft Premier Support (MPS) site is a secure Web site for the exclusive use of Premier Support customers. Note some of the links below may only be available to Premier Support customers. If you currently have Premier Support and don’t have access to MPS, please let your TAM know. Your TAM can get you access to the site.
  • Premier Services Twitter: Microsoft Premier Services is now on Twitter!  This account will share new proactive offerings and product information, links to important articles on TechNet or Microsoft Blogs, important lifecycle updates, and security alerts.  Start following us today!
  • otection, detection, and response.
  • Malware Protection Center: The Microsoft Malware Protection Center (MMPC) provides world class antimalware research and response capabilities that support Microsoft’s range of security products and services. With laboratories in multiple locations around the globe the MMPC is able to respond quickly and effectively to new malicious and potentially unwanted software threats wherever and whenever they arise.
  • My Bulletins: Is an online tool that provides you with a personalized list of the Microsoft security bulletins that matter most to you. Support for My Bulletins is being deprecated as Microsoft moves to simplify the update terminology and process. Going forward please use the Security Update Guide (SUG) for a more relevant and customizable way to track updates.
  • Security Update Guide (SUG): The Security Update Guide is the authoritative source of information on our security updates. It provides guidance, response, bulletin and advisory information and you can even get Security Update information through the API.
  • CSS YouTube Channel for customers: Now customers get a seamless self-service experience across our support sites and social channels. Check out the new CSS channel on YouTube – view it, like it, subscribe to it. The more activity, the easier it is for customers to find us when they need our help.
  • Blog MS Monthly: Blog MS consolidates a large number of highly relevant and up to date information sources across the Microsoft product and online services portfolio. You can expect to find important announcements and details of Microsoft news, product releases, service packs and important support issues.
  • Microsoft Premier Support: The Microsoft Premier Support (MPS) site is a secure Web site for the exclusive use of Premier Support customers. Note some of the links below may only be available to Premier Support customers. If you currently have Premier Support and don’t have access to MPS, please let your TAM know. Your TAM can get you access to the site.
  • Premier Services Twitter: Microsoft Premier Services is now on Twitter!  This account will share new proactive offerings and product information, links to important articles on TechNet or Microsoft Blogs, important lifecycle updates, and security alerts.  Start following us today!

 

Advertisements

Security News!

Security News:

 

2019/06/11:

Two critical vulnerabilities in Microsoft’s NTLM authentication protocol consisting of three logical flaws make it possible for attackers to run remote code and authenticate on machines running any Windows version.

https://www.bleepingcomputer.com/news/security/microsoft-ntlm-flaws-expose-all-windows-machines-to-rce-attacks/

 

2019/03/11:

Operating Systems can be detected using Ping Command, Ping is a computer network administration software utility, which used to find the Availability of a host on an Internet Protocol (IP) network.

https://gbhackers.com/operating-systems-can-be-detected-using-ping-command/

 

2019/02/22:

Corporate firewalls can block reverse and bind TCP connections.However, corporate firewalls are behind internal networks.So we can use PING ICMP Shell:

https://gbhackers.com/icmp-shell-secret/

 

2019/02/03:

Exploiting Malwarebytes antimalware!

https://acru3l.github.io/2019/02/02/exploiting-mb-anti-exploit/

 

2019/01/24: redteam

Koadic, or COM Command & Control, is a Windows post-exploitation rootkit similar to other penetration testing tools such as Meterpreter and Powershell Empire. The major difference is that Koadic does most of its operations using Windows Script Host (a.k.a. JScript/VBScript):

https://github.com/zerosum0x0/koadic

 

 

 

A collection of security articles and web sites, KB, tips and tricks especially for System and Network Administrators, DevOps, Pentesters or Security Researchers.

https://github.com/trimstray/the-book-of-secret-knowledge

 

hacking web sites:

https://thehackernews.com/

https://www.bleepingcomputer.com/

https://www.zataz.com/

 

Passwords databases:

https://haveibeenpwned.com/

https://www.dehashed.com/

https://ghostproject.fr/

https://leaksify.com/

 

The Cyber Swiss Army Knife – a web app for encryption, encoding, compression and data analysis:

https://gchq.github.io/CyberChef/

 

 

 

Basic network capture methods: https://blogs.technet.microsoft.com/askpfeplat/2016/12/27/basic-network-capture-methods/

  1. Network Monitor 3.4 (Netmon) – https://www.microsoft.com/en-us/download/details.aspx?id=4865 (NOTE: Network Monitor is no longer under active development)
  2. Wireshark (v 2.2.2 as of 11/16/16) – https://wireshark.org/#download
  3. Netsh Trace – built-in to operating system
  4. Microsoft Message Analyzer (MMA) (v 1.4 as of 6/13/16) – https://www.microsoft.com/en-us/download/details.aspx?id=44226

Message analyzer operating guide: http://technet.microsoft.com/en-us/library/jj649776.aspx

How to message analyzer on YouTube: https://www.youtube.com/watch?v=e0v0RsQVdT8

As you might guess from the name, Message Analyzer is much more than a network sniffer or packet tracing tool.  Key capabilities include:

  • Integrated “live” event and message capture at various system levels and endpoints (client and server remotely !)
  • Remote capture (capture multiple point concurrently)
  • Parsing and validation of protocol messages and sequences
  • Automatic parsing of event messages described by ETW manifests
  • Summarized grid display – top level is  “operations”, (requests matched with responses)
  • User controlled “on the fly” grouping by message attributes
  • Ability to browse for logs of different types (.cap, .etl, .txt) and import them together
  • Automatic re-assembly and ability to render payloads
  • Ability to import text logs, parsing them into key element/value pairs
  • Support for “Trace Scenarios” (one or more message providers, filters, and views)

Other articles:

Use message analyzer to convert a .etl to .cap: https://blogs.msdn.microsoft.com/benjaminperkins/2018/03/09/analyze-netsh-traces-with-wireshark-or-network-monitor/

 

Capture a network trace using netsh:

https://blogs.msdn.microsoft.com/benjaminperkins/2018/03/09/capture-a-netsh-network-trace/

 

  1. To learn more about your nmcap options, enter “nmcap /?” or “nmcap /examples”
  2. Wireshark training can be found at https://www.wireshark.org/#learnWS.
  3. For more information on Message Analyzer, check out the blog at https://blogs.technet.microsoft.com/messageanalyzer/.
  4. Message Analyzer training videos can be found at https://www.youtube.com/playlist?list=PLszrKxVJQz5Uwi90w9j4sQorZosTYgDO4.
  5. Message Analyzer Operating Guide – https://technet.microsoft.com/en-us/library/jj649776.aspx
  6. Information on the Message Analyzer PowerShell module can be found at https://technet.microsoft.com/en-us/library/dn456518(v=wps.630).aspx.
  7. Remote captures with MMA – https://blogs.technet.microsoft.com/messageanalyzer/2013/10/17/remote-capture-with-message-analyzer-and-windows-8-1/

Windows Admin Center:

https://docs.microsoft.com/en-us/windows-server/manage/windows-admin-center/understand/windows-admin-center

Installation:

You can install Windows Admin Center on the following Windows operating systems:

Version Installation mode
Windows 10, version 1709 or newer Desktop mode
Windows Server Semi-Annual Channel Gateway mode
Windows Server 2016 Gateway mode
Windows Server 2019 Gateway mode

Desktop mode: Launch from the Start Menu and connect to the Windows Admin Center gateway from the same computer on which it’s installed (i.e. https://localhost:6516)

Gateway mode: Connect to the Windows Admin Center gateway from a client browser on a different machine (i.e. https://servername.contoso.com)

Hi folks, here are web resources to implement and  troubleshoot MS DFS and MS DFS-R:

DFS Replication in Windows Server 2012 R2 : http://blogs.technet.com/b/filecab/archive/2013/08/20/dfs-replication-in-windows-server-2012-r2-if-you-only-knew-the-power-of-the-dark-shell.aspx

DFS Replication Initial Sync in Windows Server 2012 R2: http://blogs.technet.com/b/filecab/archive/2013/08/21/dfs-replication-initial-sync-in-windows-server-2012-r2-attack-of-the-clones.aspx

DFS Replication in Windows Server 2012 R2: Restoring Conflicted, Deleted and PreExisting files with Windows PowerShell: http://blogs.technet.com/b/filecab/archive/2013/08/23/dfs-replication-in-windows-server-2012-r2-restoring-conflicted-deleted-and-preexisting-files-with-windows-powershell.aspx

Understanding DFS (how it works):

https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2003/cc782417(v=ws.10)

=> Several mechanisn are used: routing, DNS, AD sites and subnets topology, WINS,  FW ports and rules shoud be open (RPC, SMB…):

NetBIOS Name Service:  Domain controllers; root servers that are not domain controllers; servers acting as link targets; client computers acting as link targets: TCP/UDP 137

NetBIOS Datagram Service: Domain controllers; root servers that are not domain controllers; servers acting as link targets; client computers acting as link targets: TCP/138

NetBIOS Session Service: Domain controllers; root servers that are not domain controllers; servers acting as link targets; client computers acting as link targets: TCP/139

LDAP Server: Domain controllers TCP/UDP 389

Remote Procedure Call (RPC) endpoint mapper: Domain controllers TCP/135

Server Message Block (SMB): Domain controllers; root servers that are not domain controllers; servers acting as link targets; client computers acting as link targets: TCP/UDP 445

Extract from the MS technet: “When a client requests a referral from a domain controller, the DFS service on the domain controller uses the site information defined in Active Directory (through the DSAddressToSiteNames API) to determine the site of the client, based on the client s IP address. DFS stores this information in the client site cache”
“DFS clients store root referrals and link referrals in the referral cache (also called the PKT cache). These referrals allow clients to access the root and links within a namespace. You can view the contents of the referral cache by using Dfsutil.exe with the /pktinfo “
“You can view the domain cache on a client computer by using the Dfsutil.exe command-line tool with the /spcinfo parameter”

Implementing DFS-R: http://technet.microsoft.com/en-us/library/cc770925.aspx AND DFS-R FAQ: http://technet.microsoft.com/en-us/library/cc773238.aspx, delegate DFS-R permissions: http://technet.microsoft.com/en-us/library/cc771465.aspx

DFS-R limits:

The following list provides a set of scalability guidelines that have been tested by Microsoft on Windows Server 2012 R2:
• Size of all replicated files on a server: 100 terabytes.
• Number of replicated files on a volume: 70 million.
• Maximum file size: 250 gigabytes.

The following list provides a set of scalability guidelines that have been tested by Microsoft on Windows Server 2012, Windows Server 2008 R2, and Windows Server 2008:
• Size of all replicated files on a server: 10 terabytes.
• Number of replicated files on a volume: 11 million.
• Maximum file size: 64 gigabytes

Implementing DFS Namespace: http://technet.microsoft.com/en-us/library/cc730736.aspx AND DFS-N FAQ: http://technet.microsoft.com/fr-fr/library/ee404780(v=ws.10).aspx

Consolidation of multiple DFS namespaces in a single one : http://blogs.technet.com/b/askds/archive/2013/02/06/distributed-file-system-consolidation-of-a-standalone-namespace-to-a-domain-based-namespace.aspx

Netmon trace digest: http://blogs.technet.com/b/josebda/archive/2009/04/15/understanding-windows-server-2008-dfs-n-by-analyzing-network-traces.aspx

SMB and Wireshark: https://thebackroomtech.com/2019/05/22/using-wireshark-to-sniff-an-smb-transmission/

DFS and Wireshark: https://www.youtube.com/watch?v=9sLUVQPriOk

DFS 2008 step by step: http://technet.microsoft.com/en-us/library/cc732863(WS.10).aspx

DFS tuning and troubleshooting:

How to troubleshoot Distributed File System Namespace access failures in Windows: https://support.microsoft.com/en-au/help/975440/how-to-troubleshoot-distributed-file-system-namespace-access-failures

FS-N et DFS-R en ligne de commande: http://blogcastrepository.com/blogs/benoits/archive/2009/08/22/dfs-n-et-dfs-r-en-ligne-de-commande.aspx

DFSR les commandes les plus utiles: http://www.monbloginfo.com/2011/03/02/dfsr-les-commandes-les-plus-utiles/

and http://blogs.technet.com/b/filecab/archive/2009/05/28/dfsrdiag-exe-replicationstate-what-s-dfsr-up-to.aspx

Tuning DFS: http://technet.microsoft.com/en-us/library/cc771083.aspx and Tuning DFS Replication performance : http://blogs.technet.com/b/askds/archive/2010/03/31/tuning-replication-performance-in-dfsr-especially-on-win2008-r2.aspx

DFSutil command line:  http://technet.microsoft.com/fr-fr/library/cc776211(v=ws.10).aspx AND http://technet.microsoft.com/en-us/library/cc779494(v=ws.10).aspx and https://technet.microsoft.com/en-us/library/cc776211%28WS.10%29.aspx

Performance tuning guidelines for Windows 2008 R2: http://msdn.microsoft.com/en-us/windows/hardware/gg463392.aspx

Monitoring:

DFSRMon utility: http://blogs.technet.com/b/domaineetsecurite/archive/2010/04/14/surveillez-en-temps-r-el-la-r-plication-dfsr-gr-ce-dfsrmon.aspx

or  DfsrAdmin.exe in conjunction with Scheduled Tasks to regularly generate health reports: http://go.microsoft.com/fwlink/?LinkId=74010

Server side:

DFS: some notions: A referral is an ordered list of targets that a client computer receives from a domain controller or namespace server when the user accesses a namespace root or folder with targets. After the client receives the referral, the client attempts to access the first target in the list. If the target is not available, the client attempts to access the next target.

tip1) dfsutil domain : Displays all namespaces in the domain ; dfsutil /domain:mydomain.local /view

tip2) You can check the size of an existing DFS namespace by using the following syntax in Dfsutil.exe:

dfsutil /root:\\mydomain.local\rootname /view (for domain-based DFS)
dfsutil /root:\\dfsserver\rootname /view (for stand-alone DFS)

tip3) Enabling the insite setting of a DFS server is useful when: You don’t want the DFS clients to connect outside the site.
You don’t want the DFS client to connect to a site other than the site it is in, and hence avoid using expensive WAN links.
dfsutil /insite:\\mydomain.local\dfsroot /enable

tip4) You want DFS clients to be able to connect outside the internal site, but you want clients to connect to the closest site first, saving the expensive network bandwidth:

ex: dfsutil /root:\\mydomain.local\sales /sitecosting /view or /enable or /disable

If you do not know if a root is site costing aware, you can check its status by substituting the /display parameter for the /sitecosting parameter.

tip5) Enable root scalability mode: You enable root scalability mode by using the /RootScalability parameter in Dfsutil.exe, which you can install from the \Support\Tools folder on the Windows Server 2003 operating system CD. When root scalability mode is enabled,  DFS root servers get updates from the closest domain controller instead of the server acting as the PDC emulator master.
As a result, root scalability mode reduces network traffic to the PDC emulator master at the expense of faster updates  to all root servers. (When you make changes to the namespace, the changes are still made on the PDC emulator master,  but the root servers no longer poll the PDC emulator master hourly for those changes; instead, they poll the closest domain controller.)
With this mode enabled, you can have as many root targets as you need, as long as the size of the DFS Active Directory object (for each root)  is less than 5 MB. Do not use root scalability mode if any of the following conditions exist in your organization:Your namespace changes frequently, and users cannot tolerate having inconsistent views of the namespace.  Domain controller replication is slow. This increases the amount of time it takes for the PDC emulator master  to replicate DFS changes to other domain controllers, which, in turn, replicate changes to the root servers.  Until this replication completes, the namespace will be inconsistent on all root servers.

ex: dfsutil /root:\\mydomain.local\sales /rootscalability /view or /enable or /disable

tip6) Dfsdiag utility: http://blogs.technet.com/b/filecab/archive/2008/10/24/what-does-dfsdiag-do.aspx

/testdcs: With this you can check the configuration of the domain controllers. It performs the following tests:

  • Verifies that the DFS Namespace service is running on all the DCs and its Startup Type is set to Automatic.
  • Check for the support of site-costed referrals for NETLOGON and SYSVOL.
  • Verify the consistency of site association by hostname and IP address on each DC.

To run this command against your domain mydomain.local just type:

DFSDiag /testdcs /domain:mydomain.local

DFSDiag /testdcs > dfsdiag_testdcs.txt

/testsites: Used to check the configuration of Active Directory Domain Services (AD DS) sites by verifying that servers that act as namespace servers or folder (link) targets have the same site associations on all domain controllers.

So for a machine you will be running something like: DFSDiag /testsites /machine:MyDFSServer

For a folder (link): DFSDiag /testsites /dfspath:\\mydomain.local\MyNamespace\MyLink /full

For a root: DFSDiag /testsites /dfspath:\\mydomain.local\MyNamespace /recurse /full

/testdfsconfig:  With this you can check the DFS namespace configuration. The tests that perform are:

  • Verifies that the DFS Namespace service is running and that its Startup Type is set to Automatic on all namespace servers.
  • Verifies that the DFS registry configuration is consistent among namespace servers.
  • Validates the following dependencies on clustered namespace servers that are running Windows 2008 (non supported for W2K3 clusters L):
    • Namespace root resource dependency on network name resource.
    • Network name resource dependency on IP address resource.
    • Namespace root resource dependency on physical disk resource.

To run this you just need to type:  DFSDiag /testdfsconfig /dfsroot:\\mydomain.local\MyNamespace

/testdfsintegrity: Used to check the namespace integrity. The tests performed are:

  • Checks for DFS metadata corruption or inconsistencies between domain controllers
  • In Windows 2008 server, validates that the Access Based Enumeration state is consistent between DFS metadata and the namespace server share.
  • Detect overlapping DFS folders (links), duplicate folders and folders with overlapping folder targets (link targets).

To check the integrity of your domain mydomain.local:

DFSDiag /testdfsintegrity /dfsroot:\\mydomain.local\MyNamespace

DFSDiag.exe /testdfsintegrity /dfsroot:\\mydomain.local\MyNamespace /recurse /full > dfsdiag_testdfsintegrity.txt

Additionally you can specify /full, /recurse, which in this case, /full verifies the consistency of share and NTFS ACLs in all the folder targets. It also verifies that the Online property is set in all the folder targets. /recurse performs the testing including the namespace interlinks.

/testreferral: Perform specific tests, depending on the type of referral being used.

  • For Trusted Domain referrals, validates that the referral list includes all trusted domains.
  • For Domain referrals, perform a DC health check as in /testdcs
  • For Sysvol and Netlogon referrals perform the validation for Domain referrals and that it’s TTL has the default value (900s).
  • For namespace root referrals, perform the validation for Domain referrals, a DFS configuration check (as in /testdfsconfig) and a Namespace integrity check (as in /testdfsintegrity).
  • For DFS folder referrals, in addition to performing the same health checks as when you specify a namesapace root, this command validates the site configuration for folder target (DFSDiag /testsites) and validates the site association of the local host

Again for your namespace mydomain.local:

DFSDiag /testreferral /dfspath:\\mydomain.local\MyNamespace

DFSDiag.exe /testreferral /dfspath:\\mydomain.local\MyNamespace /full > dfsdiag_testreferral.txt

There is also the option to use /full as an optional parameter, but this only applies to Domain and Root referrals. In these cases /full verifies the consistency of site association information between the registry and Active Directory.

Domain controllers:

Evaluate domain controller health, site configurations, FSMO ownerships, and connectivity:

Use Dcdiag.exe to check if domain controllers are functional. Review this for comprehensive details about dcdiag:

    Dcdiag /v /f:Dcdiag_verbose_output.txt

    Dcdiag /v /test:dns /f:DCDiag_DNS_output.txt

    Dcdiag /v /test:topology /f:DCDiag_Topology_output.txt

Active Directory replication

If DCDiag finds any replication failures and you need additional details about them, Ned wrote an excellent article a while back that covers how to use the Repadmin.exe utility to validate the replication health of domain controllers:

    Repadmin /replsummary * > repadmin_replsummary.txt

    Repadmin /showrepl * > repadmin_showrepl.txt

Always validate the health of the environment prior to utilizing a namespace.

Clients:

  • dfsutil /root:\\mydomain.local\myroot /view /verbose    ; display the content of root dfs (links…)
  • dfsutil /pktinfo     ;to display the client cache
  • dfsutil /spcinfo     ; the domain cache on a client computer
  • dfsutil /purgemupcache ; cache stores information about which redirector, such as DFS, SMB, or WebDAV, is required for each UNC path
  • dfsutil /pktflush   ; Dfsutil /PktFlush is a special problem repair command that should only be executed on the client.
  • dfsutil cache referral flush   ; to flush the client cache
  • dfsdiag /testdfsintegrity /dfsroot:\\mydomain.local\dfsroot /recurse /full > dfsdiag_testdfsintegrity.txt   ; to test root dfs from local client
  • dfsutil client siteinfo <ip> ; to display the remote client AD client site
  • dfsutil /sitename:<ip address>  or nltest /dsgetsite ; to display the local AD client site
  • To display a target dfs (primary/active) from cmd line: dfsutil client property state \\mydomain.local\dfsroot\dfsfolder1
  • To change a target dfs (primary/active) from cmd line: dfsutil client property state active \\mydomain.local\dfsroot\dfsfolder1 \\server.mydomain2.net\share  ; but you need a special hotfix for win7/2008r2 clients: http://support.microsoft.com/kb/2783031/en-us
  • Understanding: DFS override referral ordering: http://blogs.technet.com/b/askds/archive/2011/10/27/dfs-override-referral-ordering-messing-with-the-natural-order.aspx

Dfsutil examples: https://technet.microsoft.com/en-us/library/cc776211%28WS.10%29.aspx

 

How to monitor LDAP, NTLM, Kerberos to your domain controllers ?

http://blogs.technet.com/b/askpfeplat/archive/2013/12/16/domain-and-dc-migrations-how-to-monitor-ldap-kerberos-and-ntlm-traffic-to-your-domain-controllers.aspx

Troubleshooting high LSASS CPU ?

http://blogs.technet.com/b/askds/archive/2007/08/20/troubleshooting-high-lsass-cpu-utilization-on-a-domain-controller-part-1-of-2.aspx

Root cause:

The root cause of LSASS CPU% peaks could be multiple:

  • Identify circular nested groups in the domain (=> https://gallery.technet.microsoft.com/scriptcenter/fa4ccf4f-712e-459c-88b4-aacdb03a08d0 )
  • Removal of Cipher protocols on DCs (=> use IISCrypto from Nartac software to remediate)
  • Malformed LDAP query on Applications (Linux-Unix-Java based)
  • LDAP configuration problems on Applications (Linux/Java based)
  • conf, sssd.conf, … config problems on Linux/Unix
  • Local Antivirus running on the Domain controllers is not well configured to exclude DC folders and files (NTDS, Sysvol…)
  • Centrify configuration settings not optimized on Linux/Unix
  • Centrify ZPA not well configured
  • Vmware Vcenter not well integrated to a windows domain
  • Vmware ESX not well integrated to a windows domain
  • Storage appliances not well integrated to a windows domain

DC fails logons or experiences LDAP timeouts:

http://blogs.msdn.com/b/spatdsg/archive/2011/06/24/dc-fails-logons-or-experiences-ldap-timeouts.aspx

https://blogs.technet.microsoft.com/askds/2014/10/24/understanding-atq-performance-counters-yet-another-twist-in-the-world-of-tlas/

Tips and tricks:

  • Identify missing subnets and add them on dssite.msc
  • Add more CPU and RAM on domain controllers
  • Move to 2012 R2 domain controllers
  • Disable Netbios on the DC but this may not be an option for everyone, so the site subnet mapping or DNS name resolution should also fix this kind of an issue.
  • Educate developers to perform the right LDAP queries
  • Configure client applications properly (ldap filters)
  • We have seen the LDAP ATQ threads get depleted at a customer due to high volume of LDAP clients using NTLM for authentication. These were overloading the Netlogon service, ran into MaxConcurrentApi bottleneck.
  • By default there are 4 threads per processor allocated to the LDAP thread pool, we can change that via LDAP policies, specifically MaxPoolThreads: MaxPoolThreads = Maximum number of threads created by the domain controller for query execution (4 per processor). Set to 8 per proc.

Enable LDAP query logging using NTDS diagnostic values:

http://technet.microsoft.com/en-us/library/cc961809.aspx

https://blogs.technet.microsoft.com/askpfeplat/2015/05/10/how-to-find-expensive-inefficient-and-long-running-ldap-queries-in-active-directory/

with PowerShell script: https://gallery.technet.microsoft.com/scriptcenter/Event-1644-reader-Export-45205268

https://community.spiceworks.com/how_to/68929-how-to-trace-ldap-calls-generated-by-a-specific-software

http://www.frickelsoft.net/blog/?p=243

Basically, you want to set the following registry values:

Path: HKLM\SYSTEM\CurrentControlSet\Services\NTDS\Diagnostics\15 Field Engineering
Type: DWORD
Value: 5

Path: HKLM\SYSTEM\CurrentControlSet\Services\NTDS\Parameters\Expensive Search Results Threshold
Type: DWORD
Value: 10000  (decimal – default value)

Path: HKLM\SYSTEM\CurrentControlSet\Services\NTDS\Parameters\Inefficient Search Results Threshold
Type: DWORD
Value: 1000  (decimal – default value)

Path: HKLM\SYSTEM\CurrentControlSet\Services\NTDS\Parameters\Search Time Threshold
Type: DWORD
Value: 30000  (decimal – defaut value in milliseconds)

Explanations:

Expensive LDAP calls are the searches those visit large number of entries. Default threshold for expensive search is 10,000 which means if an LDAP call visit 10,000 or more entries then it will be consider as an expensive call. Once you find such call in logs, you can figure out possible solutions to optimize it. For example a query (displayName=*John*) on root domain container will visit all objects in the domain those have any value available in displayName attribute and it will be consider an expensive call if there are 10,000 or plus such objects those have displayName attribute populated.

Inefficient LDAP calls are the searches those return less than 10% of visited entries. For example, if a query visit 10,000 entries in active directory but only return 100 entries then it will be consider inefficient query as return entries are less than 10% of total visited entries. Default visited entries threshold limit for inefficient query is 1,000 which means if a query visit less than 1000 entries then it will not be consider inefficient query even though if it return no entry.

Search Time Threshold, is available only if 2012 R2 DC or after you install the KB 2800945 installed on Server 2012, Server 2008 R2 or Server 2008 domain controllers. By default the value is 30000 milliseconds = 30 seconds ! too long and I recommend to set up to 5000 (5 secs)

These registry changes do not require a reboot but are set per server, so implementing for an entire forest/domain would best be done via Group Policy Preferences. Once set you will find the resulting logs in the Directory Service event log on the DC. They are not exactly parse-friendly but can be wrangled with some regex. The best part is it requires no external utilities/code. Because it is very verbose, don’t forget to remove those values after audit phase.

Which Tools to help?

Creating More Efficient Active Directory-Enabled Applications:

https://msdn.microsoft.com/en-us/library/ms808539.aspx

Web Resources:

http://blogs.msdn.com/b/spatdsg/archive/2011/06/24/dc-fails-logons-or-experiences-ldap-timeouts.aspx

https://blogs.technet.microsoft.com/askds/2014/10/24/understanding-atq-performance-counters-yet-another-twist-in-the-world-of-tlas/

http://technet.microsoft.com/en-us/library/cc961809.aspx

https://blogs.technet.microsoft.com/askpfeplat/2015/05/10/how-to-find-expensive-inefficient-and-long-running-ldap-queries-in-active-directory/

https://gallery.technet.microsoft.com/scriptcenter/Event-1644-reader-Export-45205268

http://social.technet.microsoft.com/wiki/contents/articles/9759.configuring-maxconcurrentapi-for-ntlm-pass-through-authentication.aspx

http://blogs.technet.com/b/askpfeplat/archive/2013/12/16/domain-and-dc-migrations-how-to-monitor-ldap-kerberos-and-ntlm-traffic-to-your-domain-controllers.aspx

http://www.nextdirectiontech.com/samples/WhosUsingMyDirectory.pdf

http://serverfault.com/questions/280042/how-do-you-trace-debug-ldap-connections-against-active-directory

https://support.quest.com/SolutionDetail.aspx?id=SOL72639&pr=

http://codeidol.com/community/ad/enabling-inefficient-and-expensive-ldap-query-logg/2294/

http://www.frickelsoft.net/blog/?p=243

https://blogs.technet.com/b/ad/archive/2009/07/06/referral-chasing.aspx

http://support.microsoft.com/kb/2550044

http://technet.microsoft.com/en-us/library/cc961809.aspx  for more on enabling diagnostics logging.

http://msdn.microsoft.com/en-us/magazine/cc163437.aspx : Debugging And Performance Tuning With ETW

http://technet.microsoft.com/en-us/library/cc749337.aspx : creating Data Collector Sets

http://books.google.ie/books?id=_1XZwfwzXo8C&pg=PA443&lpg=PA443&dq=ntds+15+field+engineering&source=bl&ots=AQwFEofd3M&sig=QwUD6BsywLtcmp_NV2hQlgtwsHE&hl=fr&sa=X&ei=02Q5U5nrHuzP0AW8toH4Ag&ved=0CE0Q6AEwBg#v=onepage&q=ntds%2015%20field%20engineering&f=false

http://blogs.technet.com/b/askds/archive/2012/02/02/purging-old-nt-security-protocols.aspx

 

Using site-to-site VPN gateway can provide better continuity for your work in hybrid cloud setup with Azure. This post will explain how to set up site-to-site VPN Gateway to enable this.

Requirements:

Before start make sure you have following in place.

1) VPN device: A VPN device is needed on-premise to create the VPN connection with Azure. A list of supported list of devices can found here.

2) Static Public IP address: The VPN device should have external public IP address and it shouldn’t be NAT.

3) Valid Azure Subscription

 

How to:

https://blogs.technet.microsoft.com/canitpro/2017/06/28/step-by-step-configuring-a-site-to-site-vpn-gateway-between-azure-and-on-premise/

https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-about-vpngateways

 

https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-howto-site-to-site-resource-manager-portal

Learn about Windows Server 2019

what’s new: https://docs.microsoft.com/en-us/windows-server/get-started-19/whats-new-19

With the release of a new version of Windows Server, it’s time to learn about what’s new and try it out. At Ignite, we had tons of sessions and those are available for you on demand. If you want to go deeper on the details, you can find the updated documentation in the Windows Server technical content library.

If you are upgrading from an older version, you can check the new Upgrade Center, where you can find useful information on the upgrade process, as well as pre and post activities.

For those of you already looking ahead, join the Insiders program. We will continue to ship new builds of Windows Server that will first land on the next Semi-Annual Channel and later in the next Long-Term Servicing Channel.

Getting started with Server core:

https://docs.microsoft.com/en-us/windows-server/get-started/getting-started-with-server-core

Very interesting article about Storage Space Direct:

https://docs.microsoft.com/en-us/windows-server/storage/storage-spaces/storage-spaces-direct-overview

 

 

 

Overview:

A very brief summary of how the protocol works: There is an “endpoint mapper” that runs on TCP port 135.
You can bind to that port on a remote computer anonymously and enumerate all the various RPC services
available on that computer.  The services may be using named pipes or TCP/IP.  Named pipes will use port 445.
The services that are using TCP are each dynamically allocated their own TCP ports,
which are drawn from a pool of port numbers. This pool of port numbers is by default 1024-5000 on XP/2003
and below, and 49152-65535 on Vista/2008 and above. (The ephemeral port range.)

You can customize that port range that RPC will use if you wish, like so:

reg add HKLM\SOFTWARE\Microsoft\Rpc\Internet /v Ports /t REG_MULTI_SZ /f /d 5200-10200
reg add HKLM\SOFTWARE\Microsoft\Rpc\Internet /v PortsInternetAvailable /t REG_SZ /f /d Y
reg add HKLM\SOFTWARE\Microsoft\Rpc\Internet /v UseInternetPorts /t REG_SZ /f /d Y

And/Or
netsh int ipv4 set dynamicport tcp start=5200 num=10200
netsh int ipv4 set dynamicport udp start=5200 num=10200
netsh int ipv6 set dynamicport tcp start=5200 num=10200
netsh int ipv6 set dynamicport udp start=5200 num=10200

I found this very interesting article about how to troubleshoot RPC communications:

http://blogs.technet.com/b/askds/archive/2012/01/24/rpc-over-it-pro.aspx

http://social.technet.microsoft.com/wiki/contents/articles/4494.aspx

http://technet.microsoft.com/en-us/magazine/2007.07.howitworks.aspx

Tools:

rpcdump (from old windows service pack)

test-server  ; powershell script here: https://gallery.technet.microsoft.com/scriptcenter/Powershell-Test-Server-e0cdea9a

test-rpc       ; powershell script here:

rpc-ping     ; powershell script here: http://www.zerrouki.com/rpc-ping/

portqry -n computer -e 135

netmon 3.4

rpcping

rpings

 

Introduction

Remote Procedure Call (RPC) is an inter-process communication technique to allow client and server software to communicate on a network. The RPC protocol is based on a client/server model. The client makes a procedure call that appears to be local but is actually run on a remote computer. During this process, the procedure call arguments are bundled and passed through the network to the server. The arguments are then unpacked and run on the server. The result is again bundled and passed back to the client, where it is converted to a return value for the client’s procedure call.

RPC is used by several components in Windows Server, such as the File Replication Service (FRS), Active Directory Replication, Certificate services, DCOM, domain join, DCPromo and RDP, NLB and Cluster, Microsoft Operations Master, Exchange and SQL.

The RPC Server

An RPC server is a communications interface provided by an application or service that allows remote clients to connect, pass commands, and transfer data using the RPC protocol. A typical example of an RPC server is Microsoft Exchange Server. Microsoft Exchange Server is an application running on a computer that supplies an RPC communications interface for an RPC client.

An application will register its RPC server with the operating system’s End Point Mapper (EPM) service so that the remote client can locate the RPC server. When the application registers with the EPM it will indicate the IP address and TCP port that it is listening on.

The RPC Client

An RPC client is an application running on any given computer that uses the RPC protocol to communicate with an RPC server. An example of a typical RPC client is the Microsoft Outlook application.

NOTE: In this document the terms RPC server and RPC client refer to the application running at both ends of an RPC communication.

RPC Quick Fixes

Common causes of RPC errors include:

  • Errors resolving a DNS or NetBIOS name.
  • The RPC service or related services may not be running.
  • Problems with network connectivity.
  • File and printer sharing is not enabled.

Use the following procedures to diagnose and repair common causes of RPC errors.

Unable to resolve DNS or NetBIOS names in an Active Directory environment

  1. Use the following commands to verify DNS is working for all DC’s or specific DC’s:
  • To get a DNS status for all DCs in forest, run the following command:
  • DCDIAG /TEST:DNS /V /E /F:<filename.log>
  • The “/e” switch runs the DNS test against all DCs in an Active Directory Forest

To get DNS health on a single DC, run the command below.

  • DCDIAG /TEST:DNS /V /S:<DCNAME> /F:<filename.log>
  • The “/s:” switch runs the DNS test against a specified domain controller.

To verify that a domain controller can be located for a specific domain, run the command below.

  • NLTEST /DSGETDC:<NetBIOS or DNS domain name>
  1. Servers and clients that are receiving the error should be checked to verify that they are configured with the appropriate DNS server. Servers should not be pointing to their ISP’s DNS servers in the preferred or alternate DNS server portion of the TCP/IP settings. The ISP’s DNS servers should only be used as forwarders in DNS.
  1. Ensure that at least one correct DNS record is registered on each domain controller.
  • To ensure that a correct DNS record is registered on each domain controller, find this server’s Active Directory replication partners that run DNS.
  • Open DNSManager and connect in turn to each of these replication partners.
  • Find the host (A) resource record registration for this server on each of the other replication partner domain controllers.
  • Delete those host (A) records that do not have IP addresses corresponding to any of this server’s IP addresses.
  • If a domain controller has no host (A) records for this server, add at least one that corresponds to an IP address on this server. (If there are multiple IP addresses for this server, add at least one that is on the same network as the domain controller you are updating.)
  1. Name resolution may also fail with the RPC Server is unavailable error if NetBIOS over TCP/IP is disabled on the WINS tab in the advanced section of the TCP/IP properties. The NetBIOS over TCP/IP setting should be either enabled or default (use DHCP).
  1. Verify that a single label domain name is not being configured. DNS names that do not contain a suffix such as .com, .corp, .net, .org or .local are considered to be single-label DNS names. Microsoft doesn’t recommend using single label domain names because they cannot be registered with an Internet registrar and domain members do not perform dynamic updates to single-label DNS zones. Knowledge base article 826743 – “Clients cannot dynamically register DNS records in a single-label domain” provides instructions on how to configure your domain to allow dynamic registration of DNS records in a single label domain.

The RPC service or related services may not be started

Verify the status and startup type for the RPC and RPC locator services on the server that gets the error:

  1. By default, Windows server 2003 domain controllers and member servers all should have the RPC service started and set to Automatic startup and the RPC Locator service stopped and set to Manual Startup.
  2. Windows 2000 domain controllers should have the RPC and RPC Locator services both set to started and automatic startup, while Windows 2000 member servers should have the RPC service started and set to automatic startup while the RPC locator service should be started and set to manual startup.
  3. If you make any changes to the RPC service or to the RPC Locator service settings, restart the computer, and then test for the problem again.
  4. Additional Services that may result in “The RPC Server is Unavailable” errors are the TCP/IP NetBIOS helper service, Distributed File System service and Remote Registry service. These services should both be set to automatic and started. The Kerberos Key Distribution Center (KDC) should be Started and Automatic on Windows 2000 and Windows 2003 DCs. It should not be started and set to Disabled in all other cases.

Network Connectivity

Verify ports needed by RPC are open

Verify that ports greater than 1024 are not blocked. Clients connect to RPC Endpoint Mapper on port 135. RPC Endpoint Mapper then tells the client which randomly assigned port between 1024-65535 a requested service is listening on.

Ports may be blocked by a hardware firewall or a software firewall. Software firewalls include Internet Connection Firewall on computers running Windows Server 2003 or Windows XP, and Windows Firewall on computers running Windows Vista, Windows 7, Windows Server 2008 and Windows Server 2008 R2. A computer might also have third-party firewall software installed, or antivirus software with built-in firewall functionality. By default, port 135 TCP/UDP and ports 1024-65535 TCP must be open for RPC to work. You can restrict the ports greater than 1024 that RPC uses. However, RPC Endpoint Mapper is always on port 135.

File and Printer Sharing is not enabled

File and Printer sharing for Microsoft Networks will produce the error “RPC Server is unavailable” when you try to view or manage services on a remote computer using the Services snap-in. See the following example:

Unable to open service control manager database on \<computer>.
Error 1722: The RPC server is unavailable.
This error message may occur if the File and Printer Sharing for Microsoft Networks component is not enabled on the remote computer.

Troubleshooting RPC

The process of an RPC client connecting to an RPC server can be broken down into four phases. This troubleshooting guide will discuss the events that occur at each phase, how to test these events, and how to identify if the phase completed successfully.

Phase 1: Name Resolution: Name resolution is the act of resolving a name to an IP address. This normally takes two forms: NetBIOS Name Resolution or the more common DNS Name Resolution.

Phase 2: TCP session establishment: TCP session establishment is the act of establishing a TCP connection between the RPC client and the RPC server. TCP sessions will be initiated by the RPC client via a TCP 3-way handshake with the RPC server.

Phase 3: RPC Discovery: When a client wants to connect to the RPC server supplied by the application it will contact the computer that hosts the RPC Server and discover how to connect to the RPC Server.

Phase 4: RPC Communication: RPC Communication is the act of making RPC requests to the application endpoint and receiving RPC responses from this application.

Data needed to troubleshoot the issue:

  • Identify the client and server computers reporting the RPC error. Identify the DNS and WINS servers used by these computers. To do this:
  • On each machine, open a command prompt and run ipconfig /all.
  • Determine the IP address of both machines. If the server is part of a cluster get the cluster resource IP address as well. Identify the DNS servers and WINS servers that the RPC client is configured to use.

Note: You can also obtain this information by opening Control PanelNetwork and Sharing Center, clicking Local Area Connection and selecting Properties.

  • Identify the application(s) reporting RPC Server Unavailable
  • Simultaneous network traces (using Wireshark, Netmon, or a comparable network sniffer) from the machines hosting the RPC client and RPC Server while reproducing the task that results in a “RPC Server Unavailable” error.
  • The network captures on both hosts should be started first.
  • From a command prompt on the client run ipconfig /flushdns and nbtstat –R to clear the name resolution caches.
  • Reproduce the error.
  • Stop the traces and save them.

Name Resolution

Name Resolution consists of one or possibly more NetBIOS or DNS queries to locate the IP address for the RPC Server. Troubleshooting this phase requires verifying that a response is received to the name resolution request and that the response contains the correct IP address for the RPC server. Compare the IP address reported by DNS or NetBIOS in the network trace for the server with the IP addresses you noted earlier. If it does not match then check DNS and WINS and note if there is a difference.

DNS Name Resolution

To identify DNS Name Resolution in a network trace use the following filter in Network Monitor or Wireshark: “dns”. DNS resolution will be occurring at the client so open the network trace taken from the RPC client machine. You will be looking for one packet that is the query from the client to the DNS server and then the response packet from the DNS server. It will look similar to this:

If the trace shows the correct IP address for the RPC server was returned by the DNS server proceed to TCP Session Establishment.

If the trace does not show a correct IP address returned or you do not see any answer from the DNS server then reference the following resources to help with DNS name resolution troubleshooting.

For details on troubleshooting Active Directory related DNS issues go here.

For general DNS troubleshooting: http://support.microsoft.com/default.aspx?scid=kb;EN-US;330511

NetBIOS Name Resolution

NetBIOS queries come in two forms, WINS or NetBIOS Broadcasts. WINS will consist of a unicast query to a WINS server and a response from the WINS server.

NetBIOS broadcasts are queries broadcast to all hosts on the local subnet so name resolution is limited to only hosts on the subnet. The host with the name listed in the NetBIOS Broadcast will respond with its IP address.

To identify NetBIOS Name Resolution in a network trace, use the following filter in Network Monitor – “nbtns”. For Wireshark, use the following filter – ”nbns”. If the trace shows a successful resolution using WINS or NetBIOS queries proceed to TCP Session Establishment.

For details on troubleshooting this NetBIOS Name Resolution further:

http://technet.microsoft.com/en-us/library/cc940110.aspx

TCP Session Establishment

TCP Sessions always begin with a TCP 3-way handshake. The handshake should look similar to what is shown below. The RPC Client will send the first packet, known as the SYN packet. The computer hosting the RPC Server will send a SYN/ACK response, and then the RPC Client will send an ACK packet.

Scenarios that may cause the TCP session to fail

Firewall/Network

If a firewall or network problem is the culprit, it is likely a failure will occur during this phase. To diagnose this you will want to look at the network traces taken from the RPC Client and RPC Server. If a firewall or other network device is causing a problem it will usually manifest as a retransmit of the TCP SYN packet by the RPC Client about 3 seconds after the first TCP SYN is sent. This can be seen in a Netmon network trace using the display filter specification of “tcpsynretransmit==1”. In other cases, firewalls will allow the 3-way handshake to succeed but may block the RPC packets due to the contents of the packet at a higher level. In these cases it is possible to see the retransmit of the RPC packet within half a second of the original packet being sent. To identify this condition in a Netmon network trace use the display filter specification of “tcpretransmit==1”. To see either of these retransmit conditions in a trace taken using Wireshark use the display filter specification of “tcp.analysis.retransmission”.

The RPC Server is not actively listening.

It was noted earlier that an RPC Server will register itself and listen on a particular port and IP address of the host computer. If for some reason that fails the TCP layer will answer the SYN packet from the client with a Reset packet.

A device in the middle between the RPC Client and RPC Server will be resetting the connection attempt.

In the client side trace it will appear as if the server sent the TCP Reset while the trace from the server indicates the client is the source of the TCP Reset.

For both these scenarios, check for the presence of a Reset packet in the TCP three way handshake by using the display filter specification of “TCP.flags.reset==1”.

For troubleshooting this step see the following sections in this document:

If the 3-way handshake is successful, continue to the RPC Discovery phase.

RPC Discovery

The RPC Discovery phase will occur one of two ways. In both methods the client will know the identifier for the RPC Server it wants to contact and will supply that to the computer hosting the RPC Server and ask for information on how to contact the RPC Server. The identifier is different depending on which method is used and the RPC client will know ahead of time which method it wishes to use.

Discovery – RPC Over TCPIP

This method is a two-step process. First the RPC client will contact the End Point Mapper (EPM) on the machine hosting the RPC Server to find out what port and IP address that Server is listening on. Upon successful completion of this the RPC client will contact the RPC Server directly on the indicated IP address and Port. Below is a sample of what this would look like and a step by step explanation below it. This step depends on the successful TCP session establishment twice, first to the EPM and then to the RPC Server.

  1. The RPC Client will open a TCP session with TCP port 135 on the computer hosting RPC Server of interest. This can be picked out using the following filter syntax in Netmon or Wireshark: “tcp.port==135”
  2. The RCP Client will send an RPC Bind request using the UUID of the End Point Mapper and the RPC EPM should respond with a Bind ACK packet.
  3. The RPC Client will make a MAP request to the EPM to locate the IP address and port of the RPC Server of interest, identifying the RPC Server based on its UUID.
  4. The EPM will send back a MAP Response that indicates the IP and port the RPC Server is listening on.
  5. The RPC Client will then open a TCP session with the IP and port it received in the EPM MAP response.
  6. The client will send an RPC Bind Request to the RPC Server specifying the UUID of the RPC Server application and should get back a Bind ACK from the RPC Server.
  7. There will be an RPC Alter Context Request/Response in which authentication will take place. If an error is noted here then see the following section for help determining why the error is occurring – Authentication
  8. Perform some RPC operations…(Go to RPC Communication phase)

Discovery – RPC Over SMB

The second method an RPC Client may use to contact an RPC Server is RPC over SMB. This method depends upon first establishing an SMB session with the computer hosting the RPC Server and then using the Named Pipes protocol to communicate using RPC. So in effect there are several levels of encapsulation – RPC over Named Pipes over SMB over TCP. We will not address the SMB session setup in this document and the TCP session establishment has already been discussed.

With a successfully opened TCP and SMB session, next:

  1. The RPC Client will issue a SMB TreeConnectAndX for the tree name “IPC$”. This is a special hidden share for inter-process communication. It should get a positive response from the computer hosting the RPC Server.
  2. The RPC Client will then issue an SMB NTCreateAndX for the name of the PIPE of the RPC Server Application and should get back a positive response. Some examples are:

EVENTLOG = The Event log service

winreg = Remote Registry

svcctl = Service Control Manager

srvsvc = Server Service

  1. Next there is a Bind handshake. This is to “bind” the RPC client to the RPC server. There are a total of four packets involved:
  1. The RPC Client bind request containing the UUID of the desired RPC Server.
  2. A Write AndX response from the RPC Server
  3. A Read AndX request from the RPC Client.
  4. A Bind ACK response from the RPC Server.

At this time a RPC request to the RPC server component is expected.

RPC Communication

At this point RPC communication is occurring between the RPC Client and RPC Server. The troubleshooting steps involved at this stage are largely based on the application reporting the RPC failure.

For Active Directory processes or services please see Active Directory Symptoms.

For Microsoft Exchange related RPC errors please see: Analyzing Exchange RPC traffic over TCP/IP

How to identify the RPC traffic in a trace

RPC network traffic can take multiple forms. It is important to understand which form is in use in order to identify which TCP session is responsible for the RPC communication.

RPC over TCPIP

This is sometimes referred to as Traditional RPC or Sockets based RPC. An example of this is Outlook without “Outlook anywhere” or without http settings configured. A TCP session on TCP port 135 is established with the RPC server. To view this traffic in a trace use the filter: “tcp.port==135”. This session will be used in the RPC Discovery phase to locate the endpoint of the desired application.

RPC over HTTP

RPC connectivity for Internet connected hosts will typically use RPC over HTTP in order to traverse firewalls. Some examples of this can be seen with Terminal Services Gateway, Outlook Web Access, Outlook via “Outlook Anywhere”. This communication will be established on one or more connections to either TCP port 80 or 443(SSL). Since this typically traverses a public network, SSL or TCP port 443 is the more common method. Use the filter “tcp.port==80 or tcp.port==443” to locate either form inside network trace.

RPC over HTTP Port 80

For sessions over TCP port 80, the HTTP requests associated with RPC over HTTP will include a UserAgent header that contains the text “OutlookConnectorDS” and the version number of the connector.

RPC over HTTP Port 443

Sessions using TCP port 443 will initially establish a TLS session. After this TLS negotiation, the TCP Payload will be encrypted in TLS/SSL and the contents of the frames will not be readable in the trace. In this phase, look for failures due to improper certificates, inaccessible Certificate Revocation Lists, or untrusted certificate chains.

For more information on troubleshooting SSL/TLS see:

http://technet.microsoft.com/en-us/library/cc783349(WS.10).aspx

RPC over SMB aka “Named Pipes”

RPC can also take advantage of SMB sessions for the purpose of RPC communication. Some examples of this can be seen with Computer Management or the Remote Registry service. With the use of RPC over SMB:

  1. Establish TCP connection on TCP port 139 or 445.
  2. Negotiate dialect request/response
  3. SessionSetupANDX request/response. This sequence is used to establish the SMB Session. Authentication occurs during the SessionSetupANDX exchange.

If a failure in step 1 occurs, see additional troubleshooting steps see: File and Printer Sharing.

Kerberos Authentication

If Kerberos is used, and the client doesn’t currently have a Kerberos ticket for the RPC server, just after the Negotiate Dialect response is received, the client will obtain a Kerberos ticket for the Servername/cifs SPN of the RPC server. This exchange will occur over the Kerberos ports TCP or UDP port 88 between the client and a Domain Controller. SessionSetupANDX follows and will consist of a single SessionSetupANDX request which includes the Kerberos ticket, followed by a SessionSetupANDX Response indicating success or failure of the authentication.

For additional troubleshooting steps during authentication, see Authentication.

NTLM Authentication

If NTLM is used, SessionSetup will result in a SessionSetupANDX response with a status of STATUS_MORE_PROCESSING_REQUIRED. This response includes the NTLM challenge. The subsequent SessionSetupANDX Request will include the hashed credentials of the client. At this time, the RPC server must validate the credentials supplied by the user. To do this, the RPC server will contact a domain controller, and validate the credentials with the netlogon service, via RPC, on the domain controller. If this is successful, the RPC server will then respond to the client with a SessionSetupANDX Response indicating STATUS_SUCCESS.

For additional troubleshooting steps during authentication, see Authentication.

Troubleshooting Authentication

Verify that authentication is working correctly by checking for Time skew, UDP Fragmentation or an Invalid Kerberos Realm.

  • Time skew can be verified by running net time /querysntp and net time /setsntp:<PDCe server name>. The /querysntp switch allows you to determine if a specific DC is manually configured as the authoritative time server. The /setsntp:<PDCe server name> switch can be used to synchronize the computer receiving the error with the PDC emulator. The PDC emulator is the authoritative time server by default.
  • UDP fragmentation can cause replication errors that appear to have a source of RPC server is unavailable. Symptoms of UDP fragmentation being at the root of this problem include clients being unable to log on to the domain, administrators being unable join computers to the domain and Event ID 40960 & 40961 errors with a source of LSASRV and Kerberos errors with an Event ID of 10 in the system log.Knowledge base article 244474 – “How to force Kerberos to use TCP instead of UDP in Windows Server 2003, in Microsoft Windows and XP, and in Microsoft Windows 2000” provides the steps to resolve this problem.
  • An incorrect Kerberos realm can also be at the root of RPC server is unavailable problems. The symptoms that will be experience when the Kerberos realm is incorrect include the following errors when opening AD management tools:Naming Convention could not be located because: No authority could be contacted for authentication. Contact your system administrator to verify that your domain is properly configured and is currently online.-or-

    Naming information cannot be located because: No authority could be contacted for authentication. Contact your system administrator to verify that your domain is properly configured and is currently online.

    To verify that the correct Kerberos realm is configured, follow the steps in 837513 – “Domain controller is not functioning correctly”.

Active Directory Symptoms:

1. If you are experiencing replication problems and getting RPC server is unavailable errors as is reported in repadmin /showreps below, use Portqry or Network Monitor to determine if RPC traffic is being blocked is the first step when attempting to troubleshoot RPC Server is unavailable errors.

[Replications Check,DC2] A recent replication attempt failed:
From DC1 to DC2
Naming Context: CN=Schema,CN=Configuration,DC=xl
The replication generated an error (1722):
The RPC server is unavailable.
The failure occurred at 2003-10-30 11:59.47.
The last success occurred at 2003-10-28 20:50.22.
26 failures have occurred since the last success.
[DC1] DsBind() failed with error 1722,
The RPC server is unavailable..
The source remains down. Please check the machine.
BermudaDC1 via RPC objectGuid: 28c78c72-3c95-499a-bcda137a250f069f
Last attempt @ 2003-10-30 11:58.15 failed, result 1722:
The RPC server is unavailable.

Troubleshooting:
If IP Security Policies in Active Directory had the Assigned Value to Server (Request Security) set to Yes then these errors will result. Knowledge base article 313190 – “How to use IPSec IP filter lists in Windows 2000” provide details about where to check these settings and more information about their impact.

2. If you are blocking all ICMP traffic between separate AD sites, you will receive the errors below in the output of DCDIAG when trying to replicate inter-site:

Testing server: contosoDC1
Starting test: Replications
* Replications Check
[Replications Check,DC1] A recent replication attempt failed:
From DC2 to DC1
Naming Context: CN=Schema,CN=Configuration,DC=litware,DC=com
The replication generated an error (1722):
The RPC server is unavailable.
The failure occurred at 2003-08-24 23:00.51.
The last success occurred at (never).
553 failures have occurred since the last success.
[DC2] DsBind() failed with error 1722,
The RPC server is unavailable..
The source remains down. Please check the machine.
REPLICATION LATENCY WARNING
DC1: A full synchronization is in progress
from DC2 to DC1
Replication of new changes along this path will be delayed.
[DC2] LDAP connection failed with error 58,
The specified server cannot perform the requested operation.

Troubleshooting: To resolve this issue, remove the ICMP traffic restriction between domain controllers. When establishing an RPC session prior to AD replication, ICMP traffic is used. If the ICMP fails, so does the RPC session establishment, and hence AD replication also fails. ISA 2004 can prevent ICMP traffic with the exception of computers specified in the Remote Management Computers computer set which can be configured in system policy.

3. The following error will appear when attempting to connect to the computer.

“computer <\servername.domain.local> cannot be managed. The network path was not found. RPC server is unavailable.

Or when viewing the properties of the remote computer you will receive the error:

“Win32: The RPC server is unavailable”.


Troubleshooting:
Computer management is one of the better tools for testing RPC connectivity. When RPC traffic is being blocked, connections to other computers using the computer management console will fail.

4. When attempting to promote an additional domain controller in an Active Directory domain while the RPC service is blocked or not running, the following error will appear:

“The domain “domain.local” is not an Active Directory domain, or an Active Directory domain controller for the domain could not be contacted.


Troubleshooting:

5. Connections to computers via Remote Desktop may fail if RPC connectivity cannot be established. When attempting to logon on to the domain via Remote Desktop the following error will be produced in the form of a popup error message if RPC connectivity is the root of the problem:

“The system cannot log you on due to the following error: The RPC server is unavailable.”

You may also see the following errors on the Terminal server:

Error 1727: The remote procedure call failed and did not execute
Error 1722: The RPC server is unavailable.
Error 1723: The RPC server is too busy to complete this operation.
Error 1721: Not enough resources are available to complete this operation.

-or-

Event ID 5719:
Source: NetLogon
Description: No Windows NT Domain Controller is available for domain domain_name.
The following error occurred: There are currently no logon servers available to
service the logon request.

Event ID: 1219
Source: Winlogon
Details: Logon rejected for CONTOSO<computername>. Unable to obtain Terminal Server
User
Configuration. Error: The RPC server is unavailable.

Troubleshooting: These errors can be a result of the TCP/IP NetBIOS Helper service being disabled on the Terminal server or NetBIOS over TCP/IP being disabled on one of the NIC’s used to access the Terminal server. You should also verify that the Client for Microsoft networks is bound to the adapter used to access the Terminal server. You can tell if this is happening by looking at a Netdiag /v from the box for the following output:

Testing redirector and browser… Failed

NetBT transports test. . . . . . . : Failed
List of NetBt transports currently configured:
[FATAL] No NetBt transports are configured.

Redir and Browser test . . . . . . : Failed
List of transports currently bound to the Redir
NetBIOSSmb
[FATAL] The redir isn’t bound to any NetBt transports.

List of transports currently bound to the browser
[FATAL] The browser isn’t bound to any NetBt transports.

Troubleshooting Tools and Methods

Methods to generate RPC Traffic

Computer Management MMC to a remote host

Outlook to an Exchange server

RPCPing – http://support.microsoft.com/kb/831051

Tools for Testing RPC

RPCPing – http://support.microsoft.com/kb/831051

PortQry – http://support.microsoft.com/default.aspx?scid=kb;EN-US;832919

Pipelist – http://technet.microsoft.com/en-us/sysinternals/dd581625.aspx

RPCDump – http://support.microsoft.com/default.aspx?scid=kb;EN-US;325930

NSLookup – http://support.microsoft.com/default.aspx?scid=kb;EN-US;200525

NBLookup – http://support.microsoft.com/default.aspx?scid=kb;EN-US;830578

Tools for monitoring RPC

Network Monitor – Download FAQ

Wireshark – Download

Using PortQry

You can use the Portqry tool to verify that the required ports are open. You should run the Portqry tool on a computer that is not receiving any RPC errors against a computer that is receiving RPC errors by using the -n switch. To this, follow these steps:

a. Click “Start”, click “Run”, type “cmd” in the “Open” box, and then click OK”.

b. Type “portqry -n <problem_server> -e 135” (without the quotation marks).

The output will appear similar to the following examples:

Querying target system called:

<problem_server>
Attempting to resolve name to IP address…
Name resolved to 169.254.1.1
querying…
<problem_server>
TCP port 135 (epmap service): LISTENING
Using ephemeral source port
Querying Endpoint Mapper Database…

Server’s response:

UUID: f5cc59b4-4264-101a-8c59-08002b2f8426 NtFrs Service
ncacn_ip_tcp:65.53.63.16[1094]
UUID: e3514235-4b06-11d1-ab04-00c04fc2dcd2 MS NT Directory DRS Interface
ncacn_ip_tcp:65.53.63.16[1025]
UUID: e3514235-4b06-11d1-ab04-00c04fc2dcd2 MS NT Directory DRS Interface
ncacn_http:65.53.63.16[1029]
UUID: e3514235-4b06-11d1-ab04-00c04fc2dcd2 MS NT Directory DRS Interface
ncacn_http:65.53.63.16[6004]

If port 135 is blocked, the following will appear:

TCP port 135 (epmap service): NOT LISTENING However, for these RPC Endpoint Mapper errors it is likely that ports greater than 1024 are blocked, and not port 135.From the output, you know the DC is using port 1094 for FRS and 1025, 1029, and 6004 for Active Directory replication. You can use the Portqry tool again to check those ports. For example, you can test all the ports at the same time by using the Portqry tool with the -o switch. For example, type

“portqry -n <problem_server> -o 1094,1025,1029,6004″(Without the quotation marks)

If the ports all respond as “LISTENING,” it’s likely that blocked ports are not causing this problem. If any ports respond as “NOT LISTENING,” the ports are probably blocked.

Resources

RPC Blogs

Basics of RPC are covered here:

RPC to Go v.1: http://blogs.technet.com/b/networking/archive/2008/10/24/rpc-to-go-v-1.aspx

Architecture and a closer look at a connection to the RPC Endpoint mapper in a network capture.

RPC to Go v.2: http://blogs.technet.com/b/networking/archive/2008/12/04/rpc-to-go-v-2.aspx

This describes how RPC commands can be sent over Named Pipes in SMB via the IPC$ Tree.

RPC to Go v.3: http://blogs.technet.com/b/networking/archive/2009/04/28/rpc-to-go-v-3-named-pipes.aspx

Troubleshooting “RPC server is unavailable” error, reported in failing AD replication scenario.

http://blogs.technet.com/b/abizerh/archive/2009/06/11/troubleshooting-rpc-server-is-unavailable-error-reported-in-failing-ad-replication-scenario.aspx

External TechNet Magazine article

This one is good. It lays out RPC basics really quickly and then moves on RPC errors. The information on MaxUserPort would need to be updated with the information about the dynamic port ranges that are used in Vista/W2008 are the high range of ports compared to the 1025-5000 for W2003.

How IT Works, Troubleshooting RPC Errors by Zubair Alexander:

http://technet.microsoft.com/en-us/magazine/2007.07.howitworks.aspxKB

KB Article

Troubleshooting RPC Endpoint Mapper errors using the Windows Server 2003 Support Tools from the product CD

http://support.microsoft.com/default.aspx?scid=kb%3bEN-US%3b839880