Category: Exchange


Azure AD Sync issues:

 

https://gallery.technet.microsoft.com/scriptcenter/Common-Azure-AD-Sync-Issues-0f0025e1#content

 

https://techcommunity.microsoft.com/t5/Azure-Active-Directory-Identity/Sync-Error-Reports-in-AzureAD-Connect-Health-are-now-in-Public/ba-p/245151

Advertisements

Attack surface analyzer:

https://www.microsoft.com/security/blog/2019/05/15/announcing-new-attack-surface-analyzer-2-0

 

DSC-EA:

https://github.com/Microsoft/DSCEA

documentation: https://microsoft.github.io/DSCEA/

 

Microsoft security compliance toolkit:

Il remplace Security Compliance Manager. Cet outil permet de planifier, créer, et monitorer des baselines de sécurité pour vos postes clients. Le remplacement a été choisi par Microsoft du fait de la complexité de SCM et de la difficulté à maintenir l’outil pour chaque version de Windows. Aujourd’hui, SCT ne supporte pas Desired Configuration Management de System Center Configuration Manager ou SCAP.

https://www.microsoft.com/en-us/download/details.aspx?id=55319

how to use it:

https://arnaudloos.com/2018/intro-to-policy-analyzer/

https://github.com/MicrosoftDocs/windows-itpro-docs/blob/master/windows/security/threat-protection/security-compliance-toolkit-10.md

 

Other references:

2012 R2 hardening (CIS):

https://www.cisecurity.org/wp-content/uploads/2017/04/CIS_Microsoft_Windows_Server_2012_R2_Benchmark_v2.2.0.pdf

Windows 10 hardening:

https://www.asd.gov.au/publications/protect/Hardening_Win10.pdf

 

 

 

Troubleshooting Logs and Tools

https://blogs.technet.microsoft.com/exchange/2016/05/31/checklist-for-troubleshooting-outlook-connectivity-in-exchange-2013-and-2016-on-premises/

SaRA tool to assess OUTLOOK client: https://diagnostics.outlook.com/#/

Also on CTRL + right click on OUTLOOK icon on the system tray! to get the connection status

Test connectivity from outside using: https://testconnectivity.microsoft.com/

Also check potential source of problems:

  • Check ADFS policies
  • Check set-CASmailbox – (post authentication) ; if POP or imap protocols are blocked for example
  • AzureAD Conditional access policies – (post authentication)
  • Authentication policies – in Exchange online (“new-authenticationpolicy”)- (pre authentication)
  • Client access rules – exchange online
  • Org level – IP blacklist – legacy authentication can be blocked
  • Org level – blacklist – EWS connections can be blocked
  • Org level – disable SMTP auth legacy – recommended
  • To protect from DDOS attack, enable ADFS extranet lockout protection and check audit log
  • IdFIX tool: https://www.microsoft.com/en-us/download/details.aspx?id=36832

Side-effect on Modern authentication:

If ADFS WAP and Internal servers are stopped ! What are the side-effects to access Outlook ??

https://www.peters.com/modern-authentication-part-2/

  1. On clients with Modern authentication or ADAL! => thanks to the access tokens but we can limit the issues (valid 90 days!)
    1. If ADFS internal is restarted => Only => problem solved (no need WAP)
  2. But for OL 2010 or OL 2013 without ADAL, we are prompted to enter USER/PASSWORD (but without success)
    1. We need also the WAP working! And not only ADFS internal… to solve the problem on old clients not supporting ADAL

 

And also check logs:

HTTP Proxy RPCHTTP Logs

In Exchange 2013, there are several logs in the logging folder. For Outlook clients one of the first logs to examine are the HTTP Proxy logs on CAS. The connection walk-through section shows the process that is used to connect to Exchange 2013. This complete process is logged in the HTTP Proxy log. Also, if it is possible, add Hosts file to the client for one specific CAS to reduce the number of logs.

The logs on CAS are located here by default: C:\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\RpcHttp

HTTP Proxy AutoDiscover Logs

Exchange 2013 has HTTP Proxy logs for AutoDiscover that are similar to the logs shown earlier that can be used to determine whether AutoDiscover is failing.

The logs on CAS are located here by default: C:\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\AutoDiscover

HTTP Error Logs

HTTP Error logs are failures that occur with HTTP.SYS before hitting IIS. However, not all errors for connections to web sites and app pools are seen in the httperr log. For example, if ASP.NET threw the error it may not be logged in the HTTP Error log. By default, HTTP error logs are located in C:\Windows\System32\LogFiles\HTTPERR. Information on the httperr log and codes can be found here.

IIS Logs

IIS logs can be used to review the connection for RPC/HTTP, MAPI/HTTP, EWS, OAB, and AutoDiscover. The full data for the MAPI/HTTP and RPC/HTTP is not always put in the IIS logs. Therefore, there is a possibility that the 200 connection successful may not be seen. IIS codes.

In Exchange 2013 IIS logs on the CAS should contain all user connections on port 443. IIS logs on the Mailbox server should only contain connections from the CAS server on port 444.

Most HTTP connections are first sent anonymously which results in a 401 challenge response. This response includes the authentication types available in the response header. The client should then try to connect again by using one of these authentication methods. Therefore, a 401 status found inside an IIS log does not necessarily indicate an error.

Note that an anonymous request is expected to show a 401 response. You can identify anonymous requests because the domain\username is not listed in the request.

RPC Client Access (RCA) Logs

The RCA logs can be used to find when a user has made a connection to their mailbox, or a connection to an alternate mailbox, errors that occur with the connection, and more information. RCA logs are located in the logging directory which is located at %ExchangeInstallPath%\Logging\RpcClientAccess. By default, these logs have a maximum size of 10MB and roll over when size limit is reached or at the end of the day (based on GMT), and the server keeps 1GB in the log directory.

Outlook ETL Logging (requires a support case with Microsoft to analyze the log) 

ETL logs are located in %temp%/Outlook Logging and are named Outlook-#####.ETL. The numbers are randomly generated by the system.

To enable Outlook logging

In the Outlook interface:

  • Open Outlook.
  • Click File, Options, Advanced.
  • Enable “Enable troubleshooting logging (requires restarting Outlook)”
  • Restart Outlook.

How to enable Outlook logging in the registry:

  • Browse to HKEY_CURRENT_USER\Software\Microsoft\Office\xx.0\Outlook\Options\Mail
  • DWORD: EnableLogging
  • Value: 1
  • Note: xx.0 is a placeholder for your version of Office. 15.0 = Office 2013, 14.0 = Office 2010

ExPerfwiz (Perfmon for Exchange)

You can use Perfmon for issues that you suspect are caused by performance. http://experfwiz.codeplex.com/

Exchange 2013 has daily performance logs that captures the majority of what is needed. These logs are by default located in C:\Program Files\Microsoft\Exchange Server\V15\Logging\Diagnostics\DailyPerformanceLogs

Log Parser Studio

Log Parser Studio is a GUI for Log Parser 2.2. LPS greatly reduces complexity when parsing logs. Additionally, it can parse many kinds of logs including IIS Logs, HTTPErr Logs, Event Logs (both live and EVT/EVTX/CSV), all Exchange protocol logs from 2003-2013, any text based logs, CSV logs and ExTRA traces that were converted to CSV logs. LPS can parse many GB of logs concurrently (we have tested with total log sizes of >60GB).

Blog with tips/how to about LPS: http://blogs.technet.com/b/karywa/

Exmon tool (aka Microsoft Exchange Server User Monitor)

We use this tool to get detailed information about client traffic.

 

By default Azure AD connect will synchronize disabled accounts from AD to AAD. It is normal and is it recommended due to Exchange hybrid and EXO requirements.

 

It is possible to create a custom rule on AD Sync rules editor to not synchronize disabled AD accounts:

https://spanougakis.wordpress.com/2016/02/28/how-to-stop-disabled-user-accounts-from-syncing-with-azure-ad-connect/

 

As we prepare for the migration from on-premises Skype for Business to Skype for Business Online, there are a few important considerations to bear in mind before you take the leap. I will be covering these in a series of posts (hopefully), today I want to share with you a common scenario we will face while preparing for migration.

We are well aware of the pre-requisite for Office 365 that demands an Active Directory synchronised user must have a publically routable User Principal Name (UPN). So critical is this requirement that it is now engrained in every consultant’s mind and increasingly customers are becoming more aware of this without us even mentioning it. However, this can often produce its own unique challenges.

Many organisations set their users up with an ambiguous username, something that does not immediately identify a user by name e.g. a134g@domain.com rather than markv@domain.com. This is to avoid name conflicts and was often used as an additional domain security measure. When a user is synchronised to Office 365 their UPN is used to provision the identity and service addresses for Exchange and Skype for Business. Often the case is that users UPNs do not match their publically available contact information such as their e-mail address. E-mail addresses are usually more personable to each users and contain their true identity e.g markv@domain.com. In order to integrate Skype for Business Online with Exchange properly it is important that the user’s SIP address matches their primary e-mail address i.e. markv@domain.com and not a134g@domain.com.

However, when you perform an AD Sync with Office 365 the user’s Skype for Business Online identity is provisioned using the UPN like so:

On-premises Identity (UPN = a134g@domain.org)

Synchronised Identity in Office 365 Portal also a134g@domain.org

PowerShell output showing primary SIP Address in Skype for Business Online


In order to change this, the solution is to look and edit the on-premises identity, as this is the source of authority for this person’s cloud identity. Specifically, we need to modify an attribute on the user’s Active Directory account called msRTCSIP-PrimaryUserAddress. In order to find this attribute, your on-premises Active Directory domain is required to be prepared for Lync / Skype for Business On-premises. Therefore, you may need to download the on premises software and run AD schema preparation to have this property available. I say may read on.

First change find and change this attribute

Click on edit and enter the desired address in this format: sip:markv@domain.org

Perform directory synchronisation using AADSync and then check the SIP Address of the online identity. You should see that is has changed

Please note that this only works for synchronised identities. Cloud identities must be provisioned with the primary SIP address as the username.

If you have not prepared your Active Directory domain from on-premises Lync / Skype for Business and do not have the msRTCSIP-PriamryUserAddress attribute, there is an alternative method you can use. Instead we can use the ProxyAddresses attribute that is natively part of Active Directory. This attribute is the same on you use for provisioning e-mail addresses to get around the same issue as we have. Open the ProxyAddressess attribute and add a new Proxy Address into the list using the following format: sip:markv@domain.org

Perform a directory synchronisation and test the SIP address has been updated correctly

https://dirkjanm.io/abusing-exchange-one-api-call-away-from-domain-admin/

and

https://mspoweruser.com/massive-vulnerability-means-lost-email-password-can-lead-to-hacked-microsoft-exchange-server-worse/

 

Explanations:

https://adsecurity.org/?p=4119

 

Microsoft recommendations:

https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/ADV190007

and

https://blogs.technet.microsoft.com/exchange/2019/02/12/released-february-2019-quarterly-exchange-updates/

 

Reference: https://docs.microsoft.com/en-us/office365/securitycompliance/enable-mailbox-auditing

Scripts:

https://github.com/O365AES/Scripts

To get status:

Get-Mailbox “office365user”| FL Audit*

AuditEnabled : True
AuditLogAgeLimit : 90.00:00:00
AuditAdmin : {Update, Copy, Move, MoveToDeletedItems…}
AuditDelegate : {Update, Move, MoveToDeletedItems, SoftDelete…}
AuditOwner : {}

 

To setup mailbox audit logging for all user mailboxes in your organization:

Get-Mailbox -ResultSize Unlimited -Filter {RecipientTypeDetails -eq “UserMailbox”} | Set-Mailbox -AuditEnabled $true

Tips and Tricks:
Tip: In Exchange Server 2016 the architecture was simplified when compared with previous versions, and nowadays we have only two roles: Mailbox and Edge. Where the Mailbox is the role that is located in the internal network with access to the Active Directory.
Tip Exchange 2013 sp1: the Edge role reappears
Tip: ReFS not supported as File System
Tip: No storage on SMB is supported
Tip: OWA support the certificates and ADFS (strong authent scenario)
Tip: dedicate CAS servers behind hw load balancer – with public URL. the Certificate is managed by internal PKI.
Tip: prefer using Win2012 R2 and Exchange 2013 SP1 (better together!)
Tip: prefer using the command lines to install exchange 2013 role
Tip: when using cmdlets: always for details do | fl prop1,*prop2  or  | ft -autosize

Topology best practice (after SP1 of Exch 2013):

internet —–  FW —– edge server (dmz / in a wkg) —– FW —– hwlb – CAS servers —– MBX servers — FW — AD / PKI servers

 

Exchange installation prerequisites:

For Exchange 2016: http://www.itprocentral.com/exchange-server-2016-prerequisites/

For Exchange 2013: http://www.itprocentral.com/prepare-windows-server-2012-r2-support-exchange-server-2013-sp1/

Exchange 2016 step by steps: https://judeperera.wordpress.com/2015/07/24/step-by-step-guide-for-installing-exchange-server-2016-preview/

Cumulative updates:

https://docs.microsoft.com/en-us/exchange/new-features/build-numbers-and-release-dates

 

Exchange and certificates:

public or internal PKI server certificates only on CAS servers, follow the recommendations here: http://blogs.technet.com/b/exchange/archive/2014/03/19/certificate-planning-in-exchange-2013.aspx

also the client computers are joined to the AD domain and have also a computer certificate.

 

Exchange and Firewalls:

http://blogs.technet.com/b/exchange/archive/2013/02/18/exchange-firewalls-and-support-oh-my.aspx

 

Deployment assistant for Exchange:

http://technet.microsoft.com/en-us/exdeploy2013/Checklist?state=2284-W-AAAAAAAAQAAAAAEAAAAAAA%7e%7e 

 

Exchange sizing:

http://blogs.technet.com/b/exchange/archive/2013/05/14/released-exchange-2013-server-role-requirements-calculator.aspx

http://blogs.technet.com/b/exchange/archive/2013/05/06/ask-the-perf-guy-sizing-exchange-2013-deployments.aspx

HP sizer for Exchange 2013:http://h71019.www7.hp.com/ActiveAnswers/cache/70255-0-0-225-121.html

 

How to dedicate DC to Exchange? and It is recommended to exclude the DC PDC server.

https://support.microsoft.com/en-us/kb/298879

http://smtpport25.wordpress.com/2009/03/20/statically-pointing-exchange-2007-servers-to-dedicated-domain-controllers-and-global-catalogs-and-excluding-other-domain-controllers/

How to separate roles for AD admins and roles for Exchange admins? ==> RBAC split permissions and AD split permissions
Troubleshooting:

Test connectivity: https://testconnectivity.microsoft.com/

ExLogAnalyzer to the rescue:http://blogs.technet.com/b/mspfe/archive/2013/12/04/dude-what-s-flowing-in-my-exchange-environment.aspx

Database maintenance: http://blogs.technet.com/b/exchange/archive/2011/12/14/database-maintenance-in-exchange-2010.aspx

With Outlook 2013 installed; CTRL+ right-click Outlook icon on the taskbar; then Check Outlook Connectivity and Test Messaging configuration

Validation and monitoring of storage:

When implementing a storage solution for Exchange, an easily overlooked step is the evaluation of storage after it has been put in place to determine a baseline for that storage. Microsoft makes tools to enable this testing. Jetstress and LoadGen available for Exchange 2010/2013 can be used to test storage or Exchange overall and establish a baseline for future comparison.

Jetstress 2013: http://www.microsoft.com/en-us/download/details.aspx?id=36849

LoadGen 2013: http://www.microsoft.com/en-us/download/details.aspx?id=40726

 How to install the Exchange management tools and PowerShell cmdlets:For Exchange 2013/2016:

https://technet.microsoft.com/en-us/library/bb232090(v=exchg.160).aspx

in short: Setup.exe /Role:ManagementTools /IAcceptExchangeServerLicenseTerms

Best practices:

Connecting to Exchange on-premises using PowerShell remoting is a simple task, and means that you do not need to go the trouble of installing the Exchange management tools on your computer just to perform day to day administrative tasks.

There are three steps to establishing a remote PowerShell session to your Exchange server:

  • Capture admin credentials
  • Create a new PSSession
  • Import the PSSession

https://practical365.com/exchange-server/powershell-function-to-connect-to-exchange-on-premises/

Usage: Connect-Exchange -URL exchange2.mydomain.local

Scripts repository for Exchange:

https://github.com/cunninghamp/Powershell-Exchange

 

 

 

Exchange 2013, how to put server in maintenance mode:

 

WEB RESOURCES

http://ehloexchange.com/exchange-maintenance-mode/

https://blogs.technet.microsoft.com/nawar/2014/03/30/exchange-2013-maintenance-mode/

http://www.enpointe.com/blog/server-maintenance-with-exchange-2013-step-by-step-guide

 

MAILBOX ROLE

Set-ServerComponentState $env:COMPUTERNAME -Component HubTransport -State Draining -Requester Maintenance

Redirect-Message -Server $env:COMPUTERNAME -Target <otherserver> -confirm:$false

Suspend-ClusterNode $env:COMPUTERNAME

Set-MailboxServer $env:COMPUTERNAME -DatabaseCopyActivationDisabledAndMoveNow $True

Set-MailboxServer $env:COMPUTERNAME -DatabaseCopyAutoActivationPolicy Blocked

Set-ServerComponentState $env:COMPUTERNAME -Component ServerWideOffline -State Inactive -Requester Maintenance

Get-ServerComponentState $env:COMPUTERNAME | Format-Table Component,State -Autosize

Get-MailboxServer $env:COMPUTERNAME | Format-Table DatabaseCopy* -Autosize

Get-ClusterNode $env:COMPUTERNAME | Format-List

 

Get-MailboxDatabaseCopyStatus |where {$_.status -like “mounted”}
Note here: You need to make sure that there are no mounted databases, if that is not the case, you need to wait several minutes and the run it again.

 

CAS ROLE

Set-ServerComponentState $env:COMPUTERNAME -Component HubTransport -State Draining -Requester Maintenance

Redirect-Message -Server $env:COMPUTERNAME -Target <otherserver> -confirm:$false

Set-ServerComponentState $env:COMPUTERNAME -Component ServerWideOffline -State Inactive -Requester Maintenance

Get-ServerComponentState $env:COMPUTERNAME | Format-Table Component,State -Autosize

 

EDGE ROLE

Set-ServerComponentState $env:COMPUTERNAME -Component ServerWideOffline -State Inactive -Requester Maintenance

Get-ServerComponentState $env:COMPUTERNAME | Format-Table Component,State -Autosize

 

 

EXCHANGE MAINTENANCE MODE DEACTIVATION

 

Access the server through RDP and open Exchange Powershell in Admin mode, with a account member of the right group (Server management / Organization Management)

 

MAILBOX ROLE

Set-ServerComponentState $env:COMPUTERNAME -Component ServerWideOffline -State Active -Requester Maintenance

Resume-ClusterNode $env:COMPUTERNAME

Set-MailboxServer $env:COMPUTERNAME -DatabaseCopyActivationDisabledAndMoveNow $false

Set-MailboxServer $env:COMPUTERNAME -DatabaseCopyAutoActivationPolicy Unrestricted

Set-ServerComponentState -Identity $env:COMPUTERNAME -Component HubTransport -State Active -Requester Maintenance

Get-ServerComponentState $env:COMPUTERNAME | Format-Table Component,State -Autosize

Get-MailboxServer $env:COMPUTERNAME | Format-Table DatabaseCopy* -Autosize

Get-ClusterNode $env:COMPUTERNAME | Format-List

 

CAS ROLE

Set-ServerComponentState -Identity $env:COMPUTERNAME -Component HubTransport -State Active -Requester Maintenance

Set-ServerComponentState $env:COMPUTERNAME -Component ServerWideOffline -State Active -Requester Maintenance

Get-ServerComponentState $env:COMPUTERNAME | Format-Table Component,State -Autosize

 

EDGE ROLE

Set-ServerComponentState $env:COMPUTERNAME -Component ServerWideOffline -State Active -Requester Maintenance

Get-ServerComponentState $env:COMPUTERNAME | Format-Table Component,State -Autosize

 

ProxyAddress attribute is used by different applications and it can store different type of user addresses (sip, smtp, x500).

If you sync accounts with non-verified domain to O365, those addresses can be replaced with the default onmicrosof.com domain.

Some interesting reading regarding this topic:

ADConnect not Syncing ProxyAccount for email Alias from on Premise AD to Azure AD

https://social.msdn.microsoft.com/Forums/azure/en-US/3226e41c-1db1-4299-9f24-0179e05fac09/adconnect-not-syncing-proxyaccount-for-email-alias-from-on-premise-ad-to-azure-ad-i-am-using?forum=WindowsAzureAD

List of attributes that are synced by the Azure Active Directory Sync Tool

https://social.technet.microsoft.com/wiki/contents/articles/19901.dirsync-list-of-attributes-that-are-synced-by-the-azure-active-directory-sync-tool.aspx

A mail user who has proxy addresses that use non-verified domains isn’t synced in an Exchange hybrid deployment

https://support.microsoft.com/en-us/help/3124148/a-mail-user-who-has-proxy-addresses-that-use-non-verified-domains-isn

Wrong domain address when synchronizing from on premise AD

https://social.msdn.microsoft.com/Forums/azure/en-US/7ddc1885-850d-487f-bf40-a91f1f5d15c8/wrong-domain-address-when-synchronizing-from-on-premise-ad?forum=WindowsAzureAD

Azure AD Connect sync: Attributes synchronized to Azure Active Directory

https://docs.microsoft.com/en-us/azure/active-directory/connect/active-directory-aadconnectsync-attributes-synchronized