Category: Exchange


Microsoft’s file systems organize storage devices based on cluster size. Also known as the allocation unit size, cluster size represents the smallest amount of disk space that can be allocated to hold a file. Because ReFS and NTFS don’t reference files at a byte granularity, the cluster size is the smallest unit of size that each file system can reference when accessing storage. Both ReFS and NTFS support multiple cluster sizes, as different sized clusters can offer different performance benefits, depending on the deployment.

Full article from MS: https://blogs.technet.microsoft.com/filecab/2017/01/13/cluster-size-recommendations-for-refs-and-ntfs/

Summary:

ReFS cluster sizes

ReFS offers both 4K and 64K clusters. 4K is the default cluster size for ReFS, and we recommend using 4K cluster sizes for most ReFS deployments because it helps reduce costly IO amplification:

  • In general, if the cluster size exceeds the size of the IO, certain workflows can trigger unintended IOs to occur. Consider the following scenarios where a ReFS volume is formatted with 64K clusters:
    • Consider a tiered volume. If a 4K write is made to a range currently in the capacity tier, ReFS must read the entire cluster from the capacity tier into the performance tier before making the write. Because the cluster size is the smallest granularity that the file system can use, ReFS must read the entire cluster, which includes an unmodified 60K region, to be able to complete the 4K write.
    • If a cluster is shared by multiple regions after a block cloning operation occurs, ReFS must copy the entire cluster to maintain isolation between the two regions. So if a 4K write is made to this shared cluster, ReFS must copy the unmodified 60K cluster before making the write.
    • Consider a deployment that enables integrity streams. A sub-cluster granularity write will cause the entire cluster to be re-allocated and re-written, and the new checksum must be computed. This represents additional IO that ReFS must perform before completing the new write, which introduces a larger latency factor to the IO operation.
  • By choosing 4K clusters instead of 64K clusters, one can reduce the number of IOs that occur that are smaller than the cluster size, preventing costly IO amplifications from occurring as frequently.

Additionally, 4K cluster sizes offer greater compatibility with Hyper-V IO granularity, so we strongly recommend using 4K cluster sizes with Hyper-V on ReFS.  64K clusters are applicable when working with large, sequential IO, but otherwise, 4K should be the default cluster size.

NTFS cluster sizes

NTFS offers cluster sizes from 512 to 64K, but in general, we recommend a 4K cluster size on NTFS, as 4K clusters help minimize wasted space when storing small files. We also strongly discourage the usage of cluster sizes smaller than 4K. There are two cases, however, where 64K clusters could be appropriate:

  • 4K clusters limit the maximum volume and file size to be 16TB
    • 64K cluster sizes can offer increased volume and file capacity, which is relevant if you’re are hosting a large deployment on your NTFS volume, such as hosting VHDs or a SQL deployment.
  • NTFS has a fragmentation limit, and larger cluster sizes can help reduce the likelihood of reaching this limit
    • Because NTFS is backward compatible, it must use internal structures that weren’t optimized for modern storage demands. Thus, the metadata in NTFS prevents any file from having more than ~1.5 million extents.
      • One can, however, use the “format /L” option to increase the fragmentation limit to ~6 million. Read more here.
    • 64K cluster deployments are less susceptible to this fragmentation limit, so 64K clusters are a better option if the NTFS fragmentation limit is an issue. (Data deduplication, sparse files, and SQL deployments can cause a high degree of fragmentation.)
      • Unfortunately, NTFS compression only works with 4K clusters, so using 64K clusters isn’t suitable when using NTFS compression. Consider increasing the fragmentation limit instead, as described in the previous bullets.

While a 4K cluster size is the default setting for NTFS, there are many scenarios where 64K cluster sizes make sense, such as: Hyper-V, SQL, deduplication, or when most of the files on a volume are large.

Download sysmon:

NEW: Sysmon 6.02 is available ! : https://technet.microsoft.com/en-us/sysinternals/sysmon  and how to use it:

Installation and usage:

List of web resources concerning Sysmon: https://github.com/MHaggis/sysmon-dfir

Mark russinovitch’s RSA conference: https://onedrive.live.com/view.aspx?resid=D026B4699190F1E6!2843&ithint=file%2cpptx&app=PowerPoint&authkey=!AMvCRTKB_V1J5ow

Sysmon config files explained:

https://github.com/SwiftOnSecurity/sysmon-config

https://github.com/ion-storm/sysmon-config/blob/master/sysmonconfig-export.xml

https://www.bsk-consulting.de/2015/02/04/sysmon-example-config-xml/

View story at Medium.com

Else other install guides:

Sysinternals Sysmon unleashed

http://www.darkoperator.com/blog/2014/8/8/sysinternals-sysmon

 

Detecting APT with Sysmon:

https://www.rsaconference.com/writable/presentations/file_upload/hta-w05-tracking_hackers_on_your_network_with_sysinternals_sysmon.pdf

 

https://www.root9b.com/sites/default/files/whitepapers/R9B_blog_005_whitepaper_01.pdf

Sysmon with Splunk:

http://blogs.splunk.com/2014/11/24/monitoring-network-traffic-with-sysmon-and-splunk/

https://securitylogs.org/tag/sysmon/

Sysmon log analyzer/parsing sysmon event log:

https://github.com/CrowdStrike/Forensics/blob/master/sysmon_parse.cmd

https://digital-forensics.sans.org/blog/2014/08/12/sysmon-in-malware-analysis-lab

https://github.com/JamesHabben/sysmon-queries

http://blog.crowdstrike.com/sysmon-2/

logparser: http://www.microsoft.com/en-us/download/confirmation.aspx?id=24659

logparser GUI: http://lizard-labs.com/log_parser_lizard.aspx

Tips and Tricks:
Tip: In Exchange Server 2016 the architecture was simplified when compared with previous versions, and nowadays we have only two roles: Mailbox and Edge. Where the Mailbox is the role that is located in the internal network with access to the Active Directory.
Tip Exchange 2013 sp1: the Edge role reappears
Tip: ReFS not supported as File System
Tip: No storage on SMB is supported
Tip: OWA support the certificates and ADFS (strong authent scenario)
Tip: dedicate CAS servers behind hw load balancer – with public URL. the Certificate is managed by internal PKI.
Tip: prefer using Win2012 R2 and Exchange 2013 SP1 (better together!)
Tip: prefer using the command lines to install exchange 2013 role
Tip: when using cmdlets: always for details do | fl prop1,*prop2  or  | ft -autosize

Topology best practice (after SP1 of Exch 2013):

internet —–  FW —– edge server (dmz / in a wkg) —– FW —– hwlb – CAS servers —– MBX servers — FW — AD / PKI servers

 

Exchange installation prerequisites:

For Exchange 2016: http://www.itprocentral.com/exchange-server-2016-prerequisites/

For Exchange 2013: http://www.itprocentral.com/prepare-windows-server-2012-r2-support-exchange-server-2013-sp1/

Exchange 2016 step by steps: https://judeperera.wordpress.com/2015/07/24/step-by-step-guide-for-installing-exchange-server-2016-preview/

Cumulative updates:

https://www.microsoft.com/en-us/download/details.aspx?id=51499

 

Exchange and certificates:

public or internal PKI server certificates only on CAS servers, follow the recommendations here: http://blogs.technet.com/b/exchange/archive/2014/03/19/certificate-planning-in-exchange-2013.aspx

also the client computers are joined to the AD domain and have also a computer certificate.

 

Exchange and Firewalls:

http://blogs.technet.com/b/exchange/archive/2013/02/18/exchange-firewalls-and-support-oh-my.aspx

 

Deployment assistant for Exchange:

http://technet.microsoft.com/en-us/exdeploy2013/Checklist?state=2284-W-AAAAAAAAQAAAAAEAAAAAAA%7e%7e 

 

Exchange sizing:

http://blogs.technet.com/b/exchange/archive/2013/05/14/released-exchange-2013-server-role-requirements-calculator.aspx

http://blogs.technet.com/b/exchange/archive/2013/05/06/ask-the-perf-guy-sizing-exchange-2013-deployments.aspx

HP sizer for Exchange 2013:http://h71019.www7.hp.com/ActiveAnswers/cache/70255-0-0-225-121.html

 

How to dedicate DC to Exchange? and It is recommended to exclude the DC PDC server.

https://support.microsoft.com/en-us/kb/298879

http://www.proexchange.be/blogs/exchange2010/archive/2012/03/15/exchange-2010-and-why-you-shouldn-t-configure-static-domain-controllers.aspx

http://smtpport25.wordpress.com/2009/03/20/statically-pointing-exchange-2007-servers-to-dedicated-domain-controllers-and-global-catalogs-and-excluding-other-domain-controllers/

How to separate roles for AD admins and roles for Exchange admins? ==> RBAC split permissions and AD split permissions
Troubleshooting:

Test connectivity: https://testconnectivity.microsoft.com/

ExLogAnalyzer to the rescue:http://blogs.technet.com/b/mspfe/archive/2013/12/04/dude-what-s-flowing-in-my-exchange-environment.aspx

Database maintenance: http://blogs.technet.com/b/exchange/archive/2011/12/14/database-maintenance-in-exchange-2010.aspx

With Outlook 2013 installed; CTRL+ right-click Outlook icon on the taskbar; then Check Outlook Connectivity and Test Messaging configuration

Validation and monitoring of storage:

When implementing a storage solution for Exchange, an easily overlooked step is the evaluation of storage after it has been put in place to determine a baseline for that storage. Microsoft makes tools to enable this testing. Jetstress and LoadGen available for Exchange 2010/2013 can be used to test storage or Exchange overall and establish a baseline for future comparison.

Jetstress 2013: http://www.microsoft.com/en-us/download/details.aspx?id=36849

LoadGen 2013: http://www.microsoft.com/en-us/download/details.aspx?id=40726

Tasks to force removal of Exchange 2013

The tasks to force removal of Exchange 2013 are:

The most common reasons are listed below:

  • The deinstallation didn’t finish properly and left attributes or entries in Active Directory
  • The Exchange server is permanent offline and Exchange should be removed
  • An Exchange installation didn’t finish properly and the attributes and entries should be removed

To remove the server open ADSIEdit and go to configuration

On ADSIEDIT:

CN=Configuration,DC=MYDOMAIN,DC=LOCAL
CN=Services
->; DELETE
CN=Microsoft Exchange
CN=Microsoft Exchange Autodiscover

CN=Default naming context,DC=MYDOMAIN,DC=LOCAL
->; DELETE
CN=Microsoft Exchange Security Groups
CN=Microsoft Exchange Security Objects

IIS: Start inetmgr
DELETE the Exchange Back End and Front End websites with the IIS-Manager:

autodiscover
ecp (-> Exchange Control Panel)
EWS (-> Exchange Web Services)
Microsoft server Activsync (-> Exchange Active Sync)
OAB (-> Offline Addressbook)
owa (-> Outlook Web App)
Rpc (-> Remote Procedure Calls)

etc…

Also don’t forget to remove the application pools (MSExchange*)

Certificates:

Remove the local computer certificates (MMC, certificates snap-in, computer store)

AD Users and Computers:
DELETE the following users in the “Users” container:

DiscoverySearch Mailbox{GUID}
Exchange Online-ApplicationAccount
FederatedEmail.GUID
Migration.GUID
*SystemMailbox{GUID}
*HealthMailboxGUID

Registry:
DELETE the key “ExchangeServer” under:
HKLM\Software\Microsoft

DELETE the keys “MSExchange*” under:
HKLM\CurrentControlSet\Services

Hard Disk directories:
On the server’s hard disk you’ve to DELETE the Exchange Server installation folder.
Usually it’s C:\Program Files\Microsoft\Exchange Server

and c:\ExchangeSetupLogs

remove also d:\ mailboxes or other Exchange logs / monitoring directories

Cleanup Recycle bin

Final reboot

 

The NSA released a PDF entitled “Spotting the Adversary with Windows Event Log Monitoring” earlier this year. The good news is it’s probably one of the most detailed documents I’ve seen in a long time. Everything from setting up Event Subscriptions, to a hardened use of Windows Remote Management, including the use of authentication and firewalls, this document tells you how to securely setup an environment where you can natively consolidate and monitor event log based entries. In addition, the NSA goes onto cover a number of areas that should be monitored – complete with event IDs:

http://www.redblue.team/2015/09/spotting-adversary-with-windows-event.html

http://www.redblue.team/2015/09/spotting-adversary-with-windows-event_21.html

Event forwarding guidance: https://github.com/iadgov/Event-Forwarding-Guidance

Malware archeology cheat sheets: http://www.malwarearchaeology.com/cheat-sheets/

Machine-specific issues – which can be indications of malicious activity

  • Application Crashes
  • System or Service Failures
  • Kernel and Device Signing
  • The Windows Firewall

Administrator Activity – specific actions performed that may be suspect

  • Clearing of Event Logs
  • Software and Service Installation
  • Remote Desktop Logon
  • Account Usage

The bad news is you’re still left to sort out a TON of event log detail and interpret whether the entries are a problem or not.

Additionally: Changes to Group Policy only show up in the events as a change to the policy, but lack detail on exactly what was changed within the Group Policy.

To truly have a grasp on whether you have an “adversary” within or not and, if so, what that adversary is doing, you’re going to require a solution that not only collects events, but can correlate them into something intelligent. Your solution should:

  • Consolidate events
  • Focus on the events you are concerned about
  • Provide comprehensive detail about the changes to your systems, security and data

Three software solutions:

  • Netwrix Auditor for AD
  • Dell change auditor for AD
  • IBM QRadar (SIEM)

Splunk (SIEM)  : Splunk Windows Auditing using the NSA guide: https://github.com/anthonygtellez/windows_auditing

MS white-paper best practices to secure AD: http://aka.ms/bpsadtrd

MS Advanced threat analytics (MS ATA): https://www.microsoft.com/en-us/server-cloud/products/advanced-threat-analytics/

Windows Event IDs useful for intrusion detection:

Windows Vista events and above

Category Event ID Description
User Account Changes 4720 Created
4722 Enabled
4723 User changed own password
4724 Privileged User changed this user’s password
4725 Disabled
4726 Deleted
4738 Changed
4740 Locked out
4767 Unlocked
4781 Name change
Domain Controller Authentication Events 4768 TGT was requested
4771 Kerberos pre-auth failed
4772 TGT request failed
Logon Session Events 4624 Successful logon
4647 User initiated logoff
4625 Logon failure
4776 NTLM logon failed
4778 Remote desktop session reconnected
4779 Remote desktop session disconnected
4800 Workstation locked
4801 Workstation unlocked
Domain Group Policy 4739 Domain GPO changed
5136 GPO changed
5137 GPO created
5141 GPO deleted
Security 1102 Event log cleared
Software and Service Installation 6 New Kernel Filter Driver
7045 New Windows Service
1022, 1033 New MSI File Installed
903, 904 New Application Installation
905, 906 Updated Application
907, 908 Removed Application
4688 New Process Created
4697 New Service Installed
4698 New Scheduled Task
External Media Detection 43 New Device Information
400 New Mass Storage Installation
410 New Mass Storage Installation
Group Changes Created Changed Deleted Members
Added Removed
Security Local 4731 4737 4734 4732 4733
Global 4727 4735 4730 4728 4729
Universal 4754 4755 4758 4756 4757
Distribution Local 4744 4745 4748 4746 4747
Global 4749 4750 4753 4751 4752
Universal 4759 4760 4763 4761 4762

Remotely enable PSRemoting and Unrestricted PowerShell Execution using PsExec and PSSession, then run PSRecon

Option 1 — WMI:
PS C:\> wmic /node:”10.10.10.10″ process call create “powershell -noprofile -command Enable-PsRemoting -Force” -Credential Get-Credential

Option 2 – PsExec:
PS C:\> PsExec.exe \\10.10.10.10 -u [admin account name] -p [admin account password] -h -d powershell.exe “Enable-PSRemoting -Force”

Next…

PS C:\> Test-WSMan 10.10.10.10
PS C:\> Enter-PSSession 10.10.10.10
[10.10.10.10]: PS C:\> Set-ExecutionPolicy Unrestricted -Force

Then…

Option 1 — Execute locally in-memory, push evidence to a share, and lock the host down:
[10.10.10.10]: PS C:\> IEX (New-Object Net.WebClient).DownloadString(‘https://github.com/gfoss/PSRecon/psrecon.ps1’)
[10.10.10.10]: PS C:\> Copy-Item PSRecon_* -Recurse [network share]
[10.10.10.10]: PS C:\> rm PSRecon_* -Recurse -Force
[10.10.10.10]: PS C:\> Invoke-Lockdown; exit

Option 2 — Exit PSSession, execute PSRecon remotely, send the report out via email, and lock the host down:
[10.10.10.10]: PS C:\> exit
PS C:\> .\psrecon.ps1 -remote -target 10.10.10.10 -sendEmail -smtpServer 127.0.0.1 -emailTo greg.foss[at]logrhythm.com -emailFrom psrecon[at]logrhythm.com -lockdown

Be careful! This will open the system up to unnecessary risk!!
You could also inadvertently expose administrative credentials when authenticating to a compromised host.
If the host isn’t taken offline, PSRemoting should be disabled along with disallowing Unrestricted PowerShell execution following PSRecon

http://blogs.technet.com/b/askds/archive/2008/08/12/event-logging-policy-settings-in-windows-server-2008-and-vista.aspx

Resources materials:

AD Security:

https://adsecurity.org/?p=1684

https://digital-forensics.sans.org/blog/2013/06/20/overview-of-microsofts-best-practices-for-securing-active-directory

Mimikatz and Active Directory Kerberos Attacks:

https://adsecurity.org/?p=556

https://www.nsa.gov/ia/_files/app/Reducing_the_Effectiveness_of_Pass-the-Hash.pdf

https://www.nsa.gov/ia/_files/app/Spotting_the_Adversary_with_Windows_Event_Log_Monitoring.pdf

http://www.gentilkiwi.com/mimikatz    /   http://blog.gentilkiwi.com/

Scripts:

https://github.com/iadgov/Pass-the-Hash-Guidance
https://github.com/iadgov/Event-Forwarding-Guidance

Domain lockdown: https://github.com/curi0usJack/activedirectory

Microsoft resources:

http://www.microsoft.com/pth

http://download.microsoft.com/download/7/7/A/77ABC5BD-8320-41AF-863C-6ECFB10CB4B9/Mitigating-Pass-the-Hash-Attacks-and-Other-Credential-Theft-Version-2.pdf

http://blogs.technet.com/b/security/archive/2014/07/08/new-strategies-and-features-to-help-organizations-better-protect-against-pass-the-hash-attacks.aspx

 

Pass the Hash – isolation technique:

passTheHash

 

https://gallery.technet.microsoft.com/Cryptography-Fundamentals-0e08b260

Using Powershell:

http://msexchange.me/2014/06/05/monitoring-event-id-thru-powershell/

http://community.spiceworks.com/topic/282720-powershell-event-log-monitor-email-alert-script-central-monitor

https://vijredblog.wordpress.com/2014/03/21/task-scheduler-event-log-trigger-include-event-data-in-mail/

Using SCOM:

http://jimmoldenhauer.blogspot.fr/2013/03/scom-2012-how-to-generate-alerts-from.html

http://scomandplus.blogspot.fr/2013/02/creating-rules-to-monitor-security-logs.html

http://thoughtsonopsmgr.blogspot.fr/2013/11/windows-event-log-monitoring-how-to-get.html

http://opsmgradmin.blogspot.fr/2011/05/scom-monitoring-windows-event-logs.html