Category: Virtualization


Microsoft’s file systems organize storage devices based on cluster size. Also known as the allocation unit size, cluster size represents the smallest amount of disk space that can be allocated to hold a file. Because ReFS and NTFS don’t reference files at a byte granularity, the cluster size is the smallest unit of size that each file system can reference when accessing storage. Both ReFS and NTFS support multiple cluster sizes, as different sized clusters can offer different performance benefits, depending on the deployment.

Full article from MS: https://blogs.technet.microsoft.com/filecab/2017/01/13/cluster-size-recommendations-for-refs-and-ntfs/

Summary:

ReFS cluster sizes

ReFS offers both 4K and 64K clusters. 4K is the default cluster size for ReFS, and we recommend using 4K cluster sizes for most ReFS deployments because it helps reduce costly IO amplification:

  • In general, if the cluster size exceeds the size of the IO, certain workflows can trigger unintended IOs to occur. Consider the following scenarios where a ReFS volume is formatted with 64K clusters:
    • Consider a tiered volume. If a 4K write is made to a range currently in the capacity tier, ReFS must read the entire cluster from the capacity tier into the performance tier before making the write. Because the cluster size is the smallest granularity that the file system can use, ReFS must read the entire cluster, which includes an unmodified 60K region, to be able to complete the 4K write.
    • If a cluster is shared by multiple regions after a block cloning operation occurs, ReFS must copy the entire cluster to maintain isolation between the two regions. So if a 4K write is made to this shared cluster, ReFS must copy the unmodified 60K cluster before making the write.
    • Consider a deployment that enables integrity streams. A sub-cluster granularity write will cause the entire cluster to be re-allocated and re-written, and the new checksum must be computed. This represents additional IO that ReFS must perform before completing the new write, which introduces a larger latency factor to the IO operation.
  • By choosing 4K clusters instead of 64K clusters, one can reduce the number of IOs that occur that are smaller than the cluster size, preventing costly IO amplifications from occurring as frequently.

Additionally, 4K cluster sizes offer greater compatibility with Hyper-V IO granularity, so we strongly recommend using 4K cluster sizes with Hyper-V on ReFS.  64K clusters are applicable when working with large, sequential IO, but otherwise, 4K should be the default cluster size.

NTFS cluster sizes

NTFS offers cluster sizes from 512 to 64K, but in general, we recommend a 4K cluster size on NTFS, as 4K clusters help minimize wasted space when storing small files. We also strongly discourage the usage of cluster sizes smaller than 4K. There are two cases, however, where 64K clusters could be appropriate:

  • 4K clusters limit the maximum volume and file size to be 16TB
    • 64K cluster sizes can offer increased volume and file capacity, which is relevant if you’re are hosting a large deployment on your NTFS volume, such as hosting VHDs or a SQL deployment.
  • NTFS has a fragmentation limit, and larger cluster sizes can help reduce the likelihood of reaching this limit
    • Because NTFS is backward compatible, it must use internal structures that weren’t optimized for modern storage demands. Thus, the metadata in NTFS prevents any file from having more than ~1.5 million extents.
      • One can, however, use the “format /L” option to increase the fragmentation limit to ~6 million. Read more here.
    • 64K cluster deployments are less susceptible to this fragmentation limit, so 64K clusters are a better option if the NTFS fragmentation limit is an issue. (Data deduplication, sparse files, and SQL deployments can cause a high degree of fragmentation.)
      • Unfortunately, NTFS compression only works with 4K clusters, so using 64K clusters isn’t suitable when using NTFS compression. Consider increasing the fragmentation limit instead, as described in the previous bullets.

While a 4K cluster size is the default setting for NTFS, there are many scenarios where 64K cluster sizes make sense, such as: Hyper-V, SQL, deduplication, or when most of the files on a volume are large.

Download sysmon:

NEW: Sysmon 6.0 is available ! : https://technet.microsoft.com/en-us/sysinternals/sysmon  and how to use it:

Installation and usage:

List of web resources concerning Sysmon: https://github.com/MHaggis/sysmon-dfir

Mark russinovitch’s RSA conference: https://onedrive.live.com/view.aspx?resid=D026B4699190F1E6!2843&ithint=file%2cpptx&app=PowerPoint&authkey=!AMvCRTKB_V1J5ow

Sysmon config files explained:

https://github.com/SwiftOnSecurity/sysmon-config

https://github.com/ion-storm/sysmon-config/blob/master/sysmonconfig-export.xml

https://www.bsk-consulting.de/2015/02/04/sysmon-example-config-xml/

View story at Medium.com

Else other install guides:

Sysinternals Sysmon unleashed

http://www.darkoperator.com/blog/2014/8/8/sysinternals-sysmon

 

Detecting APT with Sysmon:

https://www.rsaconference.com/writable/presentations/file_upload/hta-w05-tracking_hackers_on_your_network_with_sysinternals_sysmon.pdf

 

https://www.root9b.com/sites/default/files/whitepapers/R9B_blog_005_whitepaper_01.pdf

Sysmon with Splunk:

http://blogs.splunk.com/2014/11/24/monitoring-network-traffic-with-sysmon-and-splunk/

https://securitylogs.org/tag/sysmon/

Sysmon log analyzer/parsing sysmon event log:

https://github.com/CrowdStrike/Forensics/blob/master/sysmon_parse.cmd

https://digital-forensics.sans.org/blog/2014/08/12/sysmon-in-malware-analysis-lab

https://github.com/JamesHabben/sysmon-queries

http://blog.crowdstrike.com/sysmon-2/

logparser: http://www.microsoft.com/en-us/download/confirmation.aspx?id=24659

logparser GUI: http://lizard-labs.com/log_parser_lizard.aspx

Ref: https://blogs.technet.microsoft.com/matthts/2012/10/13/windows-server-sockets-logical-processors-symmetric-multi-threading/

With the release of each version of Windows Server, the maxima for supported number of sockets and logical processors have increased.

Whatever the numbers for the older OS releases, the best scalable platform for virtualization is Windows Server 2012 (and of course Hyper-V Server 2012), far exceeding VMware 5.1 and others.
Let’s start by defining common terminology.

Terminology

Logical Processor: A thread of execution on a physical processing unit which can be a core or a thread on a symmetric multi-threaded (SMT) system.

Virtual Processor: A virtualized instance of a logical processor exposed to virtual machines.

Socket: What a physical processor plugs into.

How to create and deploy a client certificate for MAC: http://blogs.technet.com/b/configmgrteam/archive/2013/04/05/how-to-create-and-deploy-a-client-cert-for-mac-independently-from-configmgr.aspx

Transforming .cer to .pem or vice-versa: https://www.sslshopper.com/ssl-converter.html

using openssl to convert a certificate format to another format: https://support.ssl.com/Knowledgebase/Article/View/19/0/der-vs-crt-vs-cer-vs-pem-certificates-and-how-to-convert-them

Exporting a private key: https://technet.microsoft.com/en-us/library/cc754329.aspx

 

 

Using Powershell:

http://msexchange.me/2014/06/05/monitoring-event-id-thru-powershell/

http://community.spiceworks.com/topic/282720-powershell-event-log-monitor-email-alert-script-central-monitor

https://vijredblog.wordpress.com/2014/03/21/task-scheduler-event-log-trigger-include-event-data-in-mail/

Using SCOM:

http://jimmoldenhauer.blogspot.fr/2013/03/scom-2012-how-to-generate-alerts-from.html

http://scomandplus.blogspot.fr/2013/02/creating-rules-to-monitor-security-logs.html

http://thoughtsonopsmgr.blogspot.fr/2013/11/windows-event-log-monitoring-how-to-get.html

http://opsmgradmin.blogspot.fr/2011/05/scom-monitoring-windows-event-logs.html

 

 

 

 

wusa <update>.msu /quiet /norestart /log

example: wusa d:\hotfixes\Windows8.1-KB29456426.msu /quiet /norestart

You can use the Windows Management Instrumentation Command-line (WMIC) to view the installed updates on your computer:

wmic qfe list

Caption CSName Description FixComments HotFixID InstallDate InstalledBy InstalledOn Name ServicePackInEffect Status

Else If the WMIC output is difficult to read, you can use Systeminfo instead, as follows:

systeminfo | findstr /i /c:”KB29456426″

[18]: KB29456426

How to use WUSA with Powershell?

Get-Item .\* | %{Expand-ZipFile -FilePath $_.FullName -OutputPath d:\hotfixes}

Get-Item d:\hotfixes\* | foreach {WUSA “”$_.FullName /quiet /norestart””;while(get-process wusa){Write-Host “Installing $_.Name”}}

Get-HotFix | Where Description -match hotfix
(Get-HotFix | Where Description -match hotfix).count

Introduction:

Event forwarding (also called SUBSCRIPTIONS) is a mean to send Windows event log entries from source computers to a collector. A same computer can be a collector or a source.

There are two methods available to complete this challenge – collector initiated and source initiated:

Parameter Collector Initiated Source Initiated
Socket direction (for firewall rules) Collector –> Source Collector –> Source
Initiating machine Collector Source
Authentication Type Kerberos Kerberos / Certificates

This technology uses WinRM (HTTP protocol on port TCP 5985 with WinRM 2.0, else TCP 80) . Be careful with the Window firewall and configure it to allow WinRM incoming requests.

WinRM is the ‘server’ component and WinRS is the ‘client’ that can remotely manage the machine with WinRM configured.

Differences you should be aware of:

WinRM 1.1
Vista and Server 2008
Port 80 for HTTP and Port 443 for HTTPS

WinRM 2.0
Windows 7 and Server 2008 R2
Port 5985 for HTTP and Port 5986 for HTTPS

WinRM 1.1 can also be downloaded and installed on pre-R2 2003 and XP from here.

Windows Server 2008 Core:

In order to forward events from a 2008 Server that is not R2, you will need to make a few changes. The first change is the default listening port, it needs to be changed from TCP 80 to TCP 5985. Additionally you may need to start the Windows Event Collector Service.

net start wecsvc
winrm set winrm/config/listener?Address=*+Transport=HTTP @{Port=”5985”}

Basic configuration:

on source computers and collector computer:  winrm quickconfig     and add the collector computer account to the local administrators group

To verify a listener has been created type winrm enumerate winrm/config/listener

WinRM Client Setup

Just to round off this quick introduction to WinRM, to delete a listener use winrm delete winrm/config/listener?address=*+Transport=HTTP

on collector computer: wecutil qc. Add the computer account of the collector computer to the Event Log Readers Group on each of the source computers

on collector computer: create a new subscription from event viewer (follow the wizard)

WinRS: WinRS (Windows Remote Shell) is the client that connects to a WinRM configured machine (as seen in the first part of this post). WinRS is pretty handy, you’ve probably used PSTools or SC for similar things in the past. Here are a few examples of what you do.

Connecting to a remote shell
winrs -r:http://hostnameofclient "cmd"
Stop / Starting remote service
winrs -r:http://hostnameofclient "net start/stop spooler"
Do a Dir on the C drive
winrs -r:http://hostnameofclient "dir c:\"

WinRS

Forwarded Event Logs:

This is configured using ‘subscribers’, which connect to WinRM enabled machines.

To configure these subscribers head over to event viewer, right click on forwarded events and select properties. Select the 2nd tab along subscriptions and press create.

This is where you’ll select the WinRM enabled machine and choose which events you would like forwarded.

Subscriptions

Right click the subscription and select show runtime status.

Error 0x80338126

Now it took me a minute or two to figure this one out. Was it a firewall issue (this gives the same error code), did I miss some configuration steps? Well no, it was something a lot more basic than that. Remember earlier on we were talking about the port changes in WinRM 1.1 to 2.0?

That’s right, I was using server 2008 R2 to set the subscriptions which automatically sets the port to 5985. The client I configured initially was server 2008 so uses version 1.1. If you right click the subscription and click properties -> advanced you’ll be able to see this. I changed this to port 80 and checked the runtime status again.

[DC2.domain.local] – Error – Last retry time: 03/02/2011 20:20:30. Code (0x5): Access is denied. Next retry time: 03/02/2011 20:25:30.”

Head back to the advanced settings and change the user account from machine account to a user with administrative rights. After making these changes the forwarded events started to flow.

Subscriptions Advanced

Additional considerations:

In a workgroup environment, you can follow the same basic procedure described above to configure computers to forward and collect events. However, there are some additional steps and considerations for workgroups:

  • You can only use Normal mode (Pull) subscriptions
  • You must add a Windows Firewall exception for Remote Event Log Management on each source computer.
  • You must add an account with administrator privileges to the Event Log Readers group on each source computer. You must specify this account in the Configure Advanced Subscription Settings dialog when creating a subscription on the collector computer.
  • Type winrm set winrm/config/client @{TrustedHosts="<sources>"} at a command prompt on the collector computer to allow all of the source computers to use NTLM authentication when communicating with WinRM on the collector computer. Run this command only once. Where <sources> appears in the command, substitute a list of the names of all of the participating source computers in the workgroup. Separate the names by commas. Alternatively, you can use wildcards to match the names of all the source computers. For example, if you want to configure a set of source computers, each with a name that begins with “msft”, you could type this command winrm set winrm/config/client @{TrustedHosts="msft*"} on the collector computer. To learn more about this command, type winrm help config.

If you configure a subscription to use the HTTPS protocol by using the HTTPS option in Advanced Subscription Settings , you must also set corresponding Windows Firewall exceptions for port 443. For a subscription that uses Normal (PULL mode) delivery optimization, you must set the exception only on the source computers. For a subscription that uses either Minimize Bandwidth or Minimize Latency (PUSH mode) delivery optimizations, you must set the exception on both the source and collector computers.

If you intend to specify a user account by using the Specific User option in Advanced Subscription Settings when creating the subscription, you must ensure that account is a member of the local Administrators group on each of the source computers in step 4 instead of adding the machine account of the collector computer. Alternatively, you can use the Windows Event Log command-line utility to grant an account access to individual logs. To learn more about this command-line utility, type wevtutil sl -? at a command prompt.

References:

http://blogs.technet.com/b/jepayne/archive/2015/11/24/monitoring-what-matters-windows-event-forwarding-for-everyone-even-if-you-already-have-a-siem.aspx

http://blogs.technet.com/b/jepayne/archive/2015/11/20/what-should-i-know-about-security-the-massive-list-of-links-post.aspx

https://technet.microsoft.com/en-us/library/cc748890.aspx

http://windowsitpro.com/security/q-what-are-some-simple-tips-testing-and-troubleshooting-windows-event-forwarding-and-collec

http://technet.microsoft.com/en-us/library/cc749140.aspx

http://blogs.technet.com/b/askperf/archive/2010/09/24/an-introduction-to-winrm-basics.aspx

http://msdn.microsoft.com/en-us/library/aa384372(v=vs.85).aspx

Video:

Tutorials:

1st: Event forwarding between computers in a Domain

http://tutorial.programming4.us/windows_7/Forwarding-Events-(part-1)—How-to-Configure-Event-Forwarding-in-AD-DS-Domains.aspx

2nd: Event forwarding between computers in workgroup

http://tutorial.programming4.us/windows_7/Forwarding-Events-(part-2)—How-to-Troubleshoot-Event-Forwarding—How-to-Configure-Event-Forwarding-in-Workgroup-Environments.aspx

Additional article talking about Event forwarding too:

http://joshuadlewis.blogspot.fr/2014/10/advanced-threat-detection-with-sysmon_74.html

 

Symptom:

You get a dialog box like “The file name is too long” or “The source file name(s) are larger than is supported by the file system”. Or “cannot delete folder: it is being used by another person or program”, or, “cannot delete file: Access is denied There has been a sharing violation”. The source or destination file may be in use.

Description:

Basically, there is a character limit set in naming or renaming files in your Windows operating system and it varies from one OS to another. Mostly it varies between 256 and 260 characters. Thus, when you transfer files with long names from one destination to another, you will experience path too long error in Windows or Linux systems.

History:

Maximum Path Length Limitation In the Windows API (with some exceptions discussed in the following paragraphs), the maximum length for a path is MAX_PATH, which is defined as 260 characters. A local path is structured in the following order: drive letter, colon, backslash, name components separated by backslashes, and a terminating null character. 1+2+256+1 or [drive][:][path][null] = 260. One could assume that 256 is a reasonable fixed string length from the DOS days. And going back to the DOS APIs we realize that the system tracked the current path per drive, and we have 26 (32 with symbols) maximum drives (and current directories). The INT 0x21 AH=0x47 says “This function returns the path description without the drive letter and the initial backslash.” So we see that the system stores the CWD as a pair (drive, path) and you ask for the path by specifying the drive (1=A, 2=B, …), if you specify a 0 then it assumes the path for the drive returned by INT 0x21 AH=0x15 AL=0x19. So now we know why it is 260 and not 256, because those 4 bytes are not stored in the path string. Why a 256 byte path string, because 640K is enough RAM.

reference: http://msdn.microsoft.com/en-us/library/aa365247%28VS.85%29.aspx

but, but, but: the NTFS filesystem supports paths up to 32k characters. You can use the win32 api and “\\?\” prefix the path to use greater than 260 characters. http://blogs.msdn.com/b/bclteam/archive/2007/02/13/long-paths-in-net-part-1-of-3-kim-hamilton.aspx

The Windows OS since Vista support path length of 32k, but unfortunately most of the applications are limited to 255 chars long !

 

Workarounds:

a) Try moving to a location which has a shorter name, or try renaming to shorter name(s) before attempting this operation.

b) To get list of files with long path, you can use this powershell script:

Write-Host “Please wait, searching…”
robocopy.exe $srcdir c:\doesnotexist /l /e /b /np /fp /njh /njs /ndl | Where-Object {$_.Length -ge 255} | ForEach-Object {$_.Substring(26,$_.Length-26)}

after this audit phase, rename the files to be shorter!

There is also a free command line tool to detect long paths, the “too long path detector”: http://sourceforge.net/projects/tlpd/

On Linux;

With GNU find (on Linux or Cygwin), you can look for files whose relative path is more than 255 characters long:

find -regextype posix-extended -regex '.{257,}'           (257 accounts for the initial ./.)

c) Use “Unlocker” http://www.filehippo.com/download_unlocker/  in the case of you cannot delete folder: It is being used by another person or program or cannot delete file: Access is denied There has been a sharing violation. Unlocker can help! Simply right-click the folder or file and select Unlocker. If the folder or file is locked, a window listing of lockers will appear.

d) But to delete a file whose name is more than 255 characters:

  1. Open a command prompt by running “CMD.EXE”
  2. Navigate to the folder holding the file
  3. Use the command “dir /x” which will display the short names of files.
  4. Delete using the short name.

i.e. if the file is named “verylongfilename.doc”, the shortname will display as something like “verylo~1.doc” and you can delete using that name.

c) In the case where you have a too long directory, you can use the subst command:

  1. Start a command prompt (no admin privileges needed)
  2. Use cd to navigate to the folder you want to go (you can use tab to autocomplete names)
  3. type subst x: . to create the drive letter association. (instead of the . you can also type the entire path)
  4. Now in explorer you have a new letter. Go to it and do whatever you need to do to copy or delete files.
  5. Go back to your cmd window and type subst /d x: to remove the drive or alternatively, restart your pc

d) Another way to cope with the path limit is to shorten path entries with symbolic links.

  1. create a c:\folder directory to keep short links to long paths
  2. mklink /J C:\folder\foo c:\Some\Crazy\Long\Path\foo
  3. add c:\folder\foo to your path instead of the long path

http://blogs.technet.com/b/askds/archive/2011/09/26/advanced-xml-filtering-in-the-windows-event-viewer.aspx

http://blog.oneboredadmin.com/2013/05/filtering-windows-event-log-using-xpath.html

 

 

 

 

How to install the Windows failover clustering from the command line ?

First, you should make sure that the nodes, running Windows Server 2012 R2 that you are intending to add to the cluster are part of the same domain, and proceed to install the Failover-Cluster feature on them. This is very similar to conventional Cluster installs running on Windows Servers. To install the feature, you can use the Server Manager to complete the installation.

Server Manager can be used to install the Failover Clustering feature:
Introducing Server Manager in Windows Server 2012
http://blogs.technet.com/b/askcore/archive/2012/11/04/introducing-server-manager-in-windows-server-2012.aspx

We can alternatively use PowerShell (Admin) to install the Failover Clustering feature on the nodes.
Install-WindowsFeature -Name Failover-Clustering -IncludeManagementTools

An important point to note is that PowerShell Cmdlet ‘Add-WindowsFeature’ is being replaced by ‘Install-WindowsFeature’ in Windows Server 2012 R2. PowerShell does not install the management tools for the feature requested unless you specify ‘-IncludeManagementTools’ as part of your command.

BONUS READ:
The Cluster Command line tool (CLUSTER.EXE) has been deprecated; but, if you still want to install it, it is available under:
Remote Server Administration Tools –> Feature Administration Tools –> Failover Clustering Tools –> Failover Cluster Command Interface in the Server Manager

The PowerShell (Admin) equivalent to install it:
Install-WindowsFeature -Name RSAT-Clustering-CmdInterface

Now that we have Failover Clustering feature installed on our nodes. Ensure that all connected hardware to the nodes passes the Cluster Validation tests. Let us now go on to create our cluster. You cannot create an AD detached clustering from Cluster Administrator and the only way to create the AD-Detached Cluster is by using PowerShell.
New-Cluster MyCluster -Node My2012R2-N1,My2012R2-N2 -StaticAddress 192.168.1.15 -NoStorage -AdministrativeAccessPoint DNS

NOTE:
In my example above, I am using static IP Addresses, so one would need to be specified. If you are using DHCP for addresses, the switch “-StaticAddress 192.168.1.15 ” would be excluded from the command.

Once we have executed the command, we would have a new cluster created with the name “MyCluster” with two nodes “My2012R2-N1” and “My2012R2-N2”. When you look Active Directory, there will not be a computer object created for the Cluster “MyCluster”; however, you would see the record as the Access Point in DNS.