Category: Quest ARS


Introduction:

Event forwarding (also called SUBSCRIPTIONS) is a mean to send Windows event log entries from source computers to a collector. A same computer can be a collector or a source.

There are two methods available to complete this challenge – collector initiated and source initiated:

Parameter Collector Initiated (PULL) Source Initiated (PUSH)
Socket direction (for firewall rules) Collector –> Source Collector –> Source
Initiating machine Collector Source
Authentication Type Kerberos Kerberos / Certificates

This technology uses WinRM (HTTP protocol on port TCP 5985 with WinRM 2.0) . Be careful with the Window firewall and configure it to allow WinRM incoming requests.

WinRM is the ‘server’ component and WinRS is the ‘client’ that can remotely manage the machine with WinRM configured.

Differences you should be aware of:

WinRM 1.1 (obsolete)
Vista and Server 2008
Port 80 for HTTP and Port 443 for HTTPS

WinRM 2.0
Windows 7 and Server 2008 R2, 2012 R2 …
Port 5985 for HTTP and Port 5986 for HTTPS

Reference for WEF and event forwarding:

Deploying WinRM using Group Policy: http://www.vkernel.ro/blog/how-to-enable-winrm-http-via-group-policy

Microsoft official document well documented:

https://docs.microsoft.com/en-us/windows/threat-protection/use-windows-event-forwarding-to-assist-in-instrusion-detection

https://www.jpcert.or.jp/english/pub/sr/ir_research.html

Fresh How-to from Intrusion detection perspective:

https://medium.com/@palantir/windows-event-forwarding-for-network-defense-cb208d5ff86f

How-to easy to follow from Intrusion detection perspective:

https://www.root9b.com/sites/default/files/whitepapers/R9B_blog_005_whitepaper_01.pdf

https://joshuadlewis.blogspot.fr/2014/10/advanced-threat-detection-with-sysmon_74.html same than previous one but more appendix

From Intrusion detection perspective:

https://hackernoon.com/the-windows-event-forwarding-survival-guide-2010db7a68c4 help to manage error of WEF deployment

Basic configuration:

on source computers and collector computer:  winrm quickconfig     and add the collector computer account to the local administrators group

To verify a listener has been created type winrm enumerate winrm/config/listener

WinRM Client Setup

Just to round off this quick introduction to WinRM, to delete a listener use winrm delete winrm/config/listener?address=*+Transport=HTTP

on collector computer: wecutil qc. Add the computer account of the collector computer to the Event Log Readers Group on each of the source computers

on collector computer: create a new subscription from event viewer (follow the wizard)

WinRS: WinRS (Windows Remote Shell) is the client that connects to a WinRM configured machine (as seen in the first part of this post). WinRS is pretty handy, you’ve probably used PSTools or SC for similar things in the past. Here are a few examples of what you do.

Connecting to a remote shell
winrs -r:http://hostnameofclient "cmd"
Stop / Starting remote service
winrs -r:http://hostnameofclient "net start/stop spooler"
Do a Dir on the C drive
winrs -r:http://hostnameofclient "dir c:\"

WinRS

Forwarded Event Logs:

This is configured using ‘subscribers’, which connect to WinRM enabled machines.

To configure these subscribers head over to event viewer, right click on forwarded events and select properties. Select the 2nd tab along subscriptions and press create.

This is where you’ll select the WinRM enabled machine and choose which events you would like forwarded.

Subscriptions

Right click the subscription and select show runtime status.

Error 0x80338126

Now it took me a minute or two to figure this one out. Was it a firewall issue (this gives the same error code), did I miss some configuration steps? Well no, it was something a lot more basic than that. Remember earlier on we were talking about the port changes in WinRM 1.1 to 2.0?

That’s right, I was using server 2008 R2 to set the subscriptions which automatically sets the port to 5985. The client I configured initially was server 2008 so uses version 1.1. If you right click the subscription and click properties -> advanced you’ll be able to see this. I changed this to port 80 and checked the runtime status again.

[DC2.domain.local] – Error – Last retry time: 03/02/2011 20:20:30. Code (0x5): Access is denied. Next retry time: 03/02/2011 20:25:30.”

Head back to the advanced settings and change the user account from machine account to a user with administrative rights. After making these changes the forwarded events started to flow.

Subscriptions Advanced

Additional considerations:

In a workgroup environment, you can follow the same basic procedure described above to configure computers to forward and collect events. However, there are some additional steps and considerations for workgroups:

  • You can only use Normal mode (Pull) subscriptions
  • You must add a Windows Firewall exception for Remote Event Log Management on each source computer.
  • You must add an account with administrator privileges to the Event Log Readers group on each source computer. You must specify this account in the Configure Advanced Subscription Settings dialog when creating a subscription on the collector computer.
  • Type winrm set winrm/config/client @{TrustedHosts="<sources>"} at a command prompt on the collector computer to allow all of the source computers to use NTLM authentication when communicating with WinRM on the collector computer. Run this command only once. Where <sources> appears in the command, substitute a list of the names of all of the participating source computers in the workgroup. Separate the names by commas. Alternatively, you can use wildcards to match the names of all the source computers. For example, if you want to configure a set of source computers, each with a name that begins with “msft”, you could type this command winrm set winrm/config/client @{TrustedHosts="msft*"} on the collector computer. To learn more about this command, type winrm help config.

If you configure a subscription to use the HTTPS protocol by using the HTTPS option in Advanced Subscription Settings , you must also set corresponding Windows Firewall exceptions for port 443. For a subscription that uses Normal (PULL mode) delivery optimization, you must set the exception only on the source computers. For a subscription that uses either Minimize Bandwidth or Minimize Latency (PUSH mode) delivery optimizations, you must set the exception on both the source and collector computers.

If you intend to specify a user account by using the Specific User option in Advanced Subscription Settings when creating the subscription, you must ensure that account is a member of the local Administrators group on each of the source computers in step 4 instead of adding the machine account of the collector computer. Alternatively, you can use the Windows Event Log command-line utility to grant an account access to individual logs. To learn more about this command-line utility, type wevtutil sl -? at a command prompt.

References:

http://blogs.technet.com/b/jepayne/archive/2015/11/24/monitoring-what-matters-windows-event-forwarding-for-everyone-even-if-you-already-have-a-siem.aspx

http://blogs.technet.com/b/jepayne/archive/2015/11/20/what-should-i-know-about-security-the-massive-list-of-links-post.aspx

https://technet.microsoft.com/en-us/library/cc748890.aspx

http://windowsitpro.com/security/q-what-are-some-simple-tips-testing-and-troubleshooting-windows-event-forwarding-and-collec

http://technet.microsoft.com/en-us/library/cc749140.aspx

http://blogs.technet.com/b/askperf/archive/2010/09/24/an-introduction-to-winrm-basics.aspx

http://msdn.microsoft.com/en-us/library/aa384372(v=vs.85).aspx

Video:

Tutorials:

1st: Event forwarding between computers in a Domain

http://tutorial.programming4.us/windows_7/Forwarding-Events-(part-1)—How-to-Configure-Event-Forwarding-in-AD-DS-Domains.aspx

2nd: Event forwarding between computers in workgroup

http://tutorial.programming4.us/windows_7/Forwarding-Events-(part-2)—How-to-Troubleshoot-Event-Forwarding—How-to-Configure-Event-Forwarding-in-Workgroup-Environments.aspx

Additional article talking about Event forwarding too:

http://joshuadlewis.blogspot.fr/2014/10/advanced-threat-detection-with-sysmon_74.html

 

Advertisements

Some interesting sites:

Reference articles to secure a Windows domain:

https://github.com/PaulSec/awesome-windows-domain-hardening

Microsoft audit Policy settings and recommendations:

https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/plan/security-best-practices/audit-policy-recommendations

Sysinternals sysmon:

https://onedrive.live.com/view.aspx?resid=D026B4699190F1E6!2843&ithint=file%2cpptx&app=PowerPoint&authkey=!AMvCRTKB_V1J5ow

On ADsecurity.org:

Beyond domain admins: https://adsecurity.org/?p=3700

Gathering AD data with PowerShell: https://adsecurity.org/?p=3719

Hardening Windows computers, secure Baseline check list: https://adsecurity.org/?p=3299

Hardening Windows domain, secure Baseline check list:

Securing Domain Controllers to Improve Active Directory Security

 

Download sysmon:

NEW: Sysmon 6.10 is available ! : https://technet.microsoft.com/en-us/sysinternals/sysmon  and how to use it:

NEW: WMI detections: https://rawsec.lu/blog/posts/2017/Sep/19/sysmon-v610-vs-wmi-persistence/

Installation and usage:

List of web resources concerning Sysmon: https://github.com/MHaggis/sysmon-dfir

Sysmon events table: https://rawsec.lu/blog/posts/2017/Sep/19/sysmon-events-table/

Mark russinovitch’s RSA conference: https://onedrive.live.com/view.aspx?resid=D026B4699190F1E6!2843&ithint=file%2cpptx&app=PowerPoint&authkey=!AMvCRTKB_V1J5ow

Sysmon config files explained:

https://github.com/SwiftOnSecurity/sysmon-config

https://github.com/ion-storm/sysmon-config/blob/master/sysmonconfig-export.xml

https://www.bsk-consulting.de/2015/02/04/sysmon-example-config-xml/

View story at Medium.com

Else other install guides:

Sysinternals Sysmon unleashed

http://www.darkoperator.com/blog/2014/8/8/sysinternals-sysmon

 

Detecting APT with Sysmon:

https://www.rsaconference.com/writable/presentations/file_upload/hta-w05-tracking_hackers_on_your_network_with_sysinternals_sysmon.pdf

https://www.jpcert.or.jp/english/pub/sr/ir_research.html

https://www.root9b.com/sites/default/files/whitepapers/R9B_blog_005_whitepaper_01.pdf

Sysmon with Splunk:

http://blogs.splunk.com/2014/11/24/monitoring-network-traffic-with-sysmon-and-splunk/

https://securitylogs.org/tag/sysmon/

Sysmon log analyzer/parsing sysmon event log:

https://github.com/CrowdStrike/Forensics/blob/master/sysmon_parse.cmd

https://digital-forensics.sans.org/blog/2014/08/12/sysmon-in-malware-analysis-lab

https://github.com/JamesHabben/sysmon-queries

http://blog.crowdstrike.com/sysmon-2/

WEF: https://docs.microsoft.com/en-us/windows/threat-protection/use-windows-event-forwarding-to-assist-in-instrusion-detection

logparser: http://www.microsoft.com/en-us/download/confirmation.aspx?id=24659

logparser GUI: http://lizard-labs.com/log_parser_lizard.aspx

The NSA released a PDF entitled “Spotting the Adversary with Windows Event Log Monitoring” earlier this year. The good news is it’s probably one of the most detailed documents I’ve seen in a long time. Everything from setting up Event Subscriptions, to a hardened use of Windows Remote Management, including the use of authentication and firewalls, this document tells you how to securely setup an environment where you can natively consolidate and monitor event log based entries. In addition, the NSA goes onto cover a number of areas that should be monitored – complete with event IDs:

http://www.redblue.team/2015/09/spotting-adversary-with-windows-event.html

http://www.redblue.team/2015/09/spotting-adversary-with-windows-event_21.html

Event forwarding guidance: https://github.com/iadgov/Event-Forwarding-Guidance

Malware archeology cheat sheets: http://www.malwarearchaeology.com/cheat-sheets/

Machine-specific issues – which can be indications of malicious activity

  • Application Crashes
  • System or Service Failures
  • Kernel and Device Signing
  • The Windows Firewall

Administrator Activity – specific actions performed that may be suspect

  • Clearing of Event Logs
  • Software and Service Installation
  • Remote Desktop Logon
  • Account Usage

The bad news is you’re still left to sort out a TON of event log detail and interpret whether the entries are a problem or not.

Additionally: Changes to Group Policy only show up in the events as a change to the policy, but lack detail on exactly what was changed within the Group Policy.

To truly have a grasp on whether you have an “adversary” within or not and, if so, what that adversary is doing, you’re going to require a solution that not only collects events, but can correlate them into something intelligent. Your solution should:

  • Consolidate events
  • Focus on the events you are concerned about
  • Provide comprehensive detail about the changes to your systems, security and data

Three software solutions:

  • Netwrix Auditor for AD
  • Dell change auditor for AD
  • IBM QRadar (SIEM)

Splunk (SIEM)  : Splunk Windows Auditing using the NSA guide: https://github.com/anthonygtellez/windows_auditing

MS white-paper best practices to secure AD: http://aka.ms/bpsadtrd

MS Advanced threat analytics (MS ATA): https://www.microsoft.com/en-us/server-cloud/products/advanced-threat-analytics/

Windows Event IDs useful for intrusion detection:

Windows Vista events and above

Category Event ID Description
User Account Changes 4720 Created
4722 Enabled
4723 User changed own password
4724 Privileged User changed this user’s password
4725 Disabled
4726 Deleted
4738 Changed
4740 Locked out
4767 Unlocked
4781 Name change
Domain Controller Authentication Events 4768 TGT was requested
4771 Kerberos pre-auth failed
4772 TGT request failed
Logon Session Events 4624 Successful logon
4647 User initiated logoff
4625 Logon failure
4776 NTLM logon failed
4778 Remote desktop session reconnected
4779 Remote desktop session disconnected
4800 Workstation locked
4801 Workstation unlocked
Domain Group Policy 4739 Domain GPO changed
5136 GPO changed
5137 GPO created
5141 GPO deleted
Security 1102 Event log cleared
Software and Service Installation 6 New Kernel Filter Driver
7045 New Windows Service
1022, 1033 New MSI File Installed
903, 904 New Application Installation
905, 906 Updated Application
907, 908 Removed Application
4688 New Process Created
4697 New Service Installed
4698 New Scheduled Task
External Media Detection 43 New Device Information
400 New Mass Storage Installation
410 New Mass Storage Installation
Group Changes Created Changed Deleted Members
Added Removed
Security Local 4731 4737 4734 4732 4733
Global 4727 4735 4730 4728 4729
Universal 4754 4755 4758 4756 4757
Distribution Local 4744 4745 4748 4746 4747
Global 4749 4750 4753 4751 4752
Universal 4759 4760 4763 4761 4762

Remotely enable PSRemoting and Unrestricted PowerShell Execution using PsExec and PSSession, then run PSRecon

Option 1 — WMI:
PS C:\> wmic /node:”10.10.10.10″ process call create “powershell -noprofile -command Enable-PsRemoting -Force” -Credential Get-Credential

Option 2 – PsExec:
PS C:\> PsExec.exe \\10.10.10.10 -u [admin account name] -p [admin account password] -h -d powershell.exe “Enable-PSRemoting -Force”

Next…

PS C:\> Test-WSMan 10.10.10.10
PS C:\> Enter-PSSession 10.10.10.10
[10.10.10.10]: PS C:\> Set-ExecutionPolicy Unrestricted -Force

Then…

Option 1 — Execute locally in-memory, push evidence to a share, and lock the host down:
[10.10.10.10]: PS C:\> IEX (New-Object Net.WebClient).DownloadString(‘https://github.com/gfoss/PSRecon/psrecon.ps1&#8217;)
[10.10.10.10]: PS C:\> Copy-Item PSRecon_* -Recurse [network share]
[10.10.10.10]: PS C:\> rm PSRecon_* -Recurse -Force
[10.10.10.10]: PS C:\> Invoke-Lockdown; exit

Option 2 — Exit PSSession, execute PSRecon remotely, send the report out via email, and lock the host down:
[10.10.10.10]: PS C:\> exit
PS C:\> .\psrecon.ps1 -remote -target 10.10.10.10 -sendEmail -smtpServer 127.0.0.1 -emailTo greg.foss[at]logrhythm.com -emailFrom psrecon[at]logrhythm.com -lockdown

Be careful! This will open the system up to unnecessary risk!!
You could also inadvertently expose administrative credentials when authenticating to a compromised host.
If the host isn’t taken offline, PSRemoting should be disabled along with disallowing Unrestricted PowerShell execution following PSRecon

http://blogs.technet.com/b/askds/archive/2008/08/12/event-logging-policy-settings-in-windows-server-2008-and-vista.aspx

Windows XP/2003/2012 and greater support drive mapping back to the client workstation during a Terminal Services (Remote Desktop) session. This means you can copy files from the server to the client and vice versa.

Each volume (removable, fixed or network) available on the client workstation is mapped (A for drive A:, C for drive C:, X for drive X: etc) and the remote Terminal Services session inherits the user’s permission. So if you are logged on to the workstation as user A and you log in to the Terminal Services server as user B, the session will have access to the drives according to A’s permissions.

Drives can also be mapped like a network drive. The client drives are accessible as \\TSCLIENT\C. Note the client workstation’s machine name is not used, it is always referenced with the generic name TSCLIENT.

To display the files on TSCLIENT:

DIR \\TSCLIENT\C

So you can map a drive as follows:

NET USE Y: \\TSCLIENT\C

or simply use the Universal Naming Convention (UNC) syntax:

COPY \\TSCLIENT\C\MYDIR\*.XLS D:\DOCUMENTS\ 

example:

ROBOCOPY \\TSCLIENT\C\MYDIR D:\DOCUMENTS *.XLS /Z /ETA

ROBOCOPY \\TSCLIENT\C\MYDIR D:\DOCUMENTS *.* /MIR /Z /ETA /r:1 /w:1 /Log+:d:\log.txt

 

Note: If you receive an “Attempt to access invalid address” error when using the UNC path \\tsclient\c, then the problem is on the client side.

Likely, the Windows firewall is turned on and blocking file shares, or “File and Printer Sharing For Microsoft Networks” is turned off in the NIC properties, the Server service is disabled, or simple file sharing is enabled on the client

 

How to create and deploy a client certificate for MAC: http://blogs.technet.com/b/configmgrteam/archive/2013/04/05/how-to-create-and-deploy-a-client-cert-for-mac-independently-from-configmgr.aspx

Transforming .cer to .pem or vice-versa: https://www.sslshopper.com/ssl-converter.html

using openssl to convert a certificate format to another format: https://support.ssl.com/Knowledgebase/Article/View/19/0/der-vs-crt-vs-cer-vs-pem-certificates-and-how-to-convert-them

Exporting a private key: https://technet.microsoft.com/en-us/library/cc754329.aspx

 

 

Using Powershell:

http://msexchange.me/2014/06/05/monitoring-event-id-thru-powershell/

http://community.spiceworks.com/topic/282720-powershell-event-log-monitor-email-alert-script-central-monitor

https://vijredblog.wordpress.com/2014/03/21/task-scheduler-event-log-trigger-include-event-data-in-mail/

Using SCOM:

http://jimmoldenhauer.blogspot.fr/2013/03/scom-2012-how-to-generate-alerts-from.html

http://scomandplus.blogspot.fr/2013/02/creating-rules-to-monitor-security-logs.html

http://thoughtsonopsmgr.blogspot.fr/2013/11/windows-event-log-monitoring-how-to-get.html

http://opsmgradmin.blogspot.fr/2011/05/scom-monitoring-windows-event-logs.html

 

 

 

 

Troubleshooting slow logons:

http://blogs.technet.com/b/askds/archive/2009/09/23/so-you-have-a-slow-logon-part-1.aspx

http://blogs.technet.com/b/askds/archive/2009/09/24/so-you-have-a-slow-logon-part-2.aspx

Logon process: http://fr.slideshare.net/ControlUp/understanding-troubleshooting-the-windows-logon-process

Tools for troubleshooting:

http://social.technet.microsoft.com/wiki/contents/articles/10128.tools-for-troubleshooting-slow-boots-and-slow-logons-sbsl.aspx

http://social.technet.microsoft.com/wiki/contents/articles/10123.troubleshooting-slow-operating-system-boot-times-and-slow-user-logons-sbsl.aspx

And powershell:

http://blogs.citrix.com/2015/08/05/troubleshooting-slow-logons-via-powershell/

Analyze GPOs load time: http://www.controlup.com/script-library/Analyze-GPO-Extensions-Load-Time/ee682d01-81c4-4495-85a7-4c03c88d7263/

 

How to use Xperf, Xbootmgr, Procmon, WPA?

xperf;xbootmgr;xperfview comes from Windows ADK (Windows performance toolkit sub part). Procmon is a sysinternal tool.

http://superuser.com/questions/594625/how-can-i-analyze-performance-issues-before-during-the-logon-process

http://blogs.technet.com/b/askpfeplat/archive/2012/06/09/slow-boot-slow-logon-sbsl-a-tool-called-xperf-and-links-you-need-to-read.aspx

http://social.technet.microsoft.com/wiki/contents/articles/10128.tools-for-troubleshooting-slow-boots-and-slow-logons-sbsl.aspx

Other interesting articles:

http://blogs.technet.com/b/askpfeplat/archive/2014/10/27/becoming-an-wpa-xpert-part-11-troubleshooting-long-group-policy-processing.aspx

https://www.autoitconsulting.com/site/performance/windows-performance-toolkit-simple-boot-logging/

https://randomascii.wordpress.com/2012/09/04/windows-slowdown-investigated-and-identified/

https://randomascii.wordpress.com/2013/04/20/xperf-basics-recording-a-trace-the-easy-way/

 

Windows Performance Analyzer (wpa.exe) youtube: https://www.youtube.com/watch?v=HGTlc_gWH_c

Xperf data collection tool: https://xperf123.codeplex.com/releases/view/66888

 

For boot tracing:

http://www.msfn.org/board/topic/140247-trace-windows-7-bootshutdownhibernatestandbyresume-issues/

xbootmgr -trace boot -traceFlags BASE+CSWITCH+POWER -resultPath C:\TEMP

with boot phases:
xbootmgr -trace boot -traceflags base+latency+dispatcher -stackwalk profile+cswitch+readythread 
       -notraceflagsinfilename -postbootdelay 120 -resultPath C:\TEMP
 

For shutdown tracing:

xbootmgr -trace shutdown -noPrepReboot -traceFlags BASE+CSWITCH+DRIVERS+POWER -resultPath C:\TEMP

For Standby+Resume:

xbootmgr -trace standby -traceFlags BASE+CSWITCH+DRIVERS+POWER -resultPath C:\TEMP

For Hibernate+Resume:

xbootmgr -trace hibernate -traceFlags BASE+CSWITCH+DRIVERS+POWER -resultPath C:\TEMP

replace C:\TEMP with any temp directory on your machine as necessary to store the output files

Analyses of the boot trace:

Boot_MainPathBoot.png

To start create a summary xml file, run this command (replace the name with the name of your etl file)

xperf /tti -i boot_BASE+CSWITCH+POWER_1.etl -o summary_boot.xml -a boot

Analyses of the shutdown trace:

The shutdown is divided into this 3 parts:

Shutdown_picture.png

To generate an XML summary of shutdown, use the -a shutdown action with Xperf:

xperf /tti -i shutdown_BASE+CSWITCH+DRIVERS+POWER_1.etl -o summary_shutdown.xml -a shutdown