Network Security Through Data Analysis. From Data

Network Security Through Data Analysis. From Data - Opis

Traditional intrusion detection and logfile analysis are no longer enough to protect today...s complex networks. In the updated second edition of this practical guide, security researcher Michael Collins shows InfoSec personnel the latest techniques and tools for collecting and analyzing network traffic datasets. You...ll understand how your network is used, and what actions are necessary to harden and defend the systems within it.In three sections, this book examines the process of collecting and organizing data, various tools for analysis, and several different analytic scenarios and techniques. New chapters focus on active monitoring and traffic manipulation, insider threat detection, data mining, regression and machine learning, and other topics.You...ll learn how to:Use sensors to collect network, service, host, and active domain dataWork with the SiLK toolset, Python, and other tools and techniques for manipulating data you collectDetect unusual phenomena through exploratory data analysis (EDA), using visualization and mathematical techniquesAnalyze text data, traffic behavior, and communications mistakesIdentify significant structures in your network with graph analysisExamine insider threat data and acquire threat intelligenceMap your network and identify significant hosts within itWork with operations to develop defenses and analysis techniques Spis treści:
Preface
Audience
Contents of This Book
Changes Between Editions
Conventions Used in This Book
Using Code Examples
OReilly Safari
How to Contact Us
Acknowledgments
I. (...) więcej Data
1. Organizing Data: Vantage, Domain, Action, and Validity
Domain
Vantage
Choosing Vantage
Actions: What a Sensor Does with Data
Validity and Action
Internal Validity
External Validity
Construct Validity
Statistical Validity
Attacker and Attack Issues
Further Reading
2. Vantage: Understanding Sensor Placement in Networks
The Basics of Network Layering
Network Layers and Vantage
Network Layers and Addressing
MAC Addresses
MAC format and access
IPv4 Format and Addresses
IPv6 Format and Addresses
Validity Challenges from Middlebox Network Data
DHCP
NAT
Proxies
Load balancing
VPNs
Further Reading
3. Sensors in the Network Domain
Packet and Frame Formats
Rolling Buffers
Limiting the Data Captured from Each Packet
Filtering Specific Types of Packets
What If Its Not Ethernet?
NetFlow
NetFlow v5 Formats and Fields
NetFlow v9 and IPFIX
NetFlow Generation and Collection
Data Collection via IDS
Classifying IDSs
IDS as Classifier
Improving IDS Performance
Enhancing IDS Detection
Configuring Snort
Enhancing IDS Response
Prefetching Data
Middlebox Logs and Their Impact
VPN Logs
Proxy Logs
NAT Logs
Further Reading
4. Data in the Service Domain
What and Why
Logfiles as the Basis for Service Data
Accessing and Manipulating Logfiles
The Contents of Logfiles
The Characteristics of a Good Log Message
Existing Logfiles and How to Manipulate Them
Stateful Logfiles
Further Reading
5. Sensors in the Service Domain
Representative Logfile Formats
HTTP: CLF and ELF
Simple Mail Transfer Protocol (SMTP)
Sendmail
Microsoft Exchange: Message Tracking Logs
Additional Useful Logfiles
Staged Logging
LDAP and Directory Services
File Transfer, Storage, and Databases
Logfile Transport: Transfers, Syslog, and Message Queues
Transfer and Logfile Rotation
Syslog
Further Reading
6. Data and Sensors in the Host Domain
A Host: From the Networks View
The Network Interfaces
The Host: Tracking Identity
Processes
Structure
PID and PPID
UID
Command and path
Memory, CPU, terminal, and start time
Filesystem
Historical Data: Commands and Logins
Other Data and Sensors: HIPS and AV
Further Reading
7. Data and Sensors in the Active Domain
Discovery, Assessment, and Maintenance
Discovery: ping, traceroute, netcat, and Half of nmap
Checking Connectivity: Using ping to Connect to an Address
Tracerouting
Using nc as a Swiss Army Multitool
nmap Scanning for Discovery
Assessment: nmap, a Bunch of Clients, and a Lot of Repositories
Basic Assessment with nmap
Using Active Vantage Data for Verification
Further Reading
II. Tools
8. Getting Data in One Place
High-Level Architecture
The Sensor Network
The Repository
Archive
Annotation
Knowledge base
Query Processing
Real-Time Processing
Source Control
Log Data and the CRUD Paradigm
A Brief Introduction to NoSQL Systems
Further Reading
9. The SiLK Suite
What Is SiLK and How Does It Work?
Acquiring and Installing SiLK
The Datafiles
Choosing and Formatting Output Field Manipulation: rwcut
Basic Field Manipulation: rwfilter
Ports and Protocols
Size
IP Addresses
Time
TCP Options
Helper Options
Miscellaneous Filtering Options and Some Hacks
rwfileinfo and Provenance
Combining Information Flows: rwcount
rwset and IP Sets
rwuniq
rwbag
Advanced SiLK Facilities
PMAPs
Collecting SiLK Data
YAF
rwptoflow
rwtuc
rwrandomizeip
Further Reading
10. Reference and Lookup: Tools for Figuring Out Who Someone Is
MAC and Hardware Addresses
IP Addressing
IPv4 Addresses, Their Structure, and Significant Addresses
IPv6 Addresses, Their Structure, and Significant Addresses
IP Intelligence: Geolocation and Demographics
DNS
DNS Name Structure
Forward DNS Querying Using dig
The DNS Reverse Lookup
Using whois to Find Ownership
DNS Blackhole Lists
Search Engines
General Search Engines
Scanning Repositories, Shodan et al
Further Reading
III. Analytics
An Overview of Attacker Behavior
Further Reading
11. Exploratory Data Analysis and Visualization
The Goal of EDA: Applying Analysis
EDA Workflow
Variables and Visualization
Univariate Visualization
Histograms
Bar Plots (Not Pie Charts)
The Five-Number Summary and the Boxplot
Generating a Boxplot
Bivariate Description
Scatterplots
Multivariate Visualization
Other Visualizations and Their Role
Pairs plots and trellising
Spider plots
ROC curves
Operationalizing Security Visualization
Rule one: Bound and partition your visualization to manage disruptions
Rule two: Label anomalies
Rule three: Use trendlines, distinguish artifacts from observations
Rule four: Be consistent across plots
Rule five: Annotate with contextual information
Rule six: Avoid flash in favor of expressiveness
Rule seven: When performing long jobs, give the user some status feedback
Fitting and Estimation
Is It Normal?
Simply Visualizing: Projected Values and QQ Plots
Fit Tests: K-S and S-W
Further Reading
12. On Analyzing Text
Text Encoding
Unicode, UTF, and ASCII
Encoding for Attackers
Base64 encoding
Informal encoding/obfuscation
Compression
Encryption
Basic Skills
Finding a String
Manipulating Delimiters
Splitting Along Delimiters
Regular Expressions
Techniques for Text Analysis
N-Gram Analysis
Jaccard Distance
Hamming Distance
Levenshtein Distance
Entropy and Compressibility
Homoglyphs
Further Reading
13. On Fumbling
Fumbling: Misconfiguration, Automation, and Scanning
Lookup Failures
Automation
Scanning
Identifying Fumbling
IP Fumbling: Dark Addresses and Spread
TCP Fumbling: Failed Sessions
Unidirectional flow filtering
Dark ports and UDP fumbling
ICMP Messages and Fumbling
Fumbling at the Service Level
HTTP Fumbling
SMTP Fumbling
DNS Fumbling
Detecting and Analyzing Fumbling
Building Fumbling Alarms
Forensic Analysis of Fumbling
Engineering a Network to Take Advantage of Fumbling
14. On Volume and Time
The Workday and Its Impact on Network Traffic Volume
Beaconing
File Transfers/Raiding
Locality
DDoS, Flash Crowds, and Resource Exhaustion
DDoS and Routing Infrastructure
Applying Volume and Locality Analysis
Data Selection
Using Volume as an Alarm
Using Beaconing as an Alarm
Using Locality as an Alarm
Engineering Solutions
Further Reading
15. On Graphs
Graph Attributes: What Is a Graph?
Labeling, Weight, and Paths
Components and Connectivity
Clustering Coefficient
Analyzing Graphs
Using Component Analysis as an Alarm
Using Centrality Analysis for Forensics
Using Breadth-First Searches Forensically
Using Centrality Analysis for Engineering
Further Reading
16. On Insider Threat
Insider Threat Versus Other Classes of Attacks
Avoiding Toxicity
Modes of Attack
Data Theft and Exfiltration
Credential Theft
Sabotage
Insider Threat Data: Logistics and Collection
Applying Sector-Based Workflow to Insider Threat
Physical Data Sources
Keeping Track of User Identity
Further Reading
17. On Threat Intelligence
Defining Threat Intelligence
Data Types
Types of threat intelligence data
Maturity and format of threat intelligence data
Provenance of threat intelligence data
Creating a Threat Intelligence Program
Identifying Goals
Starting with Free Sources
Determining Data Output
Purchasing Sources
Brief Remarks on Creating Threat Intelligence
Further Reading
18. Application Identification
Mechanisms for Application Identification
Port Number
Application Identification by Banner Grabbing
Application Identification by Behavior
Application Identification by Subsidiary Site
Application Banners: Identifying and Classifying
Non-Web Banners
Web Client Banners: The User-Agent String
Further Reading
19. On Network Mapping
Creating an Initial Network Inventory and Map
Creating an Inventory: Data, Coverage, and Files
Phase I: The First Three Questions
The default network
Phase II: Examining the IP Space
Identifying asymmetric traffic
Identifying dark space
Finding network appliances
Phase III: Identifying Blind and Confusing Traffic
Identifying NATs
Identifying proxies
Identifying VPN traffic
Phase IV: Identifying Clients and Servers
Identifying servers
Identifying Sensing and Blocking Infrastructure
Updating the Inventory: Toward Continuous Audit
Further Reading
20. On Working with Ops
Ops Environments: An Overview
Operational Workflows
Escalation Workflow
Sector Workflow
Hunting Workflow
Hardening Workflow
A hardening scenario
Forensic Workflow
Switching Workflows
Further Readings
21. Conclusions
Index O autorze: Michael Roy is a virtualization expert working for VMware, the world’s leading virtualization and cloud computing company used by 95 percent of Fortune 500 companies to manage and virtualize their data center infrastructure. He currently specializes in technical marketing for hybrid cloud services, but started at VMware working on VMware Fusion 2 in 2009 where he co-led a world-class global support team, giving customers the help they needed to get the most out of VMware Fusion. Prior to VMware, he managed a small data center in downtown Toronto as the senior systems administrator, and also spent two years working for Apple where he became a Mac expert. When not working with computers, clouds, or virtual machines, he likes to snowboard, surf, and cook, and listen to, play, and create music. Michael’s success story at VMware can be found at https://www.vmware.com/go/mikeroy-journey. mniej