What is machine data?

Machine data is information automatically generated by computer software in the form of a log message, debug message, error message, or status message, as a response to an event. Usuallly the information was generated for the purpose of troubleshooting computer software and computer hardware. Machine data includes informative information about the performance of an operating system from services and processes like package management, resource utilization, and network management.

AWS Logs

Informative data from support service monitoring, alarms and a dashboards for metrics, and can also track security-relevant activities, such as login and logout events.

Authentication

Authentication data can help identify users that are struggling to log in to applications and provide insight into potentially anomalous behaviors, such as activities from different locations within a specified time period.

Firewall

Firewall data can provide visibility into blocked traffic in case an application is having communication problems. It can also be used to help identify traffic to malicious and unknown domains.

Network Statistics

Network statistical data can provide visibility into the network's role in overall availability and performance of critical services. It's also an important source for identifying advanced persistent threats.

System Logs

System logs are key to troubleshooting system problems and can be used to alert security teams to network attacks, a security breach or compromised software.

Web Server

Web logs are critical in debugging web application and server problems, and can also be used to detect attacks, such as SQL injections.

Sensor Data

Sensor data can provide visibility into system performance and support compliance reporting of devices. It can also be used to proactively identify systems that require maintenance.

Comments

Network traffic monitoring in Linux with Python

You can investigate suspicious activity in your network traffic by collecting relevant machine data from your endpoint. You can use the machine data to create your own analysis. Before you start your investigation you will need to determine normal activity on your endpoint. Normal activity is the scope of functionality of the software on your endpoint during periods of low activity and high activity. You will need some kind of software that periodically collects specific machine data from your endpoint like my software developed in Python that's available for free download at https://github.com/davidbrennerjr/server-stats-collector Ingest one or more of the following machine data: Application specific logs from /var/log Raw dumps from sniffing at Layers 2-3 Raw dumps from /proc of kernel data structures Raw dumps of kernel routing tables General system-wide error messages from /var/log/syslog Do you

OpenStack+Ceph as Software-Defined Storage

SDS reduces the costs of the management of growing data stores by decoupling storage management from its hardware to allow for centralized management of cheaper, popular commodity hardware. The example SDS ecosystem uses open source software like OpenStack as a front-end interface on top of Ceph as the resource provider of a RADOS cluster of commodity solid-state drives. OpenStack provides user-friendly wrappers for accessing and modifying underlying Ceph storage. OpenStack comes in the form of distributed microservices with RESTful API's: Block (Cinder), File (Manila), Image (Glance), and Object (Swift). Each microservice can scale-out as a cluster of stand-alone services to accommodate the varying demands of high-growth storage. With OpenStack the underlying Ceph storage can address the block storage needs, file storage needs, image storage needs, and object storage needs of datacenters adopting open source as their new norm in an industry trend for high performace and high a

Application behavior monitoring in Linux with Python

You can monitor application behaviors by collecting relevant machine data from your endpoint. You can use the machine data to investigate suspicious activity and create your own analysis. Before you start your investigation you will need to determine normal activity on your endpoint. Normal activity is the scope of functionality of the software on your endpoint during periods of low activity and high activity. You will need some kind of software that periodically collects specific machine data from your endpoint like my software developed in Python that's available for free download at https://github.com/davidbrennerjr/server-stats-collector Ingest one or more of the following machine data from Category #1. Ingest one or more of the following machine data from Category #2. Category #1 General system-wide error messages from /var/log/syslog Auditing logs of application rulesets Auditing logs of security contexts Auditing logs of

David Brenner Jr's Random Things

Search This Blog