Introduction
About the Alert Manager
The Alert Manager adds simple incident workflows to Splunk. The general purpose is to provide a common app with dashboards in order to investigate fired alerts or notable events. It can be used with every Splunk alert and works as an extension on top of Splunk's built-in alerting mechanism.
- Awareness of your current operational situation with the incident posture dashboard
- Analyze root cause of incidents with only a few clicks
- Review and adjust the urgency of incidents to improve operations scheduling
- ITIL based impact x urgency = priority matrix to classify incidents
- Dispatch incidents to the person in charge
- Track and report incident workflow KPIs
- Tag and categorize incidents
Features
- Works as custom alert action to catch enriched metadata of fired alerts and stores them in a configurable separate index
- Each fired alert creates an incident
- Reassign incidents manually or auto-assign them to specific users
- Change incidents to another urgency and status
- Incidents can be configured to get auto-resolved when a new incident is created from the same alert
- Incidents can be configured to get auto-resolved when the alert's ttl is reached
- Incidents can be configured to get auto-resolved when the new matching suppression rules are added
- Incidents can be configured to get auto-resolved when they are informal
- Configure full-whitelabeled email notifications upon several events
- Report on incidents, their attributes and workflow updates
- External Workflow Actions for manually triggering external actions
- Support for storing external reference ids per incident
- Alert Status customization
- Quick assignment
- Optionally index alert results instead of storing in KV Store
Prerequisites
- All indexers and search heads require Splunk Enterprise version 6.3 or later.
How does it work
The Alert Manager works as a Custom Alert Action in Splunk (Learn More ). When an alert fires, the Custom Alert Action catches all the parameters and job details from the alert search and writes aggregated data to an index and a state to a collection in Splunk's App Key Value Store.
Data generated by Alert Manager
Different kind of data is written by the Alert Manager. Events are written into a configurable index.
- One metadata event per new incident with sourcetype alert_metadata (up to 5 kilobytes per incident)
- One event per each change on an incident (change owner, priority or status) with sourcetype incident_change (less than 1 kilobyte per change)
- Additionally, alert results are stored in a collection, size depends on the amount of results returned by the search
The total amount of data generated by Alert Manager depends on number of fired alerts. All event data is written to Splunk through the API and counts towards the license volume.
Datamodels used by Alert Manager
All the metadata the Alert Manager creates upon incident creation is covered by a datamodel named Alert Manager. There are two main root objects:
- All Alerts: Provides a list of incidents at the time it was created (Constraint: eventtype="alert_metadata")
- All Incident Changes: A list of modifications to incidents (status update, ownership change or comment added, Constraint:
eventtype="incident_change" action="create" OR action="change")
There are several dashboards (Incident Posture, Incident Export and Incident Overview) which use the Alert Manager datamodel to retrieve data. The datamodel is NOT accelerated by default. We recommend to enable the acceleration as dashboards, such as the Incident Posture will load faster.
Indexes used by Alert Manager
By default, the Alert Manager creates and uses an index alerts. The index name used by the App can be changed during the installation process.
Custom Commands provided by Alert Manager
The Alert Manager provides Custom Search Commands (loadindicentresults and modifyincidents) to use the data generated by the app in your own search searches. Please see the Reference for further details.
How this app fits into the Splunk picture
How to get Support
In general, the App is community supported. Feel free to ask questions on Splunk Answers (just add the "Alert Manager" tag to your question) or submit an issue at GitHub. We also provide individual installation and configuration support. Please contact support@alertmanager.info for details.