Splunk On-Call (VictorOps)

Use the Splunk On-Call integration to triage alerts and remediate them automatically.

Overview

This integration is intended for customers who have Splunk On-Call set up as the primary alerting and on-call management system.

Use Cases

The integration between Fylamynt and Splunk On-Call lets you:

• Reroute a list of Splunk On-Call incidents.

• Resolve a list of Splunk On-Call incidents.

• Trigger a workflow in Fylamynt, when a specific Splunk On-Call Incident goes into the state of ALERT.

Configure Splunk On-Call in Fylamynt

• Navigate to Settings > Integrations > Splunk On-Call (VictorOps)

• Configure a new integration instance

Details needed to set up Splunk On-Call (VictorOps) instance in Fylamynt:

Parameter	Description	Required
API ID	API ID obtained from Splunk On-Call (VictorOps)	True
API Key	API Key obtained from Splunk On-Call (VictorOps)	True
Webhook API Key Name	Fylamynt API Key name	True
Splunk On-Call Webhook URL	Fylamynt Webhook URL to be used by Splunk On-Call (VictorOps) service

Webhook is used to fetch alerts from Splunk On-Call (VictorOps) in real-time. You can also Enable / Disable Splunk On-Call (VictorOps) Webhook.

Follow the steps listed below in your Splunk On-Call (VictorOps) Account and get the desired parameters to enter in Fylamynt.

1. In your Splunk On-Call Account, go to "Integrations "-->" API"

• "Activate API" if API Access if OFF, else copy the available API ID and API Key.

• Add the API ID and API Key details to the "Authorize Fylamynt" panel.

2. Now setup Splunk On-Call to Fylamynt Outgoing Webook from"Integrations" --> "Outgoing Webhook"

"Add Webhook" with below details: -- Event: Select "Incident-Triggered" from dropdown -- Method: POST -- Content Type: application/json -- Custom Headers: Key: x-api-key Value: add "Webhook API Key Value" from right side panel --To: Copy webhook URL from "Webhook URL" in right side panel --Payload: Copy below to Payload section

{
  "INCIDENT.INCIDENT_ID": "${{INCIDENT.INCIDENT_ID}}",
  "INCIDENT.INCIDENT_NAME": "${{INCIDENT.INCIDENT_NAME}}",
  "INCIDENT.CURRENT_PHASE": "${{INCIDENT.CURRENT_PHASE}}",
  "INCIDENT.ENTITY_TYPE": "${{INCIDENT.ENTITY_TYPE}}",
  "INCIDENT.SERVICE": "${{INCIDENT.SERVICE}}",
  "INCIDENT.ENTITY_STATE": "${{INCIDENT.ENTITY_STATE}}",
  "INCIDENT.POLICIES_PAGED.0.POLICY.NAME": "${{INCIDENT.POLICIES_PAGED.0.POLICY.NAME}}",
  "INCIDENT.POLICIES_PAGED.0.TEAM.NAME": "${{INCIDENT.POLICIES_PAGED.0.TEAM.NAME}}",
  "INCIDENT.MONITOR_TYPE": "${{INCIDENT.MONITOR_TYPE}}",
  "INCIDENT.MONITOR_NAME": "${{INCIDENT.MONITOR_NAME}}"
}

Instructions to get Fylamynt "Webhook API Key value": Under Authorize Fylamynt, choose one of the Webhook API Key Name from the dropdown, and the corresponding Webhook API Key Value should be shown. If no options are available for the Webhook API Key Name, first go to Settings --> API Keys --> Manage Keys to Create a New API Key

Integration Actions

You can add these actions in the Fylamynt workflow builder, as part of your workflow.

1. Splunk On-Call Alert Trigger

2. Splunk On-Call Reroute Incident

3. Splunk On-Call Resolve Incident

Splunk On-Call Alert Trigger

The integration node triggers the automatic execution of a workflow from a combined selection of a Splunk On-Call Team and Escalation Policy.

When creating a workflow, you are presented with a wizard to select the trigger type to use.

• On the workflow page, select New Workflow

• Enter the name of the Workflow.

• Select the Splunk On-Call trigger type.

• Click Create Workflow

• 

Configure the automatic execution of a workflow

To automatically run workflows with the Splunk On-Call Alert trigger, the incident type and assignment need to be configured. Follow the step-by-step instructions provided on the Incident Management - Automatic workflow execution page.

Splunk On-Call Reroute Incidents

Integration node to re-route selected Splunk On-Call incidents

Input

Parameter Name	Description	Required
Select Username	Select Username to re-route incidents	True
incident_numbers	Select one or more incidents to re-route	True
target_users	Select one or more users to re-route incident	False *
target_policies	Select one or more escalation policies to re-route incident	False *

*target_policies and/or target_policies have to be selected as mandatory for Reroute incidents action.

Output

Parameter Name	Type	Description
execution_message	Object	JSON of Splunk On-Call re-route incident execution
execution_status	String	Status of node execution

Action Example

• Drag and drop the Splunk On-call Reroute Action node onto the Workflow Editor Canvas

• Select the Action node

• Add the Incident Numbers

• Select the Username

• Select either or both of the following:

  • Select one or more target users to re-route the Splunk On-Call incident

  • Select one or more escalation policies to re-route the Splunk On-Call incident

Splunk On-Call Resolve Incidents

Integration node to resolve selected Splunk On-Call incidents

Input

Parameter Name	Description	Required
Username	Select Username to resolve incidents	True
incident_numbers	Select one or more incidents to resolve	True

Output

Parameter Name	Type	Description
execution_message	Object	JSON of Splunk On-Call resolve incident execution
execution_status	String	Status of node execution

Action Example

• Drag and drop the Splunk On-call Resolve Action node onto the Workflow Editor Canvas

• Select the Action node

• Add the Incident Numbers

• Select the Username