Event Resolution and Escalation Policy
Written by Robert Green
Updated at February 15th, 2021
Purpose
This document defines policy regarding how Dizzion responds to customer and service delivery events and when escalations are warranted due to events.
Problem Detection: Alarms and Alerts
Problem detection typically occurs by alerts or alarms being generated as a result of system or network event. Once a problem has been detected, the Operations Department is notified and is prompted to investigate and resolve the issue. The monitoring system or network that generated the alert will automatically issue a Priority 4 ticket. This priority assignment is modified as additional information is gathered about the alarm/alert. Depending on the severity of the issue, an Escalation may be called to action to assist in resolving the issue. Each priority number has a specific definition detailed below.
Problem Resolution
Monitoring systems are watched and reviewed by the Operations Department. When an alarm is generated, the monitoring system or an Operations Department team member, create a ticket and the Operations Department investigates the event. The trouble ticket serves as an information tracking tool and allows other Dizzion staff members to maintain awareness of the situation. Once the ticket has been issued and the initial investigation has been completed, the Operations Department will take at least one of four possible actions:
Work to Resolve the Issue |
Support Engineers address many customer impacting issues as they arise. If resolution of a service impacting issue is not readily apparent after the initial investigation, the Support Engineer is to go back to the ticket priority definitions and follow the procedures for the appropriate trouble ticket priority. Support Engineers are not limited by the assigned priority procedure and are requested to change the priority and/or escalate as they deem necessary based on their investigation with the exception of Priority 1 and 2 tickets. These tickets are never downgraded. |
Contact the Customer |
Many alarms are generated as a result of unannounced customer maintenance, network problems at customer or user locations, or general distress in the customer or user environment. Dizzion will contact customers in the event of a service impacting issue unless the customer has instructed otherwise. |
Escalation |
Architects or SME’s are available 24x7x365 for escalation support. An escalated event will be brought to the attention of the SME by the Support Engineer. The Support Engineer will continue to monitor the progress of the situation by maintaining appropriate contact with impacted customers, gathering information, and performing more advanced diagnostics under the supervision and guidance of the SME. Issues impacting multiple customers will be escalated to a SME upon recognition. Solutions Engineers own the ticket through resolution, regardless of escalation to an SME. |
Partner/Vendor Escalation |
When an event arises that involves partner/vendor equipment or services and affects an individual or multiple Dizzion customers, Dizzion will open a ticket with the vendor and will work to resolve the issue. The issued ticket will be assigned an appropriate priority number and will follow the Dizzion escalation and guideline procedures for that specific priority number assignment. |
After an issue has been escalated to the SME, the process will take one of the following possible paths:
Work to Resolution |
All SMEs have access to appropriate network devices and systems. SMEs have the skill and authority to address any type of service impacting event. In the case of a broad issue impacting multiple customers, the SME must initiate a managerial escalation. |
Technical Escalations |
Should the on duty SME be faced with a particularly difficult situation, or a situation with which they are personally unfamiliar, they should escalate to a additional SME’s as appropriate. In this case a technical escalation is usually accompanied by a managerial escalation. |
Managerial Escalations |
This type of escalation is warranted when unfamiliar issues or events that impact multiple customers arise or an individual customer has ongoing affect. Such situations will result in the immediate notification of the appropriate management team members. |
Once an issue has reached a SME and/or a Managerial Escalation level, one of several possible approaches will be taken to address it:
Continue Working to Resolution |
At this stage, resolution becomes a team effort involving all personnel that are able to contribute to resolution. |
Vendor Escalation |
Dizzion invests in support contracts with all primary vendors to ensure that hardware and non-hardware vendors stand behind us when things go awry. |
Service Re-delivery |
Dizzion maintains inventory of spare equipment that can be deployed to replace failing hardware or to provide an alternate means of delivering services. Failed or malfunctioning equipment can be replaced from spares to restore services while Dizzion continues to address the root cause of an event. |
Customer Communication
At any time Support Engineers. SME’s or Management Team members may enlist the assistance of the customer’s Technical Account Manager to deliver information to the customer – particularly regarding limitations of the customer’s contracted services that may be a cause of the event. Dizzion staff will utilize Technical Account Management assistance when necessary and/or available, but not stop working the issue to resolution.
Priority Codes
Each ticket generated by Dizzion monitoring systems is assigned a priority code. Priority codes are determined based on the severity of the issue. Priorities range from 0 to 4. Responses to tickets with designated priority codes follow actions to be taken within a certain time of the ticket opening. The time frames provided are general ranges and will not apply to all issues or incident. Time frames begin when a ticket has been opened and each time frame is a measure of the elapsed time from when the ticket was opened.
Priority Zero (0)
Many tickets generated by Dizzion’s monitoring systems begin as Priority 0 (zero). Dizzion Support Engineers are responsible for evaluating these tickets and reassigning their priority.
Priority One (1)
Support Engineer(s) are responsible for determining whether a situation warrants a Priority 1 trouble ticket assignment. In general, a Priority 1 assignment is made any time a multiple-customer or general service impacting issue arises that requires immediate resolution to restore service. Thirty minute response time SLA.
Priority Two (2)
A Priority 2 event is defined by a disruption of Dizzion provided services to an individual customer. Sixty minute response time SLA.
Priority Three (3)
This priority is utilized for degraded state service delivery but not disrupted or down services. These events either have impact to customer services or potential to have impact if left unhandled, but are not a complete disruption of services. Four hour response time during business hours. After business hours next business day SLA.
Priority Four (4)
Ticket priorities 4 are reserved for non-service impacting issues. These tickets are reviewed by Support Engineers and may be re-prioritized and or forwarded to other groups as deemed necessary.
Ticket Handoff or Forwarding
With a majority of the customer requests and problem notifications going through Support Engineers, there will be events that need to be worked by someone outside of this group. All alerts and service requests that are not critical and cannot be resolved by Support Engineers should be directed to the customers Technical Account Manager for review, redirection and/or escalation as appropriate.