Incident communication templates
Short updates for each stage of an outage, written the way customers read them: what is affected, what you are doing, and when they will hear from you next.
Templates
Replace the parts in [brackets].
Investigating
Within minutes of the first alert, even if you know very little yet.
We're investigating reports that [service] is [unavailable / slow / returning errors] for [some / all] customers. [What still works, if anything.] Next update by [time, UTC].
Identified
Once you know the cause, before the fix is finished.
We've found the cause: [plain description, e.g. a database server ran out of disk space]. We're [what you are doing to fix it]. [Workaround, if there is one.] Next update by [time, UTC].
Monitoring
After the fix is in place, while you watch that it holds.
A fix is in place and [service] is working normally again. We're watching it to make sure it stays that way. [Anything customers need to do, e.g. retry a failed payment.] Next update by [time, UTC].
Resolved
When you are confident the problem is over.
This incident is resolved. From [start time] to [end time] UTC ([duration]), [what customers experienced]. The cause was [one sentence]. [What you are changing, or: we'll publish a full write-up within (n) days.] We're sorry for the disruption.
Slower than usual
When the service works but is degraded, not down.
[Service] is working but slower than usual: [e.g. pages take several seconds to load]. No data has been lost. We're looking into it. Next update by [time, UTC].
A provider is down
When the cause is a vendor you depend on.
[Provider], which we use for [purpose], is having an outage. Until it recovers, [what customers will notice]. We're following their status page and will update here when it is resolved.
Writing tips
- Post the first update quickly. "We're looking into it" within five minutes is worth more than a full explanation an hour later.
- Name what is affected in your customers' words ("checkout", "login"), not your service names.
- Always give the time of the next update, and post one by then even if nothing has changed.
- Don't guess the cause in the first update. Say what you see; the cause goes into Identified.
- Use one time zone everywhere and say which one.
In OpenUptime
- A monitor that crosses its failure threshold opens an incident by itself, with a first Investigating update that says what the checks saw.
- Each update carries one of four stages, Investigating, Identified, Monitoring or Resolved, and appears on the status page timeline.
- Updates go by email to the status page's subscribers who follow the affected component.
- Mark an update internal to keep it inside the team: it never reaches the status page, its feed or subscriber emails.
Publish these on a status page
Status pages, incident updates and subscriber emails are included on the Free plan. No card needed.
Questions
How often should I post incident updates?
Every 30 to 60 minutes during a major outage is common, and more often in the first hour. What matters more is to promise a time for the next update and keep it.
Should I say what caused the outage?
Yes, once you know it, in one plain sentence. Customers want to know whether it can happen again, not the internals. Keep details that could help an attacker out of public updates.
What is the difference between Monitoring and Resolved?
Monitoring means a fix is live and things look normal, but you are still watching. Resolved means you are confident it is over. Moving to Resolved too early, then reopening, costs more trust than waiting.
故障通报模板
按故障的每个阶段准备好的简短通报，按客户的阅读习惯来写：哪些功能受影响、你在做什么、下一次更新在什么时候。
模板
替换 [方括号] 中的内容。
调查中
收到第一条告警后的几分钟内发出，哪怕还知道得很少。
我们正在调查 [服务] 对 [部分 / 全部] 用户 [无法访问 / 响应缓慢 / 报错] 的问题。[仍然可用的功能，如有。] 下一次更新不晚于 [时间，北京时间]。
已定位
找到原因之后、修复完成之前。
我们已经找到原因：[用通俗的话描述，例如一台数据库服务器磁盘已满]。目前正在 [修复措施]。[临时解决办法，如有。] 下一次更新不晚于 [时间，北京时间]。
观察中
修复上线之后，确认它是否稳定的阶段。
修复已经上线，[服务] 已恢复正常，我们会继续观察，确认不再复发。[用户需要做的事，例如重新提交失败的付款。] 下一次更新不晚于 [时间，北京时间]。
已恢复
确认问题已经结束时。
本次故障已经恢复。北京时间 [开始时间] 至 [结束时间]（共 [时长]），[用户遇到的情况]。原因是 [一句话说明]。[后续改进，或：我们会在 (n) 天内发布完整的复盘。] 给你带来的不便，我们深表歉意。
性能下降
服务可用但变慢，并未中断时。
[服务] 目前可以使用，但比平时慢：[例如页面需要几秒才能打开]。没有数据丢失，我们正在排查。 下一次更新不晚于 [时间，北京时间]。
第三方服务故障
原因出在你依赖的第三方服务时。
我们用于 [用途] 的 [服务商] 正在发生故障。在它恢复之前，[用户会遇到的情况]。我们正在关注对方的状态页，恢复后会在这里更新。
写作建议
- 第一条通报要快。五分钟内发出一句“正在排查”，比一小时后的完整解释更有用。
- 用客户的说法描述受影响的部分（“下单”“登录”），不要用内部服务名。
- 每次都写明下一次更新的时间，到点就更新，哪怕没有新进展。
- 第一条通报不要猜原因，只写看到的现象；原因放到“已定位”里再写。
- 全程使用同一个时区，并注明是哪个时区。
在 OpenUptime 中
- 监控达到失败阈值时会自动创建故障，并附上一条“调查中”更新，写明检查发现了什么。
- 每条更新都属于调查中、已定位、观察中、已恢复四个阶段之一，并显示在状态页的时间线上。
- 更新会通过邮件发送给关注受影响组件的状态页订阅者。
- 把更新设为内部可见，它就只留在团队内部，不会出现在状态页、订阅源或订阅邮件中。
常见问题
故障期间多久更新一次？
严重故障时，常见做法是每 30 到 60 分钟更新一次，第一个小时内更频繁。更重要的是写明下一次更新的时间，并按时更新。
需要说明故障原因吗？
需要，在确认之后用一句通俗的话说明。用户关心的是会不会再发生，而不是内部细节。可能帮到攻击者的细节不要写进公开通报。
“观察中”和“已恢复”有什么区别？
“观察中”表示修复已上线、看起来正常，但仍在观察；“已恢复”表示你确认问题已经结束。过早标记为已恢复、随后又重新打开，比多等一会儿更伤信任。