An incident postmortem is the written review of an incident after it is over: what happened, the impact, a timeline, the root cause, what fixed it and the follow-up work. Google's SRE book calls it a written record of an incident, its impact, the actions taken, the root causes and the follow-up actions. A blameless postmortem looks for causes in systems and process, not in people.
- A postmortem has a summary, impact, timeline, root cause, resolution and action items with owners.
- Blameless means asking what made the mistake possible, not who made it.
- Write it within days, while people remember. PagerDuty sets 3 calendar days for its worst incidents.
- Security postmortems add detection, containment and eradication, and are usually shared more narrowly.
What is an incident postmortem?
Google's SRE book defines it this way: "a postmortem is a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions to prevent the incident from recurring" (Google SRE book). The root cause is the underlying condition that let the incident happen, as opposed to the trigger that set it off.
The same document goes by several names. PagerDuty lists "Learning Review After-Action Review Incident Review Incident Report Post-Incident Review Root Cause Analysis (or RCA)" as terms organizations use (PagerDuty). Security teams more often say post-incident review. Common reasons to write one, per Google, include user-visible downtime, data loss of any kind and "on-call engineer intervention (release rollback, rerouting of traffic, etc.)" (Google SRE book).
What should a postmortem template include?
PagerDuty publishes the template it uses for major incidents (PagerDuty template). Its sections, in plain words:
| Section | What goes in it |
|---|---|
| Overview | One or two sentences: what broke, for how long, and the impact |
| What happened | A short description of the event |
| Contributing factors | Conditions that led to it, including actions that made it worse |
| Resolution | What fixed it, and any temporary fix still in place |
| Impact | Exact numbers: time at each severity, users and accounts affected, tickets raised |
| Responders | Incident commander, scribe and others involved |
| Timeline | Timestamped events, each with a link to the data that shows it |
| How we did | What went well and what did not |
| Action items | Follow-up work, each as a ticket with an owner |
On the timeline, the template asks for the time the contributing factor began, the time of the page, the status page update, significant actions, the end of the incident and "links to tools/logs that show how the timestamp was arrived at" (PagerDuty template). It also asks that "each action item should be in the form of a JIRA ticket", which makes the follow-up work traceable.
What is a blameless postmortem?
A blameless postmortem looks for what in the system made a mistake possible or likely, not for someone to punish. Google writes that "for a postmortem to be truly blameless, it must focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior" (Google SRE book). Atlassian puts it as: "in a blameless postmortem, it's assumed that every team and employee acted with the best intentions based on the information they had at the time" (Atlassian).
Blameless does not mean no accountability. Action items still have owners. PagerDuty adds that "doing the postmortem is not a punishment, and the owner is not the person that "caused" the incident" (PagerDuty). In practice, write "the deploy pipeline allowed an untested config change" rather than "Sam pushed a bad config".
Postmortem example 1: an outage after a config change
The postmortem below is an illustration written for this page, about a fictional online store. A load balancer change takes down part of checkout. Note how the trigger (a changed health-check path) is kept apart from the root cause (older servers that could not answer it), and how each action item has a ticket.
Postmortem: checkout errors after a load balancer change (INC-2291)
Status: complete, 2 of 4 action items done Severity: SEV-2
Summary For 41 minutes, about 30% of checkout requests failed with HTTP 502.
Impact Errors for customers in two regions; no data lost.
Trigger A health-check path changed in a load balancer config change (PR #812).
Root cause The new path returned 404 on older app servers, so the balancer
marked them unhealthy and sent all traffic to the newer servers.
Detection Error-rate alert paged the on-call engineer at 14:07 UTC.
Resolution Rolled back PR #812 at 14:44 UTC; errors stopped at 14:48 UTC.
Timeline (UTC)
13:58 PR #812 merged and applied by the deploy pipeline
14:07 Page: checkout 5xx rate over 5% for 5 minutes
14:15 Incident declared, incident commander named, chat channel opened
14:31 Responders link the start of errors to the 13:58 change
14:44 Rollback applied (PR #815)
14:48 Error rate back to normal; incident resolved at 15:05
Action items
1. Add a health-check test to the load balancer pipeline (TICKET-4410) done
2. Roll config changes to one region first (TICKET-4411) open
3. Alert on unhealthy-host count, not only on errors (TICKET-4412) done
4. Update runbook "Checkout 5xx" with the rollback steps (TICKET-4413) open
For a longer published example, the Google SRE book includes a full sample postmortem of a fictional search outage: "Shakespeare Search down for 66 minutes during period of very high interest in Shakespeare due to discovery of a new sonnet" (Google SRE book, appendix D).
Postmortem example 2: a security event
A security post-incident review uses the same frame with security phases added: detection, containment, eradication and recovery, as in the incident response playbook phases. NIST's incident response guide notes that "lessons learned from incident response activities and root cause analysis help improve cybersecurity risk management" (NIST SP 800-61r3). The example below is fictional and written for this page.
Post-incident review: leaked deploy key used against a build server (SEC-118)
Status: complete Severity: high Personal data affected: none found
Summary A deploy key committed to a public fork was used to log in to a build
server. The attacker listed files and tried to reach the artifact
store, and was blocked by the firewall. Access lasted 3 hours.
Detection SIEM alert at 02:12 UTC: SSH login from a new country to build-07.
Containment Key revoked 02:41; build-07 isolated 02:44; outbound rule added 02:50.
Eradication build-07 rebuilt from a known image; all deploy keys rotated.
Root cause Keys were stored in a repo config file; secret scanning ran only on
the main repository, not on forks.
Timeline (UTC)
23:05 Key appears in a public fork (found later in the hosting provider's logs)
23:40 First login to build-07 from 198.51.100.23
02:12 SIEM alert; SOC ticket SEC-118 opened
02:41 Key revoked (change CHG-3307)
02:44 Host isolated by the endpoint tool
02:50 Firewall egress rule added (PR #77 in the firewall config repo)
09:30 Review meeting; 5 action items filed
Action items
1. Scan forks and history for secrets (SEC-119)
2. Move deploy keys to the secret manager (SEC-120)
3. Alert on logins from new countries to build hosts within 15 minutes (SEC-121)
Two differences from an outage postmortem: the timeline starts before anyone noticed (when the key leaked), and the review records whether personal data was affected, because that decides who must be notified.
Where are public postmortem examples published?
Many cloud, hosting and software companies publish postmortems for major outages on their status pages and engineering blogs. A long-running public list on GitHub is described simply as "a collection of postmortems" (danluu/post-mortems), grouped by type of failure, such as config errors, hardware and time. PagerDuty's postmortem guide and Google's SRE book both carry worked samples.
Public postmortems are edited for customers. They leave out internal tickets, chat logs, names and the exact change that fixed the problem. The internal version, joined to its ticket and its fix, is a different and fuller record. For how the full record is built, see the incident timeline template.
Common mistakes in incident postmortems
These come up again and again in postmortem reviews.
- Stopping at the trigger ("a bad deploy") and never reaching the condition that let it cause harm.
- Naming a person as the cause. People then stop reporting what they did.
- A timeline with no sources, so nobody can check a timestamp later.
- Action items with no owner or ticket, which quietly never get done.
- Writing it weeks later, after the chat history and memories have faded.
- Copying customer data, passwords or tokens from logs into the document.
Are incident postmortems also a data asset?
They can be, when they are joined to the rest of the incident. AI data buyers have asked Origin for incidents joined to the change that fixed them: the ticket, the chat thread, the config diff or pull request, and the outcome, together with the postmortems, runbooks and on-call records from the same period. The records must cover the company's own infrastructure (a client's only where that client has agreed), and the company must own them. A postmortem alone is a summary. A postmortem linked to its ticket and its fix shows the reasoning and the result. Licensing software incident reviews covers postmortems as a record type, and the hub page covers who holds these records.
Origin is an independent data partner. Your company decides what is shared and signs any license itself, and a buyer may pay Origin when a deal closes.
- Secrets and keys quoted in timelines or diffs: find them and rotate them first (remove secrets from git history).
- Customer names, account IDs and impact numbers tied to named customers.
- Names of responders: replace them with consistent role labels such as "on-call 1".
- IP addresses and hostnames in log excerpts (anonymize logs and captures).
- Postmortems written for clients, and security reviews under a confidentiality or regulator duty, stay out unless the duty allows sharing.
Does your company hold records like these? See what your data is worth.
Common questions
What is the difference between a postmortem and a post-incident review?
Mostly the name. Both are the written review held after an incident ends, with a timeline, the cause, the fix and follow-up work. Operations and SRE teams tend to say postmortem; security teams and many IT service teams say post-incident review. PagerDuty lists both among the names for the same process.
What makes a postmortem blameless?
A blameless postmortem looks for the conditions that made a mistake possible, not for a person to punish. It assumes people acted in good faith on what they knew at the time. Action items still have owners, so accountability stays, but the fixes target systems and process.
How soon after an incident should a postmortem be written?
Within days, while chat history and memories are fresh. PagerDuty's internal policy is 3 calendar days for its most severe incidents and 5 business days for the next level. Waiting weeks usually produces a thinner timeline and fewer useful action items.
What is a good incident postmortem example to copy?
A good example has a one-line summary, exact impact numbers, a timeline where every timestamp links to its source, a root cause separate from the trigger, and action items that each have a ticket and an owner. The Google SRE book and PagerDuty's guide both publish worked samples.
Should a security incident postmortem be shared with the whole company?
Usually not in full. Security reviews can reveal open weaknesses, attacker methods and personal data. Many teams share a short summary widely and keep the full review, with logs and indicators, to a smaller group. Legal or regulator duties may also limit who can see it.
What is the difference between a trigger and a root cause?
The trigger is the event that set the incident off, such as a config change or a traffic spike. The root cause is the condition that let the trigger do harm, such as servers that could not answer a new health check. Fixing only the trigger leaves the same failure waiting.
Sources and further reading
- Google, Site Reliability Engineering, chapter 15: Postmortem Culture
- Google, Site Reliability Engineering, appendix D: Example Postmortem
- PagerDuty Postmortem Guide: What is a postmortem?
- PagerDuty Incident Response Guide: Postmortem Template
- Atlassian: How to run a blameless postmortem
- GitHub: danluu/post-mortems, a collection of postmortems
- NIST SP 800-61 Rev. 3: Incident Response Recommendations and Considerations for Cybersecurity Risk Management
