During an incident the two most valuable things are usually in tension: getting systems back, and understanding what happened. Move too fast and you destroy the evidence that would tell you how they got in and whether they are still there. Move too slowly and the business bleeds.
Incident response is the discipline of doing both properly — containing the problem while preserving the record.
What We Do
- Containment guidance — limit the damage without destroying the evidence trail.
- Evidence preservation — captures, images and logs collected defensibly, in case this becomes a legal or insurance matter.
- Traffic and log analysis — reconstruct sessions, identify command-and-control patterns, trace data movement. Tooling described on our forensics page.
- Root-cause reconstruction — how entry was gained, what was reached, what was taken.
- Scope determination — the question everyone asks first and that takes the longest to answer honestly.
- Reporting — a written account that stands up to auditors, insurers and regulators.
When It Crosses Into OT
An incident that reaches building or industrial systems needs someone fluent in both worlds. Corporate IR tooling and instincts do not transfer cleanly to a network running BACnet and a Niagara station, and treating OT like IT during an incident is how a security event becomes an operational one.
Prevention Is Cheaper, Obviously
If you are reading this before an incident, the useful work is testing, secure development, and proving your recovery actually works — which is exactly what our recovery validation platform is being built to do.
Get in touch — whether you are in an incident now or want a responder who already knows your environment before you need one.
Related services
- OT and Network Forensics — Related: OT and network forensics.
What the First Hour Costs You
Before any outside responder arrives, someone inside has already made decisions that shape the whole investigation. Reimaging the affected machine and getting the user working again is the understandable instinct, and it destroys the host side of the evidence: running processes, open connections, loaded modules, and whatever the intruder left in memory. Where you can afford to isolate a host at the network level and leave it powered on, do that first.
Memory holds things that never reach disk. Some of it survives a shutdown in a hibernation file, a pagefile, a crash dump or a suspended virtual machine, and those are worth collecting, but none of them substitutes for a capture taken while the system is still running. Two other habits help more than they look like they will. Start a plain written timeline the moment you notice something, with timestamps and who did what, because reconstructing your own actions later is surprisingly hard. And decide early who is authorized to speak about the incident, internally and externally, so a partial picture does not reach a customer or a regulator before anyone understands it.
Who Else Belongs in the Room
Technical containment is one track. Legal counsel, your cyber insurer and, depending on sector and jurisdiction, one or more notification clocks run in parallel. Your policy may set a notification deadline and may restrict which responders can be engaged, sometimes to a named panel, so read it before anyone is appointed, including us. Some organizations run investigations under legal privilege, which affects how the work is commissioned and how reports are worded. That is a decision for your counsel, and it is far easier to make at the start than to retrofit halfway through. We will work under whatever structure counsel sets, and we will tell you early if the policy rules us out.
Authorization Before Anyone Touches a Disk
Response work is intrusive by definition: isolating hosts, capturing memory and disk images, pulling logs from systems that may belong to a provider rather than to you. It runs under written authorization from a party entitled to grant it, against a scope naming the systems and accounts in play. Where a cloud platform, a hosting provider or a managed service provider owns part of the environment, their authorization is needed too, and obtaining it takes time nobody has once an incident is live. Evidence handling belongs in the same document: how images are stored, who holds them, how long they are kept, and how they are destroyed. This is the authorization chain set out on our service delivery and coverage page, agreed in advance rather than negotiated at two in the morning.
What Forensics Can and Cannot Recover
How much of the story comes back depends almost entirely on what was recorded before the incident. Endpoint telemetry, authentication logs, DNS and proxy records, firewall flows and cloud audit trails each answer different questions, and each has a retention period that may already have expired. Where an intruder cleared logs or operated only in memory, some questions stay open permanently. A report worth paying for separates conclusions supported by evidence from inference, and states plainly what could not be established. Confirming that the right things are being logged, and kept long enough to be useful, is much easier before you need them, and that work sits inside platform security and compliance readiness rather than inside a response.
Not Every Incident Needs an Investigation
A single phished mailbox where multifactor held, the session was revoked, and audit logs show no rule creation or bulk downloads may be well within reach of your own team. What changes the answer is obligation and uncertainty: a contract or regulation requiring you to establish scope, any suspicion of lateral movement, any sign of persistence, or a system holding regulated data. If you cannot say confidently what the intruder could reach, that uncertainty is itself the argument for bringing someone in.
What a Retainer Buys, and What It Does Not
A retainer buys familiarity. Someone has already seen the network diagram, knows where the logs live, and has an agreed contact path, so the early hours go on analysis instead of orientation. The trade is that you are paying for readiness you may never use. What it does not buy is unlimited availability: response windows, contact paths and escalation exist only where they have been written into engagement terms, and without terms in place we cannot promise you a time. If you already have an internal team with forensic capability and a plan that has been exercised, a retainer may add little. If the plan has never been rehearsed and the person who knows the environment best is also the person who would be restoring it, the case is stronger.
How Evidence Is Actually Acquired
Acquisition is judged later on whether the copy can be shown to be identical to the original. That means a cryptographic hash computed at the moment of collection, usually SHA-256, recomputed after every copy, with both values written down rather than merely glanced at. Disk acquisition normally goes through a hardware or software write blocker, because a source device attached to a running workstation will be modified by the act of mounting it within seconds. Container formats such as E01 embed checksums across the image and carry the acquisition metadata with the data; a raw dd image carries neither, so the hash and the collection notes have to travel alongside it and stay attached to it.
Chain of custody is a per-item record rather than a policy: which host or device the item came from, its identifiers, who collected it, when in UTC, and every transfer of possession afterwards, with the hash reverified at each handoff. What gets challenged later is rarely the tooling and usually the gaps, a stretch where nobody can say who held the drive, or an image copied to a laptop that never appears in the log. Keeping the record as you go is the whole trick, since a custody log written up afterwards is worth much less than the images it describes.
Acquisition is a sequence before it is a toolset. Evidence has an order of volatility, set out in RFC 3227, and the rule is to take what disappears first while it still exists, even when the slower sources look more interesting. In practice that ordering runs roughly as follows:
- Live state, most of which a reboot or even a console login rewrites: RAM, the ARP and DNS resolver caches, the routing table, and the keys to any mounted encrypted volume while they are still resident.
- Fast-moving disk artifacts: the file system journals, temporary directories, and anything subject to routine rotation or cleanup while the host stays powered on.
- The full disk, acquired at the physical device level rather than by copying out of the mounted volume, so unallocated space, file slack and the journals come with it.
- Records held off the affected host, each export stamped with the retention window that was in force at the time of collection, so that a later absence can be told apart from a deletion.
- Archived and backup copies last, since they change most slowly, though a backup retention schedule can still age out the pre-incident baseline while an investigation is under way.
Every Timestamp Comes From Somewhere
A timeline is assembled from clocks that were never synchronized to each other and from formats that disagree about what a time even is. Windows event logs store time as UTC and render it in the local zone of whoever is looking, so two analysts can produce screenshots that differ by hours from the same record. BSD-style syslog under RFC 3164 carries neither a year nor a time zone in the line itself; RFC 5424 fixed both, and plenty of devices still emit the older format anyway. NetFlow v5 reports flow start and end relative to the exporter's uptime, which makes those records meaningless as absolute times until the exporter's boot time is known. Normalize once, to UTC, and keep the original string beside the converted one.
Two further things break timelines quietly. A device whose time synchronization has been broken for months can be minutes out, which is more than enough to invert cause and effect between two sources and produce a confident wrong answer, so the observed offset for each source is worth recording at collection and correcting in the timeline rather than in the evidence. The second is that an address identifies a host only at a particular moment: a firewall line means nothing without the DHCP lease active at that timestamp, and behind NAT it means nothing without the translation table entry for that second. Where those records are not kept, a complete network picture cannot be attached to a machine.
File times on NTFS have a useful property. Every file carries two sets of timestamps, one in $STANDARD_INFORMATION and one in $FILE_NAME, and the common timestomping tools alter the first and not the second, so a mismatch between them is itself a finding. The change journal and $LogFile record file system activity independently of both, which is one of the concrete reasons acquisition takes the volume rather than copying interesting files out of it.
Cloud Incidents Leave a Different Record
In a cloud tenant there is frequently no host to image, and the record is whatever the provider's audit logging was configured to keep. The defaults are thinner than most people assume. AWS CloudTrail shows management events in Event history for 90 days and keeps them longer only where a trail was configured to deliver them to storage, while data events such as S3 object reads and Lambda invocations are not recorded at all unless they were explicitly enabled beforehand. Microsoft Purview Audit (Standard) holds most records for 180 days, with the detailed mailbox access events including MailItemsAccessed sitting in the higher audit tier, and Entra ID keeps sign-in and audit logs for 30 days on paid tiers and 7 on the free one unless they are exported somewhere that keeps them longer.
Identity is where cloud incidents live, and password resets do less than people expect. A stolen refresh token or session cookie keeps working until it is explicitly revoked, and an access token already issued stays valid until it expires on its own. Consent granted to an OAuth application, and any service principal holding application permissions, is independent of the user's password entirely, so an app registration added during the incident survives every credential action taken afterwards. Enumerating and removing those grants is a separate task from resetting accounts, and an easy one to leave undone.
The gap that matters most is between the control plane and the data plane. Control plane logging shows that a role was assumed, a policy attached, a key created; whether anything was actually read through that access appears only in data event logging, which is off by default and billed separately, which is why what was accessed or downloaded is so often unanswerable after the fact. Acquisition has its own cloud specifics: take a volume snapshot before anything else, remove the instance from its scaling group or disable termination so it is not replaced automatically, and accept that instance store volumes are gone the moment the instance stops.
OT Collection Is Passive by Default
Collection on the operational side runs from a mirror port or a passive network TAP, with no agent deployed to a controller or to an engineering workstation while the process is running. A TAP is preferable to a SPAN port because a SPAN session silently drops frames when the switch is loaded and gives no indication that it did, and for evidence, unannounced loss is worse than no capture. Many controllers also have no spare capacity for anything additional, and vendor support terms often restrict what may be installed on them at all.
Capture is frequently the only place a control action is recorded, because the field protocols were not designed to leave an audit trail. Modbus/TCP on port 502 has no authentication and no integrity checking in the base specification, so a write from an accepted address is indistinguishable at the protocol level from a legitimate one; an optional TLS variant on port 802 exists and is rarely deployed. BACnet/IP on UDP 47808 is in the same position, with BACnet/SC defined later as a secure transport and not widely fielded. DNP3 has Secure Authentication defined in IEEE 1815, which is likewise usually left disabled.
Two OT sources carry more weight than they are usually given. The historian and the alarm and event journals hold timestamped process values, which can bound when a setpoint actually changed even when nothing recorded who changed it. The engineering workstation holds project files, often with revision history, and comparing the logic running in a controller against the last known good project file is how an unauthorized change is demonstrated rather than suspected. That workstation is normally an ordinary Windows machine, so standard host forensics applies to it even where it does not apply one hop further down.
Eradication Is a Scheduled Event, Not a Gradual One
Removal happens in one planned window, against a list built from the investigation, and not as findings arrive. Evicting an intruder piece by piece tells them they have been noticed while they still hold access nobody has found yet, and the usual outcome is a quieter return through a path that never made it into the report. It also means the investigation has to reach a defensible position on scope before the window is scheduled rather than during it.
Credential material is the part that outlives the cleanup. A password reset does not invalidate a Kerberos ticket already issued, which by default remains usable for about ten hours and renewable for days, and it does nothing to certificates issued to that account, which stay valid until they are revoked. Where domain compromise is suspected, the krbtgt account is reset twice, with enough time between the resets for replication to complete and existing tickets to age out. Service accounts, API keys, SSH keys and secrets held in build and deployment systems belong on the same list, and they are the ones a normal password process never reaches.
Restoration has a date attached to it. A backup is clean only if it predates the earliest confirmed access, and that date can move backwards as the investigation continues, which makes an early restore a decision that may have to be taken twice. Where the earliest access cannot be established at all, rebuilding is safer than restoring, and it changes the scheduling considerably.
Persistence is the list that tends to go unowned. It is not one mechanism but a spread of them across hosts, identity and anything reachable from outside, and each item has to be checked rather than assumed:
- Scheduled tasks and services, particularly ones named to resemble platform components while running from a path no vendor binary uses.
- WMI event subscriptions, an event filter bound to a consumer, which live in the CIM repository rather than on the file system and are missed by any check that looks only at disk and registry.
- Registry run keys, startup folders, and DLL search order abuse or side loading against an otherwise signed executable.
- On Linux and Unix hosts, cron entries and systemd timers, unit files with an added ExecStartPre, entries in /etc/ld.so.preload, and public keys appended to authorized_keys.
- Anything internet facing: a web shell dropped into a document root, an IIS module or handler registered to run on every request, and stored credentials or deploy keys in a build system that can push to production.
Frequently Asked Questions
Is incident response the same thing as monitoring or detection?
No. Detection is a continuous function that decides something has happened, while response as a discipline begins from a finding it did not generate itself, whether that came from an alert, a user, a customer, or an outside notification. The two are connected in one direction that matters: what was being recorded before the incident sets the ceiling on what the response can establish afterwards. An environment with good detection and no response plan and an environment with a plan and no telemetry both fail, in different places.
What do you need from us before analysis can start?
Read access to the consoles and systems holding the relevant records, provisioned as separate accounts rather than shared with existing staff logins, so that responder activity is separable from everyone else's inside the same logs being examined. An out-of-band contact list and a channel that does not run on the environment under investigation, since mail and chat inside a compromised tenant are readable by whoever is in it. Read-only is normally sufficient for analysis, and any permission that can change state should be granted separately and deliberately.
Does the investigation require taking systems offline?
Collection usually does not. Memory is captured from a running host and disks can be imaged from a snapshot or acquired live, so the analysis work itself rarely needs an outage. The downtime that does occur belongs to the eradication window, which is planned in advance and sized around credential rotation and rebuilds rather than around evidence gathering. Virtualized hosts are easier again, since a snapshot taken with memory included captures both halves in a single action.
How does an engagement proceed once authorization is in place?
Access provisioning first, then collection from a small initial set of systems chosen because there is a reason to think they are involved, then triage of that set until something concrete appears: a file, an account, an address, a parent process. Each confirmed indicator is swept across the rest of the estate, which usually surfaces more systems, which yields more indicators. That loop is why the affected list grows before it settles, and it is expected rather than a sign the first pass was wrong. Eradication is scheduled once the loop stops producing new systems, and the written account is assembled from material gathered along the way rather than composed at the end.
Can you work from logs alone if the affected machines have already been rebuilt?
Partly, and it depends on what was kept off the host. Records held elsewhere survive the rebuild and can establish where an account was used and which systems communicated with each other. What they cannot do is confirm what executed on the machine, recover a deleted tool or file, or separate an intruder's use of a legitimate program from the owner's. Where the host itself is gone, the remaining options are backups, any image taken by anyone before the rebuild, and whatever endpoint telemetry was centralized, and the achievable answer is bounded by which of those exist.
Who decides whether an OT process can be touched during a response?
The person accountable for the process does, and that decision does not transfer to the responders. Whether a segment can be interrupted, whether a controller can be taken out of service, and whether any active probing is acceptable are operational questions with safety consequences that only the operator can weigh. What the technical side contributes is what each option costs in evidence: passive capture costs nothing operationally but sees only what crosses the monitored link, while anything more assertive alters the system being investigated. Expect the operator, and often the equipment vendor, to be part of those calls.
Who decides when it is safe to go back online?
You do, and the technical input to that decision is a list of the access paths that were closed and verified rather than a general assurance. In practice that means the eradication actions were completed inside the planned window and confirmed afterwards, rotation covered service and machine identities as well as user accounts, and detection is in place for the specific indicators found so that a return by the same route is visible quickly. A defined watch period after restoration belongs in the decision rather than after it, because reuse of a surviving access path is the ordinary way an incomplete eradication announces itself.