Asciiroot Systems

Linux troubleshooting guide

Linux system-file truncation: evidence and recovery

A truncated file followed by a service outage does not establish a causal link. Identify the original file, command and failure evidence before changing system libraries or assigning a root cause.

Before you begin: These examples apply to Linux; path checks use GNU coreutils, storage checks use util-linux and journal checks require systemd. Current state may differ from the state before recovery. These examples are not executed or verified against your system. Obtain authorization, protect sensitive output and involve the responsible owner.

Clarify the incident

Collect the relevant scope and timeline before choosing an action:

  • What exact command, path, host and timestamp were involved, and was the reported path a regular file, directory, archive, symbolic link or hard link before the restore?
  • Which service and OS version were affected, what was the first error or exit signal, and which exact files and versions did the administrators restore?
  • Did the command or a script touch more than one path, was the file an active filesystem or disk-image backing file, and are pre-change inode, mount, audit, backup-manifest or checksum records available?
  • What was the service impact and timing, did it recover after the restore, and were application transactions and restored-file integrity verified?
  • Was the operation authorized, and should the incident, system, backup or security owners preserve additional evidence?

Possible explanations

These are alternatives to investigate, not confirmed causes:

  • Truncating one regular file changes that file, not independent source files inside a separate ordinary archive or copy. A symbolic or hard link can refer to the same affected file, while a script or glob may touch additional paths. An active file-backed filesystem or disk image could instead contain the affected dependencies; truncating its backing file can cause data loss or I/O errors.
  • If a real application dependency was truncated, a later read or dynamic-library load could fail, or access to a mapped page beyond the new end may raise SIGBUS. The delay and restore do not prove this mechanism; a separate failure remains possible.
  • A direct regular-file truncation of a directory normally fails, although a command or script can affect files beneath it. A .debug name does not establish that the object is debugging information or an application dependency.

Read-only checks

Replace example names and paths only after confirming the applicable platform, permissions and change policy. Command output can contain sensitive data.

Identify the reported object

On the authorized Linux host with GNU coreutils, replace the placeholder with the reported path. Inspect its current type, device, inode, link count, size and resolved path. The path may now be missing or changed; these results do not prove its state before restoration. Preserve the exact command and any audit evidence.

ls -ld -- /path/to/reported/file
stat -c 'type=%F device=%d inode=%i links=%h size=%s ctime=%z %n' -- /path/to/reported/file
readlink -e -- /path/to/reported/file

Correlate the failure timeline

For a Linux systemd service, replace EXAMPLE.service and both timestamps with the confirmed unit and a narrow UTC window spanning the truncation, failure and restore. Review the first application, loader, kernel and service-manager errors; current status may reflect a later restart. The journalctl -k command defaults to the current boot; if the event was on an earlier retained boot, select the incident boot with -b. Logs can contain sensitive data.

journalctl --list-boots --no-pager
systemctl show EXAMPLE.service -p ActiveState -p SubState -p Result -p ExecMainCode -p ExecMainStatus
journalctl -u EXAMPLE.service --since 'YYYY-MM-DD HH:MM:SS UTC' --until 'YYYY-MM-DD HH:MM:SS UTC' -o short-iso --utc --no-pager
journalctl -k --since 'YYYY-MM-DD HH:MM:SS UTC' --until 'YYYY-MM-DD HH:MM:SS UTC' -o short-iso --utc --no-pager

Compare known file identities

Only after identifying the exact paths, replace both placeholders. GNU stat -L follows symbolic links; the same device and inode indicate the same resolved file at inspection time, not necessarily before a restore. Compare against historical audit records, backup manifests and service errors.

stat -Lc 'device=%d inode=%i type=%F size=%s links=%h %n' -- /path/to/reported/file /path/to/restored/system/file

Check for file-backed storage

If util-linux is available and the owner approves, list current loop-device backing files and mounts and compare them with the relevant historical host and mount namespace. An active disk image may back a filesystem containing service dependencies. Current listings can be incomplete or changed after recovery; no loop-device match does not exclude other backing mechanisms.

losetup --list --output NAME,BACK-FILE
findmnt --output TARGET,SOURCE,FSTYPE,OPTIONS

Review process and restore evidence

If available, preserve the service exit signal, core-dump metadata, application and kernel logs, audit records, backup manifest, file hashes and package versions. Core dumps and logs can contain secrets. A restored file list or a deleted mapping alone does not establish which file was truncated.

Check the restored system with its owner

Compare restored paths with a trusted same-version, same-architecture baseline. Review symlinks, ownership, permissions, ACLs, security labels, extended attributes and dependencies. Package checks are not complete integrity guarantees; some RPM verification runs package scripts, so review the version, scripts and plugins before using it.

Interpret the evidence

An independent file or archive

Truncating an ordinary archive of copies does not itself truncate the original source files. Check the original command and whether other paths were changed.

A symbolic link or hard link

A symbolic link can direct an operation to another target; hard-link names refer to the same file. Compare the device and inode and corroborate the relationship at the incident time.

An active filesystem backing file

A disk image or other active backing file can contain a filesystem used by the service; truncating it may cause data loss or I/O errors. Correlate historical backing-device, mount-namespace and dependency evidence.

A service signal or loader error

A truncated dependency can be a possible explanation, but a signal or later service failure does not identify the affected file by itself. Correlate the first error, process, mapping and file evidence.

Files restored from backup

Determine the exact paths, versions and restore result. A list of restored files does not establish what was damaged, whether the service recovered or whether the underlying cause was corrected.

Plan recovery and verification

  1. Preserve evidence and coordinate: Record impact, owners and exact times; preserve logs, audit data, backup manifests and any approved snapshots before additional changes. Escalate an active outage and involve security if the operation was unauthorized or suspicious.
  2. Establish the actual scope: Determine the original command, path type, mount namespace, inode and backing-file relationships, and whether a script changed multiple paths. An ordinary archive of copies does not by itself modify the original files; an active filesystem backing file is different.
  3. Test competing explanations: Correlate the first service error and affected file with the event timeline, and compare alternative dependency, resource, storage and configuration failures. Treat SIGBUS, loader failure and the two-minute timing as hypotheses until supported by evidence.
  4. Plan any further recovery: If recovery remains necessary, use an owner-approved, distribution- and version-compatible procedure with an intact recovery point and rollback plan. Do not blindly overwrite system trees, copy libraries from another host or repeat the restore.
  5. Document and prevent recurrence: Draft a postmortem with reported impact, timeline, verified facts, unresolved cause, evidence for and against hypotheses, restore and verification results, security review, owners and prevention actions. Review privileged access, change controls and restore testing.

Watch for recurrence: After an approved recovery, verify the original user transaction and monitor for at least the reported delay and the longer period agreed by the owner. If symptoms return, preserve the new timeline and compare file, service and dependency evidence.

Documentation pointers

Check the documentation for your product and installed version. These links are references, not live verification.

Explore a safe demonstration

The scenario lab illustrates a fixed example. It does not reproduce this incident or verify a production fix, and it never executes your commands.

Open the related scenario lab

Related Linux guides