CAN Bus-Off Recovery in Robots: Restart Without Replaying Motion

In short: Separate SocketCAN recovery from motion permission with a recovery evidence matrix, Linux diagnostics and a tested synthetic command-admission example.

The CAN interface comes back. Telemetry starts moving again. A queued velocity command reaches a drive, and the robot resumes an action whose context disappeared during the outage. The interface recovery succeeded; the recovery procedure did not.

After CAN bus-off, restore communication with motion authority held closed. Invalidate commands from before the fault, establish fresh evidence from every required drive, reconcile their operating state, and require a new authorization to resume. A Linux interface reporting ERROR-ACTIVE is evidence about one CAN controller. It is not evidence that a wheel drive is ready, that a previous setpoint has been cleared, or that the robot should move.

First establish which controller stopped communicating

Bus-off belongs to a controller, not to the robot as a whole. The Linux gateway can enter bus-off while another node remains operational. A drive can lose its own CAN connection while the gateway still exchanges traffic with a second drive. Name the affected interface and required devices in the incident record; “the bus recovered” is too ambiguous to authorize anything.

The Linux SocketCAN documentation describes the controller states, device statistics and restart controls. A nonzero restart-ms enables automatic recovery after bus-off; zero disables that automatic restart. Error notifications also require an error-frame subscription on a raw socket. Application silence alone does not identify the fault.

On an already configured Linux test system, these commands collect interface statistics and CAN traffic:

ip -details -statistics link show can0
candump -t a -e -d 'can0,0:0,#FFFFFFFF'

The explicit filter captures data and error frames; -e decodes error frames and -d reports detected receive drops. These options are documented in the upstream can-utils candump source. Check the installed version and driver capabilities. These are documentation-checked commands, not commands executed against hardware for this article. A quiet capture without drops is not proof that every remote device is healthy.

Keep the before/after counters, timestamps, bitrate configuration and device identities. Inspect whether faults correlate with a connector movement, a drive reset or a power event. The existing bus-selection guide addresses topology and interface choice; the robot power-integrity procedure helps distinguish supply disturbances from software symptoms. Neither a higher retry count nor a repeated restart repairs a physical fault.

Close the command path before restarting the interface

Define recovery as two separate operations: recover transport, then qualify motion. The distinction has to exist below the ROS 2 planner or AI supervisor, in the bridge that actually owns actuator writes.

At fault detection, latch motion admission closed and advance a recovery generation, or epoch. Cancel pending application commands and invalidate the old motion permit. A new bridge process must also begin disarmed; an in-memory counter reset must not make an old command valid again. Use a boot/session identity as part of the production epoch.

That host-side fence cannot recall a frame already handed to a kernel queue, USB adapter, CAN controller or drive. Before allowing transport to resume, establish how each queue and the drive’s retained setpoint are cleared or rendered ineffective. Keep the independent motion inhibit effective through this step. Do not assume that closing a socket flushes every downstream queue. If queue behavior is undocumented, qualify it on the actual adapter and firmware before relying on automatic recovery.

This ordering matters with restart-ms: the controller may recover before the supervisory process handles its fault notification. Automatic transport recovery is appropriate only when the actuator-side behavior remains acceptable during that race. Otherwise use supervised recovery, with motion inhibition and queue handling already established. For a controller currently in bus-off, SocketCAN exposes a manual restart:

# Only after the recovery procedure has inhibited motion and handled queues.
sudo ip link set can0 type can restart

The command requests controller recovery. It does not clear application faults, authorize a drive or erase a trajectory. Configure the chosen automatic/manual policy during commissioning and verify its actual behavior rather than changing it opportunistically during a fault.

The physical response to communication loss also belongs in commissioning. A wheeled test robot, a gravity-loaded joint and a suspended load may need different braking or holding behavior. A software request to stop over the failed link cannot be the only mechanism that establishes that response.

Collect evidence at the drive, not just at the gateway

Device recovery semantics differ. For a concrete example, the current ODrive CAN protocol documentation says bus-off sends an autobaud-enabled device back to scanning; with autobaud disabled, it restarts its CAN interface and repeats if the fault persists. The same documentation specifies which setpoint messages feed its watchdog. Repeatedly sending an old target can therefore keep a watchdog satisfied. Check these behaviors against the deployed firmware; they are not generic CAN guarantees.

Use the following as a recovery review sheet. The final column describes evidence the integration team must obtain, not a capability supplied automatically by CAN.

Boundary	Misleading success signal	Evidence required before motion admission
Host controller	Interface is up again	Expected controller state, recovery event recorded, persistent faults investigated
Outbound path	Application queue is empty	Old frames cannot become effective through driver, adapter or controller queues
Each required drive	Some traffic appears	Correct device identity, fresh post-recovery status and known boot/configuration state
Drive application	Heartbeat arrives	Faults reconciled; mode, retained target and feedback valid for the intended operation
Motion authority	Old planner action still exists	New permit tied to this recovery epoch; old action canceled
New command	Timestamp looks recent	Created after rearm, unexpired and associated with the current epoch

An arrival timestamp is not necessarily a measurement timestamp. Buffered pre-fault status can arrive after restart. Establish freshness through protocol sequence or boot identifiers, a correlated query response where supported, or a qualified queue-drain and observation procedure. Merely attaching the new epoch to every received CAN frame invents provenance. The distinction is the same one discussed in ROS 2 clock domains and sensor timestamps.

A CAN identifier alone does not establish trusted device identity. Reconcile the installed node mapping and any available serial/boot information; handle duplicate IDs and unexpected replacements explicitly. If the protocol cannot expose enough evidence to distinguish a reboot or stale response, keep that limitation in the recovery procedure and require an appropriate commissioning or operator step.

A small admission test that exposes stale evidence

This Python 3.9+ example models a two-drive bridge after a separate recovery procedure has established queue isolation and a valid rearm permit. It deliberately performs no CAN I/O. The observations are synthetic fixtures representing already validated device evidence, not raw frames with newly attached labels.

Times use one monotonic clock in milliseconds. The 100 ms status age limit and command deadlines are illustrative inputs, not recommended robot settings. Each required drive needs its own justified deadline in a real implementation.

from dataclasses import dataclass, replace

@dataclass(frozen=True)
class Observation:
    epoch: int
    measured_ms: int
    ready: bool

@dataclass(frozen=True)
class Command:
    epoch: int
    issued_ms: int
    expires_ms: int

def admit(command, observations, *, epoch, recovered_ms,
          armed_ms, now_ms, link_active, queues_isolated):
    if not link_active or not queues_isolated or armed_ms is None:
        return False
    if not (recovered_ms <= armed_ms <= command.issued_ms
            <= now_ms < command.expires_ms):
        return False
    if command.epoch != epoch:
        return False
    for node in ("left", "right"):
        obs = observations.get(node)
        if obs is None or obs.epoch != epoch or not obs.ready:
            return False
        if not (recovered_ms <= obs.measured_ms <= now_ms
                and now_ms - obs.measured_ms <= 100):
            return False
    return True

facts = dict(epoch=7, recovered_ms=1000, armed_ms=1040,
             now_ms=1080, link_active=True, queues_isolated=True)
good = {n: Observation(7, 1050, True) for n in ("left", "right")}
cmd = Command(7, 1060, 1200)
assert admit(cmd, good, **facts)
assert not admit(replace(cmd, epoch=6), good, **facts)
assert not admit(replace(cmd, issued_ms=1030), good, **facts)
assert not admit(replace(cmd, expires_ms=1080), good, **facts)
assert not admit(cmd, {"left": good["left"]}, **facts)
assert not admit(cmd, good, **(facts | {"armed_ms": None}))
assert not admit(cmd, good, **(facts | {"queues_isolated": False}))
assert not admit(cmd, good, **(facts | {"link_active": False}))
assert not admit(cmd, good, **(facts | {"now_ms": 1160}))
for bad in (Observation(6, 1050, True), Observation(7, 990, True),
            Observation(7, 1090, True), Observation(7, 1050, False)):
    assert not admit(cmd, good | {"right": bad}, **facts)

The first assertion admits a new command with current evidence from both drives. The failures cover a previous epoch, a pre-rearm command, expiry, a missing drive, no permit, unhandled queues, an inactive link, ageing feedback and four invalid observations. The assertions were executed as a synthetic logic check. They do not establish a hardware recovery time or prove that a queue was flushed.

Notice the test at 1160 ms: the command is still unexpired, but the last drive observations are too old. A successful rearm must not latch “healthy forever.” Evaluate current evidence at dispatch and continue enforcing freshness during operation. Repeated faults revoke the permit and start another recovery epoch. Serialize fault processing, permit changes and admission in the bridge; this pure function does not solve races between a check and a physical transmission.

The command-validation boundary still checks command meaning, limits and authority. Passing this recovery predicate only supplies one prerequisite to that broader decision.

Qualify the sequence on a constrained bench

Choose a fixture that limits physical motion and a device-specific loss-of-communication response before injecting faults. Record the actual controller, adapter, driver, firmware, bitrate, restart policy and watchdog configuration. Repeat the following scenarios with an independent record of actuator behavior:

1. Lose the host link while a nonzero command is pending. Restore it and verify that the old command cannot take effect.
2. Recover the host while one required drive remains unavailable. Admission must stay closed even while unrelated telemetry continues.
3. Restart a drive during the outage. Verify that old mode, calibration and position assumptions are invalidated where that device loses them.
4. Delay pre-fault status until after recovery. Verify that receipt time alone cannot qualify it as fresh evidence.
5. Rearm, then stop feedback while the command remains unexpired. Confirm the defined degraded response before the permitted age is exceeded.
6. Repeat bus-off during qualification and restart the bridge process. Neither path may retain or resurrect the earlier permit.

For each test, distinguish fault detection, transport recovery, evidence qualification and permission to move. Measure the interval to the actual physical response separately. A successful Linux restart is only one event in that timeline.

The useful recovery record ends with a new command and the evidence that allowed it. If it ends with can0 is up, the integration work is still unfinished.