From Lab to Line: Hardening ROS 2 Systems for 24/7 Industrial Operation

ROS 2 is the default framework for industrial robotics, and the distance between a node graph that works on a bench and one that runs three shifts a day for four years is larger than most teams budget for. The framework does not stop you shipping something fragile. This is the checklist we apply before a platform is allowed on a customer's floor.

1. DDS is a distributed system pretending to be a message bus

The single most common production failure we are called in to diagnose is not a bug in anyone's node. It is discovery. ROS 2 delegates transport to DDS, and DDS defaults are tuned for a developer's laptop on a quiet network, not for forty participants on an industrial VLAN sharing bandwidth with PLC traffic.

Every participant announces itself by multicast and maintains bilateral liveliness with every other participant. Participant count therefore drives an O(n²) growth in discovery traffic. At around thirty participants on a congested network we routinely measure discovery storms severe enough to delay user data — which surfaces as intermittent latency spikes that no application-level profiler will explain.

Two mitigations matter. Isolate domains so unrelated subsystems never discover each other, and replace multicast discovery with a static peer list wherever the network is not fully under your control. Managed switches with IGMP snooping misconfigured will silently drop multicast, and the symptom — nodes that see each other on a bench but not in the cell — is one of the most expensive to chase on-site.

<CycloneDDS>
  <Domain id="any">
    <General>
      <Interfaces>
        <NetworkInterface name="eth0" priority="default"/>
      </Interfaces>
      <AllowMulticast>false</AllowMulticast>
    </General>
    <Discovery>
      <ParticipantIndex>auto</ParticipantIndex>
      <Peers>
        <Peer address="10.20.0.11"/>
        <Peer address="10.20.0.12"/>
        <Peer address="10.20.0.13"/>
      </Peers>
      <MaxAutoParticipantIndex>20</MaxAutoParticipantIndex>
    </Discovery>
  </Domain>
</CycloneDDS>

2. QoS is a contract, and the defaults are the wrong contract

ROS 2's default profile is RELIABLE with KEEP_LAST depth 10. For a 40 Hz LiDAR that is actively harmful: reliable delivery means the middleware retransmits scans that are already stale, consuming bandwidth to deliver data the consumer will discard. Sensor streams want BEST_EFFORT with depth 1. Commands want RELIABLE. Configuration wants TRANSIENT_LOCAL so a node that starts late still receives the last value rather than blocking on a publisher that has already spoken.

Deadline is the policy most teams ignore and the one that pays for itself first. A deadline QoS turns "the LiDAR stopped" from a condition you infer minutes later from degraded behaviour into an event the middleware raises within one period. Set it, and subscribe to the event.

rclcpp::QoS sensor_qos(rclcpp::KeepLast(1));
sensor_qos.best_effort()
          .durability_volatile()
          .deadline(50ms);          // 2 x period for a 40 Hz scanner

rclcpp::SubscriptionOptions opts;
opts.event_callbacks.deadline_callback =
  [this](rclcpp::QOSDeadlineRequestedInfo & e) {
    // total_count_change > 0 means we missed a scan window
    fault_bus_->raise(Fault::SENSOR_TIMEOUT, "scan", e.total_count_change);
  };

scan_sub_ = create_subscription<sensor_msgs::msg::LaserScan>(
  "scan", sensor_qos, std::bind(&Perception::on_scan, this, _1), opts);

3. Lifecycle nodes for anything in the safety, control or perception path

An ordinary ROS 2 node begins doing work the moment its constructor returns. That is fine for a visualiser and unacceptable for a motor controller, because it removes any window in which a supervisor can verify the system is ready before it is live. Managed nodes give you the deterministic state machine — unconfigured, inactive, active, finalized — that field operations actually require.

The practical rule: allocate in on_configure, activate publishers in on_activate, and do nothing irreversible before the transition succeeds. A node that cannot reach inactive is a node the supervisor can refuse to activate, which is exactly the behaviour you want after a power event where a sensor came up in a bad state.

4. Three layers of supervision, because each catches what the others cannot

A single watchdog is a single point of failure with a reassuring name. Production platforms carry three, and the distinction between them is what each is physically capable of observing.

• Node-level: a heartbeat tied to lifecycle state. Catches a node that is alive but wedged — a callback blocked on a mutex, an executor starved by a long-running handler.
• System-level: a supervisor process monitoring heartbeats, RSS, file descriptors and CPU share. Catches slow degradation — the memory leak that would take eleven days to OOM.
• Hardware: the SoC watchdog at /dev/watchdog, petted only by the supervisor. Catches the case where the kernel itself stops scheduling, which no userspace monitor can observe by definition.

The ordering matters. Each layer can only detect faults strictly below its own trust boundary, so a hardware watchdog petted by the application it is supposed to be watching provides no protection whatsoever — a wedged application with a live timer thread will keep petting it forever.

5. Executors and the latency you did not budget for

The default SingleThreadedExecutor serialises every callback in a node. One handler that takes 80 ms blocks the control loop behind it, and the resulting jitter is invisible in unit tests where callbacks run in isolation. Multi-threaded executors help only if callback groups are declared: without them, mutually exclusive is the default and the parallelism is theoretical.

from rclpy.callback_groups import MutuallyExclusiveCallbackGroup, ReentrantCallbackGroup

self.control_cb = MutuallyExclusiveCallbackGroup()   # strictly serialised
self.percep_cb  = ReentrantCallbackGroup()           # may overlap

self.create_timer(0.01, self.control_step,
                  callback_group=self.control_cb)    # 100 Hz, must not slip
self.create_subscription(PointCloud2, 'points', self.on_cloud, 5,
                         callback_group=self.percep_cb)

executor = MultiThreadedExecutor(num_threads=4)

For the control path specifically, the remaining sources of nondeterminism are the allocator and the scheduler. Preallocate messages, lock memory with mlockall to prevent page faults mid-loop, and run the control thread under SCHED_FIFO on a PREEMPT_RT kernel with the CPU isolated from the general scheduler. Without these, worst-case latency is bounded by whatever the kernel decides to do, which is not a specification you can hand a safety assessor.

6. OTA updates for machines that cannot come back to the lab

A field robot receives updates for its whole service life, and every update is an opportunity to brick an asset that is expensive to recover. The only arrangement we ship is A/B partitioning with atomic switch and automatic rollback: write the inactive slot, verify a signature over the whole image, flip the boot flag, and require the new system to affirmatively mark itself healthy within a bounded window or the bootloader reverts on the next cycle.

• Never update a robot that is not stationary, docked and confirmed idle by the supervisor.
• Verify signatures before the switch, not after — a corrupted image that boots is worse than one that does not.
• Version the message interfaces, not just the binaries: a fleet mid-rollout is a fleet running two versions that must interoperate.
• Treat the rollback path as a first-class feature and test it on every release, because it is the one path you will need under pressure.

What this buys

None of the above is novel. It is all documented, and none of it is difficult in isolation. It is nevertheless the work that separates a demonstration from an asset, and it is almost always deferred, because a platform without it looks identical on the day it ships. The difference appears at three in the morning in month seven, and by then the cost of retrofitting it is an order of magnitude higher than the cost of doing it first.