crtdrops
NOTE · ENGLISH ·

An open socket is not a connected device

How long the server takes to find out that a connection died, at the median: 12.8 minutes with three missing heartbeats, 2.2 hours with Linux's TCP keepalive, 10.3 minutes with whichever comes first; and 24 of 988 commands sent that never arrived.

An operator sends a command to immobilize a vehicle. The server looks up the device’s connection, writes the command to the socket, and the write succeeds. That does not show the device received it, much less that it carried it out.

In vehicle tracking, the TCP connection is only a means of communication. It is not a guarantee of availability. That is what this note is about.

The figures in the examples are illustrative: they come from a simulator with an assumed fleet, not from measurements of our gateways or of any carrier.

Connections that die without notice

We have worked with receiving gateways for GPS devices, telemetry and vehicle cameras. With JT808 and other tracking protocols, the device opens a persistent TCP connection and keeps it to send positions, alarms and other messages. How it behaves depends on the protocol and the manufacturer.

With mobile devices, that connection can stop working without the server receiving a close: the vehicle loses coverage, switches networks, or the carrier’s NAT forgets the connection for being idle. The server keeps a socket that looks open, even though the device no longer has connectivity.

Three ways to find out

There are three mechanisms to learn that a connection died, and they are not equivalent:

In the simulator, a fleet of 2,000 devices sends a position every 30 seconds while moving and a heartbeat every five minutes while parked, and each device loses coverage about four times a day. In one day, 7,762 connections die without notice. This is how long the server takes to find out, at the median:

The system’s keepalive, at its factory values, is no use for this, and on top of that it is off by default on each socket. And none of them is instant: during those minutes, the socket looks open.

I do not have a single heartbeat interval documented for our whole fleet: it depends on the protocol, the manufacturer and the configuration. Nor would I attribute a NAT expiry time to a carrier without having measured it. In the simulator they are assumptions, and you can change them.

What a heartbeat costs

A small heartbeat does not cost only its bytes. A 20-byte heartbeat, with the TCP/IP headers, its response and the ACK, is about 160. In the simulator, the fleet spends 1.22 MB a month per device just staying connected; counting only the heartbeat and its response it would look like 0.30. In a large fleet, that difference adds up.

Stretching the heartbeat to save data has a limit: the NAT. If the heartbeat is 15 minutes and the NAT forgets after 10, every heartbeat from a parked device hits a connection that no longer exists and forces a reconnection. In the simulator, the data drops by a quarter, from 1.22 to 0.91 MB, and the connections that die without notice go from 7,762 to more than 170,000.

Sent is not received

The command case is the most delicate. That is why I would distinguish four states:

  1. Command generated.
  2. Command sent to the connection.
  3. Command confirmed by the device.
  4. Action executed and verified.

Going from the second to the third takes a confirmation from the protocol; from the third to the fourth, where it applies, an independent check of the final state.

On the simulator’s day, 988 commands are sent. Of them, 24 are written to a socket that was already dead: the server “sent” them and they never arrived. If “sent” counted as done, those would be 24 false successes. With a lifecycle that waits for the confirmation, eight are resent over the new session and 16 expire unconfirmed, in plain view of whoever sent them. And if the heartbeat goes past the NAT, the commands written to dead sockets rise to 322.

Two sessions from the same device

A device loses communication, reconnects and opens a new session before the server detects that the previous one no longer works. If the system manages connections only by socket, it can end up with two sessions from the same device.

My approach is a session registry per device, with a session identifier or generation. When a new, authenticated, valid session arrives, it replaces the previous one under an explicit policy, and the old one is invalidated and closed in a controlled way. You also have to keep a late close of the old session from accidentally wiping out the reference to the new one. I do not present this as a failure we suffered, but as a condition the design has to resolve.

The simulator compares the two registries. In its day, devices reconnect 4,579 times with the old session still open. With a registry by socket, which registers the device with every message and deletes the entry when any of its sessions closes, 4,566 of those late closes wipe out the new session. The device is still connected and the system takes it as disconnected until its next message: 335 device-hours in the day. If the registry were only updated at authentication, the device would stay “disconnected” until it reconnected again: 20,801 device-hours, and 429 commands would expire without finding a session. With a registry by generation, none.

Several instances

With several gateways, each device may be connected to a different instance. Knowing its IMEI is not enough: you have to know which instance holds its connection. I would design a distributed session registry with the device’s identifier, the instance, the session’s identifier and generation, the last valid activity, and its state and validity.

The command is routed to that instance, and the instance checks that it still holds the session before transmitting it. I would not send it to every gateway. And you have to account for the device moving to another instance while the command is in transit. The registry finds the connection, but it does not replace the protocol’s confirmation or guarantee that the command was executed.

The restart that becomes a second failure

When a gateway restarts, the sessions in that process are cut. What happens next depends on the devices: some reconnect at fixed intervals, others with other strategies, and it cannot always be controlled from the platform. I have no documented incident of a mass reconnection, or a figure for devices that reconnected at once. What I do consider essential is that recovery must not turn into a second failure.

And three things have to be told apart: an accepted TCP connection, an authenticated device and a session ready to receive commands. In the simulator, the gateway is down for 15 seconds; it accepts 1,000 connections a second with a queue of 511 (the one Node.js uses unless told otherwise), authenticates 100 devices a second and initializes 200 sessions. With 2,000 devices retrying together every 30 seconds, 2,000 attempts arrive in one second. The queue fills up: Linux drops 3,323 SYNs, which the devices resend; 900 attempts are abandoned after waiting too long, and every session is ready for commands after 79 seconds. With a random wait of up to a minute, the peak is 74 attempts a second, nothing is dropped and all are ready after 60 seconds, a time that cannot go below that wait: the last device does not try before it. With 5,000 devices the gap grows: 195 seconds against 64, and with a fixed interval 27,363 SYNs are dropped.

What matters is not how many attempts arrive per second, but how long the system takes to recover every session. In a new design I would use staggered restarts where the architecture allows it, load distribution, and random waits on reconnection where the firmware supports them.

The limits you do not see

I have no documented incident from running out of file descriptors, memory or ports in our tracking gateways. But they are part of what has to be sized: the available descriptors, memory per connection, event loop usage, TCP accept queues and socket states. I would not assume a server can handle thousands of connections just because it has memory and CPU to spare.

What I would do today

In our current architectures we mainly use Node.js and NestJS for TCP reception, with specialized services per protocol, and RabbitMQ to decouple reception from processing. If I designed a gateway for thousands of devices from scratch today:

I would separate four responsibilities. Transport: sockets, connections, disconnections, timeouts and bytes. Protocol: identifying devices, parsing frames, validating messages and handling confirmations. Sessions: the device’s identity, its current connection and the instance holding it. Messaging and processing: handing telemetry to the rest of the platform without blocking reception.

I would give commands a lifecycle of their own, with unique identifiers, execution states, expirations and confirmations.

I would measure from the start how many connections there are, how many are really active, how many reconnections happen per minute, how long it takes to detect a dead session and how many commands are left unconfirmed.

I would run load tests that go beyond sending positions: thousands of persistent connections, silent disconnections, duplicate sessions and gateway restarts.

To try it with another fleet, another heartbeat, another NAT or another gateway, there is the connection simulator: it runs in the browser, and everything it says is a model, not a measurement.

An open socket does not show that the device is effectively connected. A command sent does not show that it arrived. And an acknowledgment of receipt does not necessarily show that the action was carried out. When we manage many devices, reliability depends less on how many connections we can open and more on our ability to know the real state of each session and to recover correctly when communications fail.

The real challenge is not keeping thousands of sockets open, but managing thousands of devices whose connectivity can change at any moment.

Israel Negrete Lepe is an electronics engineer. Twenty-five years making different systems work together under real conditions: from the sensor, the vehicle or the transaction to the backend, the data, security and operations. Today, increasingly, edge AI.