You can have full test coverage, code review and monitoring. None of it covers the scenario where your process is not running.
It happened to us on 23 July 2026. A terminal sat off for thirteen hours with open positions. Nothing was watching them. The daily-loss cutout only fired on restart, with the damage already done.
Why tests do not see it
A test executes your code. That is the precondition of any test.
Scenarios where your code does not execute are invisible to the test framework by construction. It is not that you forgot to write the test — it is that the test cannot be written inside the system you are testing.
The list of those scenarios is short and always the same:
- The process is down.
- The machine is off.
- The network is not responding.
- The process is alive but wedged and making no progress.
- The process runs but its actions are rejected by the external system.
Each of them breaks the assumption that "if X happens, my code reacts".
What changes once you accept it
Once you accept that your process may not be there, protections split into two classes that previously looked identical:
Protections that require you to be alive. All conditional logic: thresholds, counters, decisions based on accumulated state. These are the smart, tunable ones. They are worth zero when you are not running.
Protections that persist outside you. In our case, a stop order resting on the broker's server. It is dumb — it is just a price — but it exists with the machine off.
The design mistake was not having no protections. We had three. The mistake was that all three were of the first class.
The pattern, outside trading
This is not specific to financial systems. It is the same hole in any long-running process:
- A spend cap evaluated by your service does not cover you when your service is stuck in a loop. The cap has to live at the provider.
- A cleanup job scheduled inside your application does not run if the application does not start. It needs an external trigger.
- A timeout in your client does not cover you when the client dies. It has to exist on the server too.
- A retry in the producer does not cover you when the producer is what went down. You need durability in the queue.
The useful question, in any design review:
For each protection mechanism: where does it live? Does it still exist when the component containing it disappears?
If the answer is no, it is not a protection. It is a happy-path convenience.
What we did
Two things, neither sophisticated:
- Mirror the stop broker-side, so it exists independently of our process. It is placed on open and repositioned as positions are added.
- Document the gap that remains. A private server keeping the terminal up is the only cover for "the machine is not on". Software cannot solve that: it is an operational decision for the client, and it has to be said plainly rather than buried in a manual.
That second part is the hardest thing to write on a product page. It is also what stops someone relying on a guarantee you never gave them.
From building Cerberus at Tuurt Labs. High-risk instrument: trading leveraged CFDs can cost you your entire capital. This is not investment advice.