What running 300 properties taught us
Honest notes from operating a hospitality control fleet: the patterns that paid off, the shortcuts that cost more than they saved, and why we stopped polling.
We keep an internal retrospective on the fleet. It is not a celebration document and not a complaint document — it exists so that when somebody proposes a design that resembles something we already tried, we can check whether that thing actually worked.
Some of it is worth publishing. If you are specifying a building system, these are the things we would want you to ask us about.
What paid off
One database per property. Each hotel’s data lives in its own database and the central layer aggregates. This is more work up front than one shared pool with a tenant column, and it is the best decision in the project. Isolation is structural rather than a filter somebody has to remember to apply. A busy property cannot slow down a quiet one. And when a customer asks whether their data can be seen by another operator, the answer is a fact about the architecture rather than a promise about our code.
A durable queue at the property. Events are written to a local outbox before they are sent anywhere, and drained by an agent. It survives WAN flaps, restarts and reboots. It is the single most resilient thing we run.
Push, not polling. Nearly every performance problem we have had traced back to something asking repeatedly instead of being told. We replaced polling with live connections across the platform and the entire class of problem went away. If you are evaluating a system and its dashboard has a refresh button, ask what it is doing to the server at scale.
Staged rollouts. Firmware and agent versions are assigned per property, so a release reaches a pilot group before it reaches the fleet. This has contained a bad build to a handful of sites more than once. Without it, that same release would have gone everywhere.
Writing everything down. Every meaningful action records who did it, to what, with what payload, and when. Reconstructing an incident three days later is only possible if the record includes what was actually sent. Every time we trimmed an audit payload to save space, we regretted it.
What cost us
Polling dashboards. Covered above, but worth restating as a mistake rather than a preference. We built admin views that polled endpoints taking hundreds of milliseconds, at intervals measured in seconds, across a growing fleet. The arithmetic catches up with you quickly.
Conflating things that look similar. We aggregated common-area fixtures together with guest rooms because both are “things with power state”. They are not the same: occupancy means nothing for a corridor, and a report that mixes them is wrong in a way nobody notices until a customer queries a number.
Trusting a timestamp from the far end. Staleness detection based on a time stamped by the remote agent fails when that agent’s clock is wrong — which happens. Measure elapsed time at the end that is asking the question.
Mixing development routes into the production codebase. Convenient, and it caused two deployments that should never have happened. Environment gating is not optional.
The one we would defend
We designed for the fleet we had, not the fleet we imagined.
The architecture was sized for hundreds of properties, with the code factored so the parts that would need to split under heavier load could split when load demanded it. Purists would say we should have designed for ten times the scale from day one.
If we had, we would still be designing it. Getting to real customer load is what surfaced every weakness in this list — none of them were visible on a whiteboard. The mistakes above are not regrets, they are the price of finding out, and the next hundred properties are cheaper to run because we paid it.
What this means if you are buying
Ask vendors what has gone wrong. Not whether — what.
Any system running at real scale for real years has had incidents. A vendor who cannot name one either has not been running long, is not paying attention, or is managing you. All three are worth knowing before you sign.
We will walk you through ours, including the ones that reached customers, and what changed afterwards. That conversation tells you more about how a system will behave in your building than any feature list.