At 10:39 UTC on March 13th 2024 the SalesScreen Engineering team deployed an update to the network configuration of the application. While the update itself was successful, the new configuration resulted in some unintended side effects for an existing network rule used for the tv.salesscreen.com domain. This resulted in users being redirected to the SalesScreen login page when trying to reload or connect screens using the tv.salesscreen.com domain.

The issue was immediately noticed by members of the Engineering team and a fix for the issue was quickly developed. Network configuration updates are by nature slow to propagate, so a full resolution of the issue was delayed until 11:18 UTC. Once the fix was fully deployed, users could once again connect to TV screens using the tv.salesscreen.com domain.

SalesScreen has robust procedures for the deployment of changes to our production environment, which was followed in this case. The network change had been tested successfully in our development environment, however this testing was limited to the added network rule. As a result we did not pick up on the issue with the TV endpoint during this testing.

Going forward we will update our internal routines for making network configuration updates to include testing of all existing network rules.