Two - Notice history

Our cloud provider has implemented a fix and are currently monitoring the result.

The third party has implemented a fix. Shopify ordering is enabled again. We are monitoring to ensure comprehensive resolution.

A 3rd party data provider failed around 04:36 UTC. This prevented our API from retrieving address information for companies in Norway and Sweden. Our system continued normal operation in all other respects, including basic company search capabilities in Norway and Sweden. Company address services returned to normal operation on recovery of the 3rd party at 05:41 UTC.

Normal service has been restored.

The third party service have acknowledged the issue and are working on a fix. They expect normal service to resume within approx 10 minutes.

This incident arose due to two root causes: (1) a gap in automated testing resulting in a bug escaping to production at 10:25am UK on 28 August; and (2) a monitoring misconfiguration which masked ongoing out-of-memory restart events and which reduced visibility of the impact of the bug. We have addressed these issues by reviewing monitoring configurations to ensure all services are correctly covered, and by both improving test coverage and ensuring that test coverage is monitored on an ongoing basis. We extend our apologies to customers for the impact caused by this incident.

The remaining intermittent errors have now been resolved. This incident is now closed.

Normal service has been restored. We are however aware of continuing low-frequency intermittent errors and are working to remediate them.