Management Board column: KOA outage was truly terrible
September 22, 2022
Friday 16 September was a black day for Royal FloraHolland. We could only start auctioning at 10.30am and that hit us right in the heart. The anger and frustration among buyers and growers is justified. You are counting on us. This is our core business.
Like every day, everyone was at the starting blocks at six o'clock. That was true for auctioneers and for buyers at home and abroad. Well before the clock of six, after reports from buyers revealed that the vast majority of buyers could not log in.
At that point, all alarm bells go off. If buying power is missing, you cannot start auctions. It leads to distorted pricing and it is unfair if buyers are locked out. Every effort was made to find the cause of the problem. It turned out there was a disruption in authentication when logging in. This is needed to ensure that the person logging in is actually allowed to use KOA (Remote buying). The communication between the part of KOA that performs this task and the system that states who is allowed access was not working properly. Finding the cause of this took an unusually long time. The specialists even had to search back through adjustments made in January and over the summer. That is not an excuse, but it is the explanation.
On the IT front, we are rebuilding while the store remains open. We are modernising our systems step by step so that they meet today's and tomorrow's requirements again. What we are also doing is standardising the way we auction across all locations. This is also necessary for the introduction of One Auctioning. Our specialists always test everything beforehand, but unfortunately certain issues only come to light in daily practice. That was also the case on Friday. The impact this time was unprecedented. Since the introduction of KOA, such a long-term disruption has fortunately never occurred. And I hope we will never experience it again.
We evaluate every major disruption to determine what caused it, what lessons we should learn from the disruption and how we can prevent its recurrence in the future. Of course, we do the same now. In this case, we are also having a review carried out by an independent external party. The latter will take some weeks to complete.
You might ask why something needs to be replaced or modified that works. The answer is, if we don't, there comes a time when the system crashes. Moreover, it means a standstill, because we cannot build in new functionalities with the outdated systems, and those systems are not fast enough and do not have the required capacity. Achieving maximum security can also be a reason. In short, we do need to innovate. KOA is a unique system, which on an average day 3000 buyers use at the same time, real-time and location independent. Our KOA system can even handle double that number.
What has happened is really terrible for everyone. That includes buyers, their customers, growers, transporters and also our own staff. Instead of getting to work, everyone could do little but wait. The uncertainty of whether we would start auctioning, and if so at what time, was of course nerve-wracking. In the meantime, every effort was made to keep buyers and growers informed of the latest state of affairs through various communication channels. As long as it was not clear whether and at what time auctioning could take place, various scenarios were taken into account. In the end, as I understand it, almost everyone was happy that we could start auctioning. We appealed to all office staff to come and help with logistics. Many answered the call. I myself went to help in Naaldwijk, as I had already been inducted as an order picker. I saw with my own eyes that people rolled up their sleeves with great commitment and spirit. Everyone realises that we do this for our growers and buyers. I am proud of that. I would like to thank everyone who contributed in their own way to minimise the inconvenience and damage. I realise very well that a lot was also asked of buyers' employees. They too had to wait for a very long time and work late. This is extremely regrettable and my thanks go to them as well.
I realise that, apart from Friday's breakdown, our logistics service is lacking. The root cause of this is not IT, but the severe shortage in the labour market. Last week, Leendert-Jan Plaisier recorded a video message about this. We are engaging with our clients on how to ease the pain as much as possible. In these very turbulent times, priority number one is the quality of our daily service in all (customer) processes. This is where we do everything in our power and do not skimp.