Only human
“To err is human, to forgive is divine” as the saying goes. And yes, on 11th January 2023 it seems we showed our human side by erring. In the interests of openness and transparency, this is the story.
Our job is to make the software that helps our clients spend less time managing donor data and more time doing what matters. We’ve been doing it for some time, and we love to see the positive impact it has made to our clients. We’re fundraisers ourselves, and helping our clients grow their fundraising is what gets us out of bed in the mornings.
On 11th January we deployed a new update to Donorfy, as we do most weeks. Along with some helpful new features the update inadvertently introduced a bug. Bugs in software are not new of course: barely a day goes by without the apps we all use being updated for “bug fixes and performance improvements”, but this bug was tricky and had evaded our code review and testing processes.
Its effect was that some attachments on some constituent timelines were permanently erased by Donorfy. However, it was not immediately apparent to us how that was happening. A couple of clients had noticed unusual behaviour, but we couldn’t pin down exactly what was going on until July when one client was able to show us precisely what they were doing to make it occur*.
So when we understood what had happened we were able to set about understanding how widespread it had become. Thankfully it had affected a minority of users of our platform, and most of them in a minor way**. But obviously for some it was potentially a serious issue: attachments were missing and needed to be reinstated.
As soon as this became apparent we (a) fixed the bug and deployed the fix so that it couldn’t happen again; (b) created a utility in the product for the affected clients to see what attachments had been erased and restore them from other copies where available; (c) notified the affected users with offers of help to discuss their situation in particular and advise appropriately***.
For some of our customers this understandably has caused some concern, and in some cases anger. We understand and we are sorry. We are standing with our customers to help them through the process of getting back what was lost. For others, the impact was minimal and they are able to move on without extra help.
So why am I telling you this, especially on a public forum? As the CEO of the business it’s my job to take responsibility for it. Is it pleasant? No. Is it embarrassing? Yes it is. Do we regret creating the bug? Of course.
One of our values is to challenge convention. Conventional behaviour in a situation like this might be to keep our heads down and wait for it to pass. Or to issue a bland apology and not admit blame.
That’s not us. Another of our values is Do What’s Right, and if values are to mean anything it’s at times like this, hence this post. In doing so we risk making it a bigger thing than it needs to be, perhaps appearing foolish in the process. But not doing so would be to trivialise it for those for whom it is a genuine concern, and that is not our intention. Clients, those considering using Donorfy and interested observers deserve an explanation, so here we are.
It’s often said that mistakes are inevitable but it’s how you respond to them that matters. What have we learned and what changes will we make?
Guard against complacency - we are reviewing our quality assurance and disaster recovery processes, and asking ourselves the “what-if” questions with a renewed energy.
Our team really cares - they responded immediately to find a solution and make things better for our clients. No one looked for anyone to blame - we’re a team.
Improve our ability to identify patterns - to help us to pin down the root cause more quickly when reports of an issue like this occur.
Communication matters - it was right to notify all of the affected clients once we knew what we were dealing with, even though in most cases the communication was likely more alarming than the issue itself.
Thanks to our clients for your patience, understanding, and your (divine) forgiveness. We’re only human but we’re sorry, and will be stronger for this experience. I encourage you to get in touch if you’d like to discuss it further.
* This is what happened: where a user uploaded an attachment with a file name that was identical to one that had already been uploaded, the older attachment was removed.
** For context, circa two percent of all attachments were erased. Of those, the majority can be, or have been automatically reinstated. All affected customers were emailed - to either their main contact or billing contact. If you’re a customer and haven’t had the email you weren’t affected.
*** It’s worth noting what it wasn’t: It was not a case of data falling into the wrong hands. We had not been hacked. It was not a security issue. Donorfy has not been compromised.