Make Your Workflows Reversible by Design
Build workflows that fail safely and fix themselves.
Most automation tools make it dangerously easy to break things at scale. Once a workflow runs and starts updating databases, sending emails, or calling third-party APIs, there's usually no way to undo its actions. Mistakes can propagate instantly, and their cost can be quite high. It doesn't matter what iPaaS you use. Sooner or later, you'll end up facing a situation where you'll need to reverse the effects of a workflow. It would be great if the tools themselves had that feature. But that's not possible because not all workflows can be reversed in the same way. So, how can you design your workflows to make them easy to reverse? Follow me as I explore the options.
This article is brought to you with the help of our supporter, n8n.
n8n is the fastest way to plug AI into your own data. Build autonomous, multi-step agents, using any model, including self-hosted. Use over 400 integrations to build powerful AI-native workflows.
You're on your way to the park on a sunny Friday afternoon, thinking about all the work your team has done throughout the week. As a last task, right before you left, you prepared a notification workflow and started it. As you were walking out, the workflow was sending notifications to thousands of people on multiple channels like email, Slack, Discord, and other chat systems. You feel proud of your work and happy with the automation system you have in place to send notifications. Suddenly, one of your team members messages you with bad news. What can it be?
Apparently, the message the workflow is sending is wrong and is misleading all the users who receive it. Your heart rate goes up. You start to panic. You and your teammate discuss options to fix the situation. But, in the meantime, the workflow keeps sending notifications. You ask your teammate to stop the workflow so you don't make the situation even worse. Fortunately, there's a way to fix the issue, but it's manual and takes time. You go back to your computer and spend the next few hours going through your CRM, seeing who received the notification, sending an apology message with the error corrected using the same channel, and, finally, updating the CRM with the new notification timestamp. Then you need to restart your workflow so it continues to send the notifications. But this time it will use the correct message, not the original one.
All these actions could have been avoided. If, during the design of the workflow, you had thought about how you could reverse each execution. Or, at least, create compensation for failed actions that can't be reversed, like sending notifications. Another tactic is to use idempotent operations. What that means is that, for the same input, the same data mutation operation should return the same output. In summary, the best moment to make your workflows reversible is during design time. Let's first see how you can create compensation actions, and then study how to make sure you're using actions that can be reversed.
You only need to compensate for actions that can't be reversed. Actions that destroy information, for example, need compensation. Actions that trigger third-party changes you can't control need compensation. Actions that send asynchronous notifications need compensation. Any action that produces an irreversible effect needs compensation. And, by compensation, I mean another action that mitigates the irreversible effect of the first one. The first step to being able to mitigate anything is to log all actions and the data before and after they run. That way, you're able to understand how data changed and attempt to reverse its change. By doing that, you'd be able to compensate for an action that deletes a database record, for example. What you need to do is create an action that grabs the data before the deletion happens and then recreates it. It won't be the same record (the ID will change, for instance), but the same data will be available again. You can follow a similar tactic to mitigate actions that change data. Unfortunately, it won't work with actions that send asynchronous notifications, such as an email message.
Compensating for an action that sends an asynchronous notification requires another action that sends another notification correcting the first one. The key here is, as before, to log the data you sent through the first action so you know how to refer to it later on. Then, create an action that sends the same kind of notification as before, but with the content updated to reflect the changes you want to mitigate. This tactic can work for notifications sent by email, SMS, Slack, Discord, or any other messaging medium. What about those actions that can be reversed? Let's see what you can do with them.
Even before discussing how you can revert certain actions, it's good to understand what reverting an action means. Reversion can happen whenever you have an action whose effects can be undone. The example I gave before of compensating for the removal of a database record isn't a reversible action because the new record will have a different ID than the original one. As a rule of thumb, it's easier to reverse actions that create information than it is to reverse those that destroy it. Reversing the creation of a record is as simple as deleting it. However, reversing the action of updating a record is also the update of the same record. Again, you follow a similar tactic of keeping a log of what the data was before the action took place, so you can update the record back to what it was before.
Altogether, there are ways of designing a workflow in a way that its actions can be reversed or, if not, compensated with a mitigation. I would create a "reverse" workflow for each critical workflow I have. If something goes wrong—as it did on that Friday afternoon—I can simply pull the reverse workflow and run it against the log of the original one. At least I know that I have an escape plan in case things fail.