The world we work and life is in constant change. It doesn’t care if we like it or not. Change is inevitable.
The more we can automate, the more we are able to accommodate for those changes. Tasks that previously were done once a year, tomorrow can become once a month, or even once a day.
If we are not ready to accommodate for this changes, we will fall under water in no time.
Our story starts with one file.
This file is use by our stakeholders to add new BUS stops to our application. It is a very manual process, but that is OK. All is dandy, because it changes once a year.
There is no point on trying to automate a process that runs once a year, right ? Unless …
Now this file can change once a week. That’s at least 52 times a year.
To add to the party, we don’t have just one business stakeholder. Each one has a different file. Now you start to see our problem …
Imagine that every time the file changes, we need to manually validate the changes and then manually push it to production.
Not a big deal if you do it once a year, a bit of a pain if you have to do it every day.
Another thing that we started noticing is that there was no versioning on this file. Our stakeholders would change it, do a mistake, and then could not revert to a previous version. No way to know who did the change and when.
Garbage in, garbage out.
Hummm, that looks fun. A good candidate for automation.
As a developer my mind raced to “This is easy, put it in GIT, versioning and comparison are built in”. Simple and effective right ? Well not that simple.
No one in this world will not be able to convince business people to use GIT or learn GIT commands. This process needs to be seamless to our stakeholders. Their part of the work needs to remain the same. Technology needs to work for us, not the other way around. So how did we do it ?
Automation for Business
When I was working in BESTSELLER, a friend of mine introduced me to Microsoft FLOW. Its an easy to use workflow creator, that integrates seamless with Office 365. Business people use Office 365 🤔 …
So in our journey to automate this process starts here.
The file will live in OneDrive. We can then share this file with our stakeholders, and they can change it there.
When the document changes our MS Flow kicks in ( see below ) :
⚫ A copy of the document ( with the timestamp ) goes on a backup folder in OneDrive.
⚫ In parallel another copy is send to AWS S3 via SFTP ( this will allow us to do more after).
⚫ After the file goes to AWS S3, an email notification goes to our Product Owner mailbox.
With this simple solution we were able to solve one of the problems with almost no effort. All changes are logged by our flow and Microsoft, and it will be possible to revert to a previous version.
The amazing thing about MS Flow, and the reason that made me write this post is that anyone can do this. The interface is so easy to use. You don’t need an IT background to create boxes and fill in some inputs.
So powerful, its amazing that we don’t use this more.
This flow solves almost all of our problems. A 10m configuration simplified a task that used to take hours. But we cannot stop there.
We have to check what changed.
If you are a developers, you know that there is an easy way to do it in text files. Any diff program will tell you the differences between 2 text file.
For us, we opt for GitHub. The diff capability is already included, and gives us possibilities for the future. Pull requests. Commits messages with the person that changed. Public repository that can be shared with the authorities, since this is not confidential information.
The integration between MS Flow and GitHub is not that good, it is possible to create issues, but nothing else.
We needed something more powerful. Something that runs code. We moved our attention to AWS.
By moving the file to S3 we were able to tap into the AWS ecosystem. In AWS we were able to write a lambda that would pick up the new file in S3 and push it to GitHub.
And voila …
Once we got that working, we were able to check the differences in GitHub with zero effort. In the case of a incident, we can now report to business what change had an impact. We can show the diff between the two documents and advice on how to rollback.
In the future, we can share all the commit logs with the authorities to quickly understand what caused the bugs and fix them faster.
Business and IT working together. But can we keep pushing ?
What more can we automate more ?
Once you start automating, the automation seed starts growing in your brain. New ideas come into mind. Why not automate everything and push it to production?
This is our next phase. Automation puts autonomy and accountability on our stakeholders. We don’t want them to depend on us.
You own the process, we own the technology. If you mess up, you are accountable. Garbage in, garbage out.
We track all changes, and make them visible to everyone. Transparency must be in the core of everything we do.
The next step is to create a pipeline that can run automated tests, build and push across our environments. Changes that would take days to verify and put in production, now will take seconds. No human intervention, No human error.
I hope this article plants the automation seed in you.
What can you automate today that will help you tomorrow ?
This for us, was a good proof of concept that changed the way we look at automation.
Our teams background is technical. It is vert easy to start thinking of complex solutions and start writing code.
We didn’t need that.
There are so many tools out there that can helps us automate our work, and the most important part was, we already had them available. Chances are, like us, you already have all this software available. MS Flow is included in every Microsoft account, even if you still ( like me ) have an old hotmail account.
👏 👏 Special thanks to 👏 👏
Maxim Levitsky for the help on setting up everything we needed on AWS.
Murat Tasova, helping research the Microsoft flow components and how to configure them, specially the tricky SFTP one.
Jochem Beek, for having this problem. Without it, we would never had this brainstorming and P.O.C.