Step Functions - what, why and how?

Just around a few weeks ago, I was looking out for a viable solution for one of the projects I was working on and that led me to AWS Step Functions.

In this blog, we’ll discuss what Step Functions are, what they are used for, how to use them, and the advantages or disadvantages that I came across while working with these over the last couple of weeks.

So, let’s get started :)

Introduction

For those who are not familiar with them, AWS Step Functions are basically Amazon’s way of providing a ‘state machine as a service’. We’ll get into what that means in just a bit.

To explain this with an example, If you have ever used AWS you might’ve probably used or at least heard about AWS Lambda. Lambda functions allows you to easily create a serverless execution environment into a stateless, highly scalable component.

They’re extremely good to a certain level, most of the times you’d have to set up some event triggers to actually invoke your lambda functions, it could be anything from serving a http request to generating a csv report/optimizing images whenever someone uploads an object in a S3 bucket.

However, a lot of more evolved use cases do require some sort of way to actually manage some sort of state. Let’s consider an example to understand this in a better way:

Imagine you have a service which utilizes three basic lambda functions :

1. To upload images from a http request to an s3 bucket.
2. Process & Optimize the uploaded images
3. Send an email notification to the user once the job is completed.

Now since these lambdas will work in a completely separate isolated environment, you’ll need an intermediatry between these 3 functions, in order to share context between all of them. In layman terms, keeping a track of the images you need to process and the user’s email address.

That’s where Step Functions comes into picture! It allows you to easily set up scalable workflows for your serverless components or in this case, lambda functions.

Okay, now let’s get back to the term ‘state as a machine’

In order to learn this, you need to be able understand a concept called FSM Model or Finite-State-Machine Model.

What’s FSM Model?

Finite-State Machines are also known as a behavioral model. The reason why we call it a ‘Finite’ State machine is because it’s an abstract machine (system) that can be in 1 (one) state at a time, but it can also switch between a finite number of states.

FSM consists of two main things : States & Transitions.

FSM model does a simple job — it uses given states and transitions to complete the tasks at hand. This machine is defined solely by its states and the relationship these states have between themselves.

To understand this deeply, let’s take a look at ‘closed-door example’ (thanks to @rehemagi).

Consider that there’s an elevator in your building. The elevator door can either be open or closed, and these are the only two possible states. Now, the transition part is the switch between two states, but you need to provide some input first to get there. When you close the door, you’replacing an input. Additionally, the sequence of opening the closed door is known as the switch between two states (transition)

The conclusion is simple — FSMs are a method of modeling your system by defining the states and transitions between those states.

So, What are Step Functions?

Now that we know what FSM (Finite-State Machines) are, let’s dive deep into AWS Step Functions.

Step Functions are AWS’s Finite-State Machines service that’s entirely managed while also being serverless. Step Functions are made of state machines (workflows) and tasks. Tasks are individual states or single units of work.

In layman terms, AWS Step Functions allows you to easily set up scalable workflows for your serverless components.

These serverless components may include AWS Lambda functions, or other microservices. Applications built with microservices are way more resilient and easier to scale. While creating these workflows, you can make use of three important states : Pass State, Parallel State, Choice State.

1. Pass State : It is used to pass the input of one state (or lambda function) as the output to the next state. We can also delay execution when we need to using wait states.

2. Parallel State : Using this we can start multiple branches of execution at the same time, such as running multiple Lambda functions at once.

3. Choice State : Using this we can add branching logic to our state machine, and make decisions based on their input.

Ever gone through those flowcharts in your introductory CS classes?

Wondering when would you ever use them? Now’s the time! 🙂

Let’s try building a simple workflow using the above states that we just discussed about.

Problem Statement : Imagine that you need to build a form that validates user account data as well, upon submission. If the account data is valid, it saves the submission data in persistent storage and an email is sent to the administrator about a successful submission.

In case it’s invalid, do not store the submission data but rather send a notification email to the administrator about an unsuccessful attempt.

Let’s try to break this down into set of smaller tasks, In order to build this workflow, we would need :

• One task that invokes a Lambda function that validates the data provided to create the account.
• One task that puts an item into a DynamoDB table.
• Two tasks that put a message to an SNS topic.
• One choice flow state, to decide which action to take, depending on the results of a Lambda function.

Awesome, now that we have a broader understanding of what we want to achieve, let’s try to visualize this workflow. For this, I’ll be using one of my favourite tools called HackerDraw, but feel free to use anything.

Awesome! Now that we have a clear picture of what we want to achieve, let’s actually build this in AWS Step Functions.

In order to build this, you need to understand that Step Functions uses something called ASL (or Amazon State Language). Don’t worry it’s basically just JSON.

Here’s a quick single state ‘Hello World’ example :

Copy 
1{
2  "StartAt": "HelloWorld",
3  "States": {
4    "HelloWorld": {
5      "Type": "Pass",
6      "Result": "Hello World!",
7      "End": true
8    }
9  }
10}

The workflow starts with StartAt which indicates which step it is going to first.

Then we have all of our states for our workflow inside the states property. We can indicate whether a state is an end state or passes on to another state using End and Next.

But, How can I add another state (or step) to this workflow?

Let’s utilize the above example itself and try to define a second step in this workflow named ‘World’.

This is how the flow of the program should look like :

In the above diagram, we’re basically splitting this state up into a Hello state which then passes on to a World state.

Copy 
1{
2  "StartAt": "Hello",
3  "States": {
4    "Hello": {
5      "Type": "Pass",
6      "Result": "Hello",
7      "Next": "World"
8    },
9    "World": {
10      "Type": "Pass",
11      "Result": "World!",
12      "End": true
13    }
14  }
15}

You might be asking that although this ASL syntax is quite simple but it seems a bit overwheming at first, doesn’t it? and there’s a bit of learning curve involved.

Someone might need to actually go through all this in order to build their first step function.

So, Isn’t there a simpler way to do this?

Yes, there is! In June of 2021, AWS announced something called Workflow Studio.

Creating Step Functions with ‘Workflow Studio’

Workflow Studio is great for us developers who are new to Step Functions, because it reduces the time to build our first workflow and provides an accelerated learning path where we can learn by doing. Workflow Studio is also useful for developers who are experienced in building workflows, because they can now develop them faster using a visual tool.

To get started,

1. Go to the Step Functions console and create a state machine. You will see an option to start designing the new state machine visually with Workflow Studio.

   

2. In the left pane, the States Browser, you can view and search the available actions and flow states.

   

   Actions are basically operations which you can perform using AWS services, like invoking an AWS Lambda function, making a request with Amazon API Gateway etc.

3. Now, In order to build what we discussed earlier. Let us drag & drop an AWS Lambda block from states browser and add it to our workflow.

   

   You can edit the configuration of your function in the right pane. For example, you can change the name (as shown in step 2). You can also edit which Lambda function should be invoked from the list of functions deployed in this account

4. Now, let’s add our ‘Account Valid’ choice state. In the states browser, click on ‘flow’ tab and let’s drag & drop a choice state, right after our Lambda invocation block.

   

   A choice will validate the input using choice rules. Based on the result of applying those rules, the state machine will direct the execution to a different path.

   In step 2, you enter a name for it. In step 3, you can define the rules. For this use case, you will have one rule with a specific condition.

   

5. In the next step, whenever there’s an error, we want to send an email to the administrator. For this you first need to set up a SNS topic.

   
   
   

   Once done, and then drag it to the state machine, as shown in step 2. Drag a Fail state flow to the state machine, as shown in step 3, so that when this branch of execution is complete, the state machine is in a fail state.