In the fiercely competitive landscape of ride-hailing applications, providing a seamless and engaging user onboarding experience is paramount to achieving high user retention and satisfaction rates. This hands-on project focuses on building a robust user onboarding workflow using AWS Step Functions and Terraform, ensuring that new users have a smooth and efficient start with your ride-hailing service. By leveraging AWS Step Functions, we can orchestrate a series of automated tasks, ensuring each step of the onboarding process is executed flawlessly. This approach not only enhances the user experience but also streamlines backend processes, making the entire system more resilient and scalable.

Building A User Onboarding Workflow For An Application

Overview

AWS Step Functions is a serverless orchestration service that allows you to combine AWS Lambda functions and other AWS services to build complex workflows. It simplifies the coordination of various tasks, handling retries, parallel execution, and error catching, ensuring that workflows run smoothly and are easy to manage. For a ride-hailing application, this means that user data validation, account creation, email notifications, and initial user preferences can all be managed in a coherent and automated manner.

In this comprehensive guide, we will explore the architecture of a user onboarding workflow for a ride-hailing application, triggered via Amazon API Gateway. This architecture includes several essential components that work together to provide a seamless user onboarding experience.

The Importance of AWS Step Functions

AWS Step Functions is integral to building scalable and resilient workflows in cloud applications. Its state machine approach allows for visual representation and straightforward debugging of complex workflows. Key benefits include:

1. Error Handling and Retry Logic: Automatically retries failed tasks and handles errors gracefully.
2. Parallel Execution: Executes tasks in parallel, reducing overall workflow execution time.
3. Task Coordination: Coordinates a sequence of AWS Lambda functions and integrates with other AWS services.
4. Logging and Monitoring: Provides detailed logs and metrics for monitoring and troubleshooting.

Click here to read a detailed and comprehensive guide to fully understanding step functions.

Core Components and Their Interactions

1. API Gateway

API Gateway serves as the entry point for user registration requests. When a new user signs up, API Gateway triggers the AWS Step Functions state machine, initiating the onboarding workflow.

2. AWS Step Functions

AWS Step Functions orchestrates the entire onboarding process, coordinating between various tasks. The workflow is defined as a state machine with each state representing a different task.

3. AWS Lambda Functions

Lambda functions are serverless compute services that execute the core logic of each task in the workflow. The following Lambda functions are used in the onboarding process:

• ValidateUserData: Ensures the incoming user data meets the required standards.
• CreateCognitoUser: Creates a new user account in Amazon Cognito for authentication.
• SendWelcomeEmail: Sends a personalized welcome email using Amazon SNS (Simple Notification Service).
• SetupInitialUserPreferences: Configures initial user preferences in a DynamoDB table.

4. Amazon Cognito

Amazon Cognito manages user authentication and authorization, providing a secure way to handle user credentials. It ensures that user accounts are created and managed securely.

5. Amazon SNS (Simple Notification Service)

Amazon SNS is used to send personalized welcome messages to new users. It ensures that notifications are delivered reliably and promptly.

6. Amazon DynamoDB

Amazon DynamoDB stores initial user preferences, ensuring quick and scalable access to user-specific settings. This allows the application to provide a personalized experience for each user from the outset.

7. IAM Roles and Policies

IAM roles and policies ensure secure and appropriate access to AWS resources for each Lambda function. They define the permissions required for each function to interact with other AWS services securely.

Communication Flow

The following steps outline the communication flow and interactions between the components:

1. Trigger via API Gateway: The process begins when a new user registration request is received via API Gateway. This triggers the Step Functions state machine.
2. Step Functions Execution: The state machine starts with the ValidateUserData Lambda function to check the validity of the input data. If the data is valid, the workflow proceeds to the next step.
3. User Creation in Cognito: The CreateCognitoUser function is invoked, creating a new user account in Amazon Cognito. This ensures that the new user is securely registered and can authenticate with the application.
4. Sending Welcome Email: Upon successful account creation, the SendWelcomeEmail function publishes a welcome message to an SNS topic. SNS then sends a personalized email to the new user, welcoming them to the service.
5. Setting Up User Preferences: The SetupInitialUserPreferences function is then invoked to store the user's initial preferences in a DynamoDB table. This step ensures that each user's preferences are saved and can be accessed quickly by the application.

6. Premium User Handling:

   1. Store user details in DynamoDB.
   2. Create a user in Amazon Cognito.

   For premium users, we have an additional step in the workflow. This step is executed in parallel for premium users, ensuring efficient handling of their onboarding process.

7. Completion: The workflow ensures that each step is executed in order, with retries and error handling managed by Step Functions. If any step fails, appropriate error handling logic is executed to retry the step or handle the failure gracefully.

Testing the Workflow

To ensure the workflow is functioning as expected, we'll test it using curl and Postman.

N.B: This project was designed and deployed in the us-east-1 region. For it to work in other regions you will have to do some minor modifcations. For simplicity, set your region to us-east-1.

Step 1: Create A Directory Structure

mkdir user-onboarding
mkdir user-onboarding/src

The src subdirectory is where we will place the python files for our different lambda functions.

Step2: Create the Source Code Files

src/validate_user_data.py:

import json

def lambda_handler(event, context):
    user_data = event.get('user_data', {})
    required_fields = ['username', 'email', 'password']
    
    for field in required_fields:
        if field not in user_data:
            raise Exception(json.dumps({'error': f'Missing required field: {field}'}))
    
    return {'message': 'User data is valid', 'user_data': user_data}
    

src/create_cognito_user.py:

import json
import boto3
import os

client = boto3.client('cognito-idp')

def lambda_handler(event, context):
    body = event['user_data']
    
    email = body['email']
    username = body['username']
    password = body['password']
    user_pool_id = os.environ['USER_POOL_ID']
    
    try:
        response = client.admin_create_user(
            UserPoolId=user_pool_id,
            Username=email,  # Use email as the username
            UserAttributes=[
                {
                    'Name': 'email',
                    'Value': email
                },
                {
                    'Name': 'email_verified',
                    'Value': 'true'
                }
            ],
            MessageAction='SUPPRESS'  # Suppress the welcome email
        )
        
        client.admin_set_user_password(
            UserPoolId=user_pool_id,
            Username=email,  # Use email as the username
            Password=password,
            Permanent=True
        )
        
        output = {
            'message:' 'user created successfully.'
            'email': email,
            'username': username,
        }

        return {
            'statusCode': 200,
            'body': json.dumps(output)
        }

    except client.exceptions.UsernameExistsException:
        raise Exception({
            'statusCode': 400,
            'body': json.dumps({
                'message': 'Username already exists'
            })
        })
    except Exception as e:
        raise Exception({
            'statusCode': 400,
            'body': json.dumps({
                'message': str(e)
            })
        })

send_welcome_email.py:

import boto3
import json
import os

sns_client = boto3.client('sns')
TOPIC_ARN = os.environ['TOPIC_ARN']

def lambda_handler(event, context):
    if type(event) == list:
        body = event[1].get('body')
    else:
        body = event.get('body')
    if body:
          # Parse the JSON string inside the body field
          body = json.loads(body)

          # Extract the username from the parsed JSON data
          user = body.get('username')
          message = f"Hello {user}, welcome to our ride-hailing service!"
          response = sns_client.publish(
               TopicArn=TOPIC_ARN,
               Message=message,
               Subject='Welcome to Our Ride-Hailing Service'
               )
          return {
               'statusCode': 200,
               'body': json.dumps({'message': 'Welcome email sent successfully'})
               }

It is important to note here that we're using SNS because this is a demonstration. In a typical production scenario, you'd want to use SES or an external provider.

setup_user_preferences.py:

import boto3
import json

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('UserPreferences')

def lambda_handler(event, context):
    user_data = event['user_data']
    
    preferences = {
        'username': user_data['username'],
        'preference1': 'default_value1',
        'preference2': 'default_value2'
    }
    
    table.put_item(Item=preferences)
    
    return {
        'statusCode': 200,
        'body': json.dumps({'message': 'User preferences set successfully'})
    }

store_premium_users.py:

import boto3
import json

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('PremiumUsers')

def lambda_handler(event, context):
    user_data = event['user_data']
    username = user_data['username']

    item = {
  "username": username
    }

    try: 
        table.put_item(Item=item)
        return {
            'statusCode': 200,
            'message': 'Premium user preferences uccessfully stored in table.'
        }
    
    except Exception as e:
        raise Exception(e)

Step 3: Create the necessary IAM Roles and Policies

policies.tf:

# API gateway policy
resource "aws_iam_policy" "api_gateway_policy" {
  name        = "APIGatewayStepFunctionsPolicy"
  description = "Policy to allow API Gateway to start Step Functions execution"

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "states:StartExecution",
      "Resource": "${aws_sfn_state_machine.user_onboarding_workflow.arn}"
    }
  ]
}
EOF
}

# Step Functions Policy
resource "aws_iam_policy" "stepfunctions_execution_policy" {
  name        = "StepFunctionsExecutionPolicy"
  description = "Policy for Step Functions to execute workflows"

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect   = "Allow",
        Action   = [
          "lambda:InvokeFunction",
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents"
        ],
        Resource = "*"
      },
      {
        Effect   = "Allow",
        Action   = "states:*",
        Resource = "*"
      }
    ]
  })
}

# IAM policy for Lambda execution role (This policies don't follow least priviledge principle. Don't use in Prod)
resource "aws_iam_policy" "lambda_execution_policy" {
  name        = "LambdaExecutionPolicy"
  description = "Policy for Lambda execution role"

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "dynamodb:PutItem",
        "cognito-idp:SignUp",
        "cognito-idp:AdminCreateUser",
        "cognito-idp:AdminSetUserPassword",
        "sns:Publish"
      ],
      "Resource": "*"
    }
  ]
}
EOF
}

roles.tf:

# IAM role for Lambda execution
resource "aws_iam_role" "lambda_execution_role" {
  name = "LambdaExecutionRole"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF
}

# Attach the IAM policy to the Lambda execution role
resource "aws_iam_role_policy_attachment" "lambda_policy_attachment" {
  role       = aws_iam_role.lambda_execution_role.name
  policy_arn = aws_iam_policy.lambda_execution_policy.arn
}

# API Gateway Role
resource "aws_iam_role" "api_gateway_role" {
  name = "APIGatewayStepFunctionsRole"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "apigateway.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF
}
# APIGW policy attacment
resource "aws_iam_role_policy_attachment" "api_gateway_policy_attachment" {
  role       = aws_iam_role.api_gateway_role.name
  policy_arn = aws_iam_policy.api_gateway_policy.arn
}

resource "aws_iam_role" "sfn_role" {
name = "SFNRole"

 assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect    = "Allow",
        Principal = {
          Service = "states.amazonaws.com"
        },
        Action    = "sts:AssumeRole"
      }
    ]
  }) 
}
# Attach polciy to sfn-role
resource "aws_iam_role_policy_attachment" "step_functions_policy_attachment" {
  role       = aws_iam_role.sfn_role.name 
  policy_arn = aws_iam_policy.stepfunctions_execution_policy.arn
}

Step 4: Configure Resources

lambda.tf:

data "archive_file" "lambda1" {
  type        = "zip"
  source_file = "src/validate_user_data.py"
  output_path = "validate_user_data.py.zip"
}
data "archive_file" "lambda2" {
  type        = "zip"
  source_file = "src/create_cognito_user.py"
  output_path = "create_cognito_user.py.zip"
}
data "archive_file" "lambda3" {
  type        = "zip"
  source_file = "src/send_welcome_email.py"
  output_path = "send_welcome_email.py.zip"
}
data "archive_file" "lambda4" {
  type        = "zip"
  source_file = "src/store_premium_users.py"
  output_path = "store_premium_users.py.zip"
}
data "archive_file" "lambda5" {
  type        = "zip"
  source_file = "src/setup_user_preferences.py"
  output_path = "setup_user_preferences.py.zip"
}

resource "aws_lambda_function" "pylambda1" {
  filename         = "validate_user_data.py.zip"
  function_name    = "validate_user_data"
  role             = aws_iam_role.lambda_execution_role.arn
  source_code_hash = data.archive_file.lambda1.output_base64sha256
  runtime          = "python3.12"
  handler          = "validate_user_data.lambda_handler"
}
resource "aws_lambda_function" "pylambda2" {
  filename         = "create_cognito_user.py.zip"
  function_name    = "create_cognito_user"
  role             = aws_iam_role.lambda_execution_role.arn
  source_code_hash = data.archive_file.lambda2.output_base64sha256
  runtime          = "python3.12"
  handler          = "create_cognito_user.lambda_handler"

  environment {
    variables = {
      USER_POOL_ID = aws_cognito_user_pool.user_pool.id
    }
}
}

resource "aws_lambda_function" "pylambda3" {
  filename         = "send_welcome_email.py.zip"
  function_name    = "send_welcome_email"
  role             = aws_iam_role.lambda_execution_role.arn
  source_code_hash = data.archive_file.lambda3.output_base64sha256
  runtime          = "python3.12"
  handler          = "send_welcome_email.lambda_handler"

  environment {
    variables = {
      TOPIC_ARN = aws_sns_topic.topic.arn
    }
  }
}
resource "aws_lambda_function" "pylambda4" {
  filename         = "store_premium_users.py.zip"
  function_name    = "store_premium_users"
  role             = aws_iam_role.lambda_execution_role.arn
  source_code_hash = data.archive_file.lambda4.output_base64sha256
  runtime          = "python3.12"
  handler          = "store_premium_users.lambda_handler"
}
resource "aws_lambda_function" "pylambda5" {
  filename         = "setup_user_preferences.py.zip"
  function_name    = "setup_user_preferences"
  role             = aws_iam_role.lambda_execution_role.arn
  source_code_hash = data.archive_file.lambda5.output_base64sha256
  runtime          = "python3.12"
  handler          = "setup_user_preferences.lambda_handler"
}

sns.tf:

resource "aws_sns_topic" "topic" {
  name = "UserOnboardingTopic"
}

resource "aws_sns_topic_subscription" "topic_sub" {
  topic_arn = aws_sns_topic.topic.arn
  protocol  = "email"
  endpoint  = var.email_address
}
variable "email_address" {
    # Create a default email
    # default = example@gmail.com
}

apigw.tf:

dynamodb.tf:

resource "aws_dynamodb_table" "premium_users" {
  name = "PremiumUsers"
  hash_key = "username"  # Username will be the hash key for efficient lookups
  billing_mode = "PAY_PER_REQUEST"

  attribute {
    name = "username"
    type = "S"  # String data type for usernames
  }
}

userpool.tf:

# Create Cognito User Pool
resource "aws_cognito_user_pool" "user_pool" {
  name = "user-onboarding-pool"
  
  auto_verified_attributes = ["email"]
  username_attributes      = ["email"]

  password_policy {
    minimum_length    = 8
    require_lowercase = true
    require_numbers   = true
    require_symbols   = true
    require_uppercase = true
  }
}

# Create Cognito User Pool Client
resource "aws_cognito_user_pool_client" "user_pool_client" {
  name         = "user-onboarding-client"
  user_pool_id = aws_cognito_user_pool.user_pool.id
  explicit_auth_flows = [
    "ALLOW_ADMIN_USER_PASSWORD_AUTH",
    "ALLOW_CUSTOM_AUTH",
    "ALLOW_USER_PASSWORD_AUTH",
    "ALLOW_USER_SRP_AUTH",
    "ALLOW_REFRESH_TOKEN_AUTH"
  ]
}

sfn.tf:

resource "aws_sfn_state_machine" "user_onboarding_workflow" {
  name = "user_onboarding_workflow"
  definition = <<EOF
{
  "Comment": "User Onboarding Workflow",
  "StartAt": "ValidateUserData",
  "States": {
    "ValidateUserData": {
      "Type": "Task",
      "Resource": "${aws_lambda_function.pylambda1.arn}",
      "Next": "CheckUserType"
    },
    "CheckUserType": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.user_data.user_type",
          "StringEquals": "premium",
          "Next": "PremiumUserTasks"
        }
      ],
      "Default": "CreateUserInCognito"
    },
    "PremiumUserTasks": {
      "Type": "Parallel",
      "Branches": [
        {
          "StartAt": "StorePremiumUserPreferencesInDynamoDB",
          "States": {
            "StorePremiumUserPreferencesInDynamoDB": {
              "Type": "Task",
              "Resource": "${aws_lambda_function.pylambda4.arn}",
              "End": true
            }
          }
        },
        {
          "StartAt": "CreateCognitoUser",
          "States": {
            "CreateCognitoUser": {
              "Type": "Task",
              "Resource": "${aws_lambda_function.pylambda2.arn}",
              "End": true
            }
          }
        }
      ],
      "Next": "SendWelcomeEmail"
    },
    "CreateUserInCognito": {
      "Type": "Task",
      "Resource": "${aws_lambda_function.pylambda2.arn}",
      "Next": "SendWelcomeEmail"
    },
    "SendWelcomeEmail": {
      "Type": "Task",
      "Resource": "${aws_lambda_function.pylambda3.arn}",
      "End": true
    }
  }
}
EOF
  role_arn = aws_iam_role.sfn_role.arn
}

providers.tf:

# Define profider
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "5.31.0"
    }
  }
}

provider "aws" {
  # Configuration options
  region = "us-east-1"
}

Step 5: Provision resources

• First of all, initialize the project:

terraform init

• Validate the terraform files

terraform validate

• Plan your resource provisioning to get an idea of what acions terraform will perform.

terraform plan

• Apply the configuration:

terraform apply -auto-approve

Here's how my state machine looks:

Step 6: test the Workflow

• On the API Gateway Console, select your API and click on "Resource" > "Method" >"Test".

Enter the following under headers and request body:

Headers:

 Content-Type:application/json

Request body:

{
 "user_data": {
   "username": "testuser4",
   "email": "test4@test.com",
   "user_type": "premium",
   "password": "Secure@Passw0rd"
 }
}

Now click on "Test".

You can also see the state machine execution:

• Using Curl:

curl -X POST https://xyz.execute-api.us-east-1.amazonaws.com/dev/user_onboarding \
  -H "Content-Type: application/json" \
  -d '{
        "username": "testuser",
        "email": "testuser@example.com",
        "password": "Password123!",
        "user_type": "regular"
      }'

Replace values as neccessary.

• Using Postman

Final Thoughts on the User Onboarding Workflow Project

Building a robust and scalable user onboarding workflow for a ride-hailing application involves integrating several AWS services such as API Gateway, Lambda, Step Functions, Cognito, SNS, and DynamoDB. By leveraging Terraform for infrastructure as code, we can efficiently manage and deploy our AWS resources, ensuring consistency and reliability across environments.

This project showcases the power of AWS Step Functions in orchestrating complex workflows. The state machine coordinates tasks such as user data validation, account creation, sending welcome emails, and setting up user preferences, all in a seamless, automated manner. The use of parallel states for handling premium users demonstrates how Step Functions can manage complex branching logic effectively.

Benefits of Executing Step Functions from a Lambda Function

While triggering Step Functions directly from API Gateway is a straightforward approach, there are several advantages to using a Lambda function as an intermediary:

1. Enhanced Security: Lambda can act as a gatekeeper, ensuring that only authenticated and authorized requests trigger the Step Functions. This adds an additional layer of security, protecting your workflows from unauthorized access.
2. Custom Business Logic: By using a Lambda function, you can implement custom pre-processing logic before starting the Step Functions execution. This includes validating inputs, transforming data, logging requests, or even enriching the event data with additional context.
3. Improved Error Handling: Lambda provides more granular control over error handling. You can catch and handle exceptions, retry logic, and send detailed error responses back to the client, ensuring a better user experience.
4. Seamless Integration: Lambda functions can integrate with other AWS services or third-party APIs before triggering the Step Functions, allowing for more complex workflows and interactions that might not be possible with API Gateway alone.
5. Decoupling API Gateway and Step Functions: Using Lambda decouples the API Gateway from the Step Functions. This separation of concerns makes the architecture more modular and easier to maintain. Changes in the workflow logic (Step Functions) do not necessarily require changes in the API Gateway configuration.
6. Improved Integration with API Gateway: Integrating the API Gateway with AWS lambda rather than directly with Step Functions helps you to have more granular control over request integration, method integration, integration response and method response. Thereby providing better user service to your app users.

In conclusion, building a user onboarding workflow for a ride-hailing application using AWS services like Step Functions, API Gateway, Lambda, Cognito, SNS, and DynamoDB showcases the power and flexibility of cloud-native architectures. Leveraging Terraform for infrastructure as code streamlines deployment and management, while AWS Step Functions efficiently orchestrates complex workflows with ease. Integrating Lambda functions as intermediaries enhances security, enables custom business logic, improves error handling, and decouples components for a more modular and maintainable architecture.

Here's a similar project using step functions for a payment processing workflow.

Visit this GitHub repository to get the complete code used in this project.

Happy Clouding !!!