I attended AWS re:Invent 2021 in-person and one of my big takeaways was the focus on AWS Step Functions. With the late September announcement that Step Functions now support 200+ AWS Services I could see why.

So, what does that look like? Check Out the Code

In this post I'll convert a Lambda to a Step Function without Lambdas (sometimes called lambda-less or function-less 😈). The original lambda will call an "External" API Gateway, do a GET from DynamoDB, make a decision and potentially WRITE to DynamoDB. The resulting step function will look like this:

Again... NO lambdas 🤯

This project uses CDK v2.3 and was initialized using cdk init --language typescript. It's not intended to be a step-by-step tutorial. If you need one of those, follow me on Twitter and keep an eye out for my upcoming CDK Crash Course video.

Creating the Mock External Service

To get started we're going to create two API Gateways: Our "actual" one and one to mock an External API Gateway, and avoid circular dependencies. If your external service wasn't hosted by AWS / API Gateway you'd probably need to use a small lambda to do the external API call.

The mock external service lives in lib/mock-external.ts. It's fairly unremarkable in that it's a Lambda with an API Gateway 🤷‍♂️. The Lambda expects an APIGateway input with a userId pathParameter... so <mockApi>/asdf where asdf would be the "userId". From there it responds with:

  const mockUser = {
    userId: `EXTERNAL#${event.pathParameters.userId}`,
    name: (event.queryStringParameters || {}).name || name.findName(),
  };

This simulates a "lookup" in an external service... here we just consistently change the userId and we generate a random name for the user. Every time the endpoint is called the same user will get a new name (this isn't intended to be a real service, just representative of something that could have changing data).

Building the Lambda-lith

The lambdalith lives in lambda/bigLambda.ts.

It calls the external URL...

  // make axios call to "external" api to get user name
  const { data: externalUser } = await axios.get(
    `${API_URL}/${event.pathParameters.userId}`
  );

Gets the "internal" user matching the external userId:

  const userGetItemInput: DocumentClient.GetItemInput = {
    TableName: TABLE_NAME,
    Key: {
      PK: externalUser.userId,
    },
  };

  const { Item: internalUser } = await db.get(userGetItemInput).promise();

It returns early if the internal user has a "LOCKED" status (set directly via DynamoDB)... otherwise it updates the DynamoDB entry (which would insert if needed).

All "names" that the user has had will also get stored in DynamoDB with the time they were inserted.

  // store user in dynamodb
  const updateItemInput: DocumentClient.UpdateItemInput = {
    TableName: TABLE_NAME,
    Key: {
      PK: externalUser.userId,
    },
    UpdateExpression: `SET #name = :name, #status = :status, ${
      !internalUser
        ? "#history = :history"
        : "#history.#historical = if_not_exists(#history.#historical, :historical)"
    }`,
    ExpressionAttributeNames: { "#name": "name", "#history": "history", "#status": "status" },
    ExpressionAttributeValues: {
      ":name": externalUser.name,
      ":status": "ACTIVE"
    },
    ReturnValues: "UPDATED_NEW",
  };
  if (!internalUser) {
    updateItemInput!.ExpressionAttributeValues![":history"] = {
      [externalUser.name]: new Date().toISOString(),
    };
  } else {
    updateItemInput!.ExpressionAttributeNames!["#historical"] =
      externalUser.name;
    updateItemInput!.ExpressionAttributeValues![":historical"] =
      new Date().getTime();
  }
  const updatedItem = await db.update(updateItemInput).promise();

All in all this is a fairly simple example that represents a few different challenges.

Converting the Lambda-lith to a Step Function

I began the conversion by opening the AWS Step Function console and playing with the shiny "new" Workflow Studio. I used this to get a sense of the different kinds of steps I'd need.

In this case, since we want an immediate (synchronous) response from the Step Function we use an Express Step Function

My first pass (pun intended) was simply this:

But ultimately I found out that there weren't enough Pass steps in there.

Pass Steps are handy... they act as easy placeholders while designing the step function and also do data transformations.

To get started with the implementation I began with the Step Function itself and tied in the API Integration later.

🙌The Data Flow Simulator was incredibly useful in piecing together the JSONPath Expressions and how they interact with the State Machine.

First Step (and Issue)

The first step (😉) of the State Machine is to make that external API Gateway call. I wasted a lot of time on this because it's not super well documented in the case of dynamic / proxied URLs. It seems like in most cases you'd use a POST with a body to hit an API Gateway to a pre-defined / static URL that points to a Step Function... and if you do that you can skip this first PASS step. There's also an open GitHub Issue related to this which is why I ultimately created the lib/ModifiedCallApiGatewayRestApiEndpoint.ts. The regular CallApiGatewayRestApiEndpoint method can't make use of JsonPath expressions to extend endpoints. What that means is in order to make use of my userId path parameter I need to first do a data transformation and then hack it in via the modified class.

    const pass = new Pass(scope, "RoutePass", {
      parameters: {
        "apiPath.$": "States.Format('/{}', $.userId)",
        "userId.$": "$.userId",
        "name.$": "$.name",
      },
    });

The first Pass takes the userId via the input parameters from the API Gateway Integration, builds the API path that will be called (including the rendered path parameter) and then the ModifiedCallApiGatewayRestApiEndpoint puts it in the step's parameters via Parameters: { "Path.$": "$.apiPath", ...orig.Parameters },. Normally you'd be able to directly assign that, but it's not exposed as part of the L2 Construct for the CallApiGatewayRestApiEndpoint.

const callExternal = new ModifiedCallApiGatewayRestApiEndpoint(
    scope,
    "Call External APIGW",
    {
    api: props.mockExternalApi,
    stageName: "prod",
    method: HttpMethod.GET,
    resultPath: "$.external",
    resultSelector: {
        user: JsonPath.stringAt("$.ResponseBody"),
    },
    }
);

I could not get queryParameters to work correctly... it didn't like the resulting object... wasn't worth the trouble for this demo but if you have any tips let me know.

From there the API step takes the passed in RestAPI and that's it. resultPath extends the previous input and places the external API's ResponseBody in the StateMachine's context ($).

Interacting with DynamoDB

Next we make the DynamoDB Get call that the lambdalith does. We can directly interact with DynamoDB without a lambda by using the DynamoGetItem construct.

const dynamoGet = new DynamoGetItem(scope, "Get Internal User", {
    key: {
    PK: DynamoAttributeValue.fromString(
        JsonPath.stringAt("$.external.user.userId")
    ),
    },
    table: props.table,
    resultPath: "$.internal",
});

It uses the context provided by the external call $.external.user.userId to do the lookup and puts the result in the context's internal key.

If you were to sniff out the context at this point it'd look something like this (if DynamoDB already had the item*):

{
    "external": {
        "user": {
            "userId": "EXTERNAL#asdf",
            "name": "Matt Martz"
        }
    },
    "internal": {
        "Item": {
            "PK": {
                "S": "EXTERNAL#asdf"
            },
            "name": {
                "S": "Matt Martz"
            },
            ...
        }
    }
}

Choices

Before we can implement the Choice (if/switch statement) we need to define the destinations for those situations. We're going to use 3 types of results... Pass, DynamoDB PutItem and DynamoDB UpdateItem.

Pass

There are 4 end cases for the API... No Changes, Locked, Inserted, and Updated. All of the Pass steps look very similar and they all respond in the same general format:

const userWasLocked = new Pass(scope, "User Was Locked", {
    parameters: {
    id