My approach building a serverless application that notifies me of gasoline prices

AWS offers a variety of products to create any application you can think of. With DynamoDB, you can trigger a Lambda function to perform additional work each time a DynamoDB table is updated.

In this post, I’ll show how these two technologies can work together. Without further ado, let’s get started.

What I'm building

NYC Open Data has a public API that details gasoline price information of all the New York Regions. A new gasoline price gets added every week.

I had the idea of building a system that can notify me of the gasoline prices with the condition when the price goes down. I’m going to show you how I did this. You might think the idea sounds a little vague or easy to do but the most important thing about the project is how I did it which is what I want to emphasize here in this blog.

Setting up the project

Getting your repository set up

I enjoy doing side projects with the latest trends in the market. In the project, I use terraform. For those of you that don’t know terraform is infrastructure as code that lets you build, change, and version cloud and on-premise resources safely and efficiently. In short words, instead of using the AWS UI, we use code to create AWS services.

Another cool technology that I’m using is Serverless. Serverless is a cloud-native development model that allows developers to build and run applications without having to manage servers. It’s really cool because we just need to provide the code and AWS will do all the heavy tasks in the background.

Building the infrastructure

Something I have learned over time using terraform is that I like to structure the different resources I want to build using modules. For example, if I’m planning to use an EC2 I normally create a compute module and allocate all the logics that falls in EC2. Here’s what I did:

module "dynamodb" {
  source = "./dynamodb"
}

module "sns" {
  source = "./sns"
  email  = var.email
}

module "lambda" {
  source                          = "./lambda"
  gasoline_prices_table_arn       = module.dynamodb.gasoline_prices_table_arn
  gasoline_price_table_stream_arn = module.dynamodb.dynamodb_stream_arn
  sns_arn                         = module.sns.sns_arn
}

Once we have the modules in place I start building each module separately and export any variable that a given module needs.

In AWS every resource that we want to create is private. If we want to execute a resource either programmatically or get access to it, we need to explicitly tell AWS to do it.

I created an IAM role in the lambda module to detail all the permissions that the Lambda needs.

resource "aws_iam_policy" "iam_role_policy_for_lambda" {

  name        = "aws_iam_policy_for_terraform_aws_lambda_role"
  description = "AWS IAM Policy for managing aws lambda role"
  policy = jsonencode({
    Statement = [{
      Action   = ["s3:GetObject", "s3:PutObject"],
      Effect   = "Allow",
      Resource = "${aws_s3_bucket.price_fetcher_deployment.arn}"
      },
      {
        Action   = ["dynamodb:Scan", "dynamodb:Query", "dynamodb:PutItem"],
        Effect   = "Allow",
        Resource = "${var.gasoline_prices_table_arn}"
      },
      {
        Action   = ["logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents", "logs:DescribeLogStreams", "logs:createExportTask"],
        Effect   = "Allow",
        Resource = "*"
      },
      {
        Action   = ["dynamodb:DescribeStream", "dynamodb:GetRecords", "dynamodb:GetShardIterator", "dynamodb:ListStreams"],
        Effect   = "Allow",
        Resource = "${var.gasoline_price_table_stream_arn}"
      },
      {
        Action = ["sns:Publish"],
        Effect   = "Allow",
        Resource = "${var.sns_arn}"
      }
    ]
    Version = "2012-10-17"
  })
}

As you can see above, we have explicitly told AWS which permissions my AWS needs to execute the Lambda. This is something that you have to think about as you continue adding functionalities to your Lambda.

If you wondered before why I was passing some variables to the lambda from the dynamodb module. In order to grant permission to our Lambda to execute queries in Dynamo DB. We need to get the reference of the table and explicitly add it to the policy.

Lambda and More

It only takes a few minutes to create a lambda using the serverless framework. Make sure you have the serverless package installed on your machine. Depending on what type of serverless template you want to create (e.g javascript, python) you specify it in the serverless CLI. For example

sls create --template aws-python3 --path myService

The serverless.yml is the heart of a serverless application. This file describes the entire application infrastructure, all the way from the programming language to resource access.

A few examples of properties we can use:

• The functions we want to run
• Environment variables
• Depending on the cloud provider we’re using, we can specify the role and the region our code will get run on

For more information about this, check out this documentation.

Here’s what my YAML looks like:

service: aws-price-fetcher

frameworkVersion: "3"

provider:
  name: aws
  runtime: python3.8
  region: us-east-1
  deploymentBucket:
    name: price-fetcher-serverlessdeploymentbucket
  iam:
    role: arn:aws:iam::008735640664:role/Price-Notifier-Role

functions:
  price_fetcher:
    handler: handler.price_fetcher
    description: Fetches gasoline price from public API
    events:
      - schedule: cron(* * 0 ? * WED *)
  price_publisher:
    handler: handler.price_publisher
    description: Publish gasoline drops

plugins:
  - serverless-python-requirements

I won’t go to each one of the properties because some of them are self-explanatory.

service is the name of our lambda in AWS.

provider/region: Lambas are region-specific. What this means is that each Lambda function lives in a specific AWS region.

deploymentBucket: when we create a lambda our lambda has to live somewhere in AWS. This is why in terraform we are creating a bucket for our lambda to live in and by this means we need to specify it in the YAML file.

role The role created in terraform along with its permissions to execute resources in AWS.

functions: When we create a lambda we can have multiple functions living in one lambda. To make this project easy, I chose to have 1 lambda with 2 functions one for fetching the gasoline price from the public API and another one to publish the gasoline drops.

plugins are just the different libraries we are using in serverless. We're using serverless-python-requirements because we need some libraries to execute our code, for example, requests. If you’re curious to see more libraries, check out this documentation that details all the plugins serverless has.

In the lambda, I created a function called price_fetcher whose purpose is to query the DynamoDB table, if there are any records it gets the most recent gasoline price, calculates against the last record we have in the database if the price has dropped it insert the record to the table otherwise, it doesn’t do the insertion.

 def price_fetcher(event: Dict[str, Any], _: Any) -> None:
    """
    Fetches gasoline price from public API
    Calculate the most recent gasoline price with the last record saved in the DynamoDB table
    if the price has dropped, it inserts that record into the table otherwise, it skips it.
    """
    items = query(GASOLINE_PRICE_TABLE_NAME)
    LOG.info(f"Got items: {items}")

    gasoline_api_response = request()

    gasoline_api_json = gasoline_api_response.json()

    most_recent_gasoline_price: Dict = gasoline_api_json[