Vertex AI Vision Queue Detection App

1. Objectives

Overview

This codelab will focus on creating a Vertex AI Vision application end-to-end to monitor queue size using retail video footage. We will use the pretrained Specialized model Occupancy analytics inbuilt features to capture the following things:

  • Count the number of people standing in the queue.
  • Count the number of people getting served at the counter.

What you'll learn

  • How to create an application in Vertex AI Vision and deploy it
  • How to set up an RTSP stream using a video file and ingest the stream into Vertex AI Vision using vaictl from a Jupyter Notebook.
  • How to use the Occupancy Analytics model and its different features.
  • How to search for videos in your storage Vertex AI Vision's Media Warehouse.
  • How to connect output to BigQuery, write SQL query to extract insights from the model's json output and use the output to label and annotate the original video.

Cost:

The total cost to run this lab on Google Cloud is about $2.

2. Before You Begin

Create a project and enable APIs:

  1. In the Google Cloud console, on the project selector page, select or create a Google Cloud project. Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project. Go to project selector
  2. Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.
  3. Enable the Compute Engine, Vertex API, Notebook API and Vision AI API. Enable the APIs

Create a service account:

  1. In the Google Cloud console, go to the Create service account page. Go to Create service account
  2. Select your project.
  3. In the Service account name field, enter a name. The Google Cloud console fills in the Service account ID field based on this name. In the Service account description field, enter a description. For example, Service account for quickstart.
  4. Click Create and continue.
  5. To provide access to your project, grant the following role(s) to your service account:
  • Vision AI > Vision AI Editor
  • Compute Engine > Compute Instance Admin (beta)
  • BigQuery > BigQuery Admin .

In the Select a role list, select a role. For additional roles, click Add another role and add each additional role.

  1. Click Continue.
  2. Click Done to finish creating the service account. Do not close your browser window. You will use it in the next step.

3. Set up Jupyter Notebook

Before creating an App in Occupancy Analytics, you must register a stream which can be used later by the App.

In this tutorial you create a Jupyter Notebook instance that hosts a video, and you send that streaming video data from the notebook. We are using jupyter notebook as it offers us flexibility to do execute shell commands as well as run custom pre/post processing code in a single place that is very good for rapid experimentation. We wiil use this notebook to:

  1. Run rtsp server as a background process
  2. Run vaictl command as background process
  3. Run queries and processing code to analyze occupancy analytics output

Create a Jupyter Notebook

The first step in sending video from a Jupyter Notebook instance is creating the notebook with our service account created in the previous step.

  1. In the console, go to the Vertex AI page. Go to Vertex AI Workbench
  2. Click User-Managed Notebooks

65b7112822858dce.png

  1. Click on New Notebook > Tensorflow Enterprise 2.6 (with LTS) > Without GPUs

dc156f20b14651d7.png

  1. Enter the name for jupyter notebook. For more information, see Resource naming convention.

b4dbc5fddc37e8d9.png

  1. Click on ADVANCED OPTIONS
  2. Scroll down to Permissions Sections
  3. Uncheck Use Compute Engine default service account option
  4. Add the Service account email created in the previous step. And click on Create.

ec0b9ef00f0ef470.png

  1. Once the instance has been created click on OPEN JUPYTERLAB.

4. Set up a Notebook to stream video

Before creating an App in Occupancy Analytics, you must register a stream which can be used later by the App.

In this tutorial we will use our Jupyter Notebook instance to host a video, and you send that streaming video data from the Notebook terminal.

Download the vaictl command-line tool

  1. In the opened Jupyterlab instance, Open a Notebook from the launcher.

a6d182923ae4ada3.png

  1. Download the Vertex AI Vision (vaictl) command line tool, rtsp server command line tool, open-cv tool using the following command in the notebook cell:
!wget -q https://github.com/aler9/rtsp-simple-server/releases/download/v0.20.4/rtsp-simple-server_v0.20.4_linux_amd64.tar.gz
!wget -q https://github.com/google/visionai/releases/download/v0.0.4/visionai_0.0-4_amd64.deb
!tar -xf rtsp-simple-server_v0.20.4_linux_amd64.tar.gz
!pip install opencv-python --quiet
!sudo apt-get -qq remove -y visionai
!sudo apt-get -qq install -y ./visionai_0.0-4_amd64.deb
!sudo apt-get -qq install -y ffmpeg

5. Ingest a video file for streaming

After you set up your notebook environment with required command line tools, you can copy a sample video file and then use vaictl to stream the video data to your occupancy analytics app.

Register a new stream

  1. Click streams tab on the left panel of Vertex AI Vision.
  2. Click on Register Button at the top eba418e723916514.png
  3. In the Stream name enter ‘queue-stream'
  4. In region, choose the same region selected during Notebook creation in the previous step.
  5. Click Register

Copy a sample video to your VM

  1. In your notebook, copy a sample video with the following wget command.
!wget -q https://github.com/vagrantism/interesting-datasets/raw/main/video/collective_activity/seq25_h264.mp4

Stream video from VM and ingest data into your stream

  1. To send this local video file to the app input stream, use the following command in your notebook cell. You must make the following variable substitutions:
  • PROJECT_ID: Your Google Cloud project ID.
  • LOCATION: Your location ID. For example, us-central1. For more information, see Cloud locations.
  • LOCAL_FILE: The filename of a local video file. For example, seq25_h264.mp4.
PROJECT_ID='<Your Google Cloud project ID>'
LOCATION='<Your stream location>'
LOCAL_FILE='seq25_h264.mp4'
STREAM_NAME='queue-stream'
  1. Start a rtsp-simple-server where we stream the video file with rtsp protocol
import os
import time
import subprocess

subprocess.Popen(["nohup", "./rtsp-simple-server"], stdout=open('rtsp_out.log', 'a'), stderr=open('rtsp_err.log', 'a'), preexec_fn=os.setpgrp)
time.sleep(5)
  1. Use ffmpeg command line tool to loop the video in rtsp stream
subprocess.Popen(["nohup", "ffmpeg", "-re", "-stream_loop", "-1", "-i", LOCAL_FILE, "-c", "copy", "-f", "rtsp", f"rtsp://localhost:8554/{LOCAL_FILE.split('.')[0]}"], stdout=open('ffmpeg_out.log', 'a'), stderr=open('ffmpeg_err.log', 'a'), preexec_fn=os.setpgrp)
time.sleep(5)
  1. Use the vaictl command line tool to stream the video from rtsp server uri to our Vertex AI Vision stream ‘queue-stream' created in previous step.
subprocess.Popen(["nohup", "vaictl", "-p", PROJECT_ID, "-l", LOCATION, "-c",