Create a Movie Recommendation Engine with Milvus and Python

Recommender systems or recommendation engines are information-filtering systems that aim to predict and suggest things users might be interested in. These items include products, services and content such as movies, books, music or news articles.

There are various types of recommender systems, such as collaborative filtering, content-based filtering, hybrid recommendation systems and vector-based recommendation systems. Vector-based systems use the vector space to find (recommend) the closest items in the database. There are various ways to store these vectors; one of the most efficient ones is using the Milvus open source vector database. This database is highly flexible, fast and reliable and allows for trillion-byte-scale addition, deletion, updating and nearly real-time search of vectors.

This article explains how to build a movie recommender with Milvus and Python. This system will use SentenceTransformers to convert the text information to vectors and store these vectors in Milvus. Milvus enables users to search for a movie in the database based on the text information they provide.

You can find the code for this tutorial in the Milvus Bootcamp repository on GitHub, and there’s also a Jupyter Notebook.

Setting Up the Environment

For this article, you’ll need the following requirements installed:

• Python 3.x
• Python Package Manager (PIP) to install packages
• Jupyter Notebook for writing code
• Docker
• A system with at least 32GB of RAM or a Zilliz Cloud account

Python Requirements

You also need to install a set of libraries that will be needed throughout this tutorial. Install the libraries using PIP:

Vectors Data Store (Milvus)

You’ll use the Milvus vector database to store the embeddings that you’ll generate using movie descriptions. The dataset is relatively large, at least for a server running on a personal computer. So you may want to use a Zilliz Cloud instance to store these vectors.

If you prefer to stay with a local instance, you can download a docker-compose configuration and run it:

You now have all the requirements to build your movie recommender system with Milvus.

Data Collection and Preprocessing

For this project, you’ll use the Movies Dataset from Kaggle which contains metadata for 45,000 movies. You can download this dataset directly or use the Kaggle API to download the dataset using Python. To do it with Python, you need to download the kaggle.json file from the profile section of Kaggle.com and put it in a location where the API will find it.

Next, set up some environment variables for Kaggle authentication. For this, you can open the Jupyter Notebook and write the following lines of code:

Once done, you can use Kaggle’s Python dependency to download the dataset from Kaggle:

Once the dataset is downloaded, you can use the read_csv() method from pandas to read the dataset:

Database records and columns

The image shows that you have 45,466 records with 24 columns of metadata. Check all these metadata columns with the following:

Dataset columns

There are a lot of columns you don’t need to create the recommender system. You can filter out the required columns with: