Skip to content
Search

DEV Community πŸ‘©β€πŸ’»πŸ‘¨β€πŸ’»

Cover image for Creating a website aggregator with ChatGPT, React, and Node.js πŸš€
novu profile image Nevo David
Nevo David for novu

Posted on

Creating a website aggregator with ChatGPT, React, and Node.js πŸš€

#webdev #javascript #programming #tutorial

A website aggregator is a website that collects data from other websites across the internet and puts the information in one place where visitors can access it.

There are many versions of website aggregators; some are search engines such as Google and Duck Duck go, and some can have more of a Product Hunt structure where you can see a picture and a short text.

You will usually scrape the website, take their metatags and h1-6 tags, scan their sitemap.xml, and use some pattern to sort the information.

Today I am going to use a different solution 😈
I will take the entire website content, send it to ChatGPT, and ask them to give me the information I need.

It's kinda crazy to see ChatGPT parses the website content

So lettttsss do it πŸš€

Scrape

In this article, you'll learn how to build a website aggregator which scrapes content from a website and determines the website's title and description using ChatGPT.

What is ChatGPT?

ChatGPTΒ is an AI language model trained byΒ OpenAIΒ to generate text and interact with users in a human-like conversational manner. It is worth mentioning that ChatGPT is free and open to public use.

Users can submit requests and get information or answers to questions from a wide range of topics such as history, science, mathematics, and current events in just a few seconds.

ChatGPT performs other tasks, such as proofreading, paraphrasing, and translation. It can also help with writing, debugging, and explaining code snippets. Its wide range of capabilities is the reason why ChatGPT has been trending.

ChatGPT is not available as an API yet :( In order to use we will have to scrape our way in 😈

Novu - the first open-source notification infrastructure

Just a quick background about us. Novu is the first open-source notification infrastructure. We basically help to manage all the product notifications. It can be In-App (the bell icon like you have in Facebook - Websockets), Emails, SMSs and so on.

Novu

I would be super happy if you could give us a star! And let me also know in the comments ❀️
https://github.com/novuhq/novu

Limitation with ChatGPT

As previously mentioned, ChatGPT is not accessible through a public API. Instead, we can use web scraping techniques to access it. This involves automating the process of logging in to the OpenAI website, solving the captcha (you can use 2captcha for this), and sending an API request with the OpenAI cookies. Fortunately, there is a public library that can handle these tasks for us. Keep in mind that this is not a formal API, so you may encounter limitations if you attempt to make a large number of requests. Additionally, it is not suitable for real-time requests. If you want to use it, consider implementing a queue system for background processing.

Project Set up

Here, I'll guide you through creating the project environment for the web application. We'll use React.js for the front end and Node.js for the backend server.

Create the project folder for the web application by running the code below:

mkdir website-aggregator
cd website-aggregator
mkdir client server
Enter fullscreen mode Exit fullscreen mode

Setting up the Node.js server

Navigate into the server folder and create a package.json file.

cd server & npm init -y
Enter fullscreen mode Exit fullscreen mode

Install Express, Nodemon, and the CORS library.

npm install express cors nodemon
Enter fullscreen mode Exit fullscreen mode

ExpressJSΒ is a fast, minimalist framework that provides several features for building web applications in Node.js,Β CORS is a Node.js package that allows communication between different domains, andΒ Nodemon is a Node.js tool that automatically restarts the server after detecting file changes.

Create an index.js file - the entry point to the web server.

touch index.js
Enter fullscreen mode Exit fullscreen mode

Set up a Node.js server using ExpressJS. The code snippet below returns a JSON object when you visit the http://localhost:4000/api in your browser.

//πŸ‘‡πŸ»index.js
const express = require("express");
const cors = require("cors");
const app = express();
const PORT = 4000;

app.use(express.urlencoded({ extended: true }));
app.use(express.json());
app.use(cors());

app.get("/api", (req, res) => {
    res.json({
        message: "Hello world",
    });
});

app.listen(PORT, () => {
    console.log(`Server listening on ${PORT}`);
});
Enter fullscreen mode Exit fullscreen mode

Install the unofficialΒ ChatGPT API library andΒ Puppeteer. The ChatGPT API uses Puppeteer as an optional peer dependency to automate bypassing the Cloudflare protections.

npm install chatgpt puppeteer
Enter fullscreen mode Exit fullscreen mode

To use the ChatGPT API within the server/index.js file, you need to configure the file to use both the require and import keywords for importing libraries.

Therefore, update the server/package.json file to contain the type keyword.

{ "type": "module" }
Enter fullscreen mode Exit fullscreen mode

Add the code snippet below at the top of the server/index.js file.

import { createRequire } from "module";
const require = createRequire(import.meta.url);
//...other code statements
Enter fullscreen mode Exit fullscreen mode

Once you have completed the last two steps, you can now use ChatGPT within the index.js file.

Configure Nodemon by adding the start command to the list of scripts in the package.json file. The code snippet below starts the server using Nodemon.

//In server/package.json

"scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "start": "nodemon index.js"
  },
Enter fullscreen mode Exit fullscreen mode

Congratulations! You can now start the server by using the command below.

npm start
Enter fullscreen mode Exit fullscreen mode

Setting up the React application

Navigate into the client folder via your terminal and create a new React.js project.

cd client
npx create-react-app ./
Enter fullscreen mode Exit fullscreen mode

Delete the redundant files, such as the logo and the test files from the React app, and update the App.js file to display "Hello World" as below.

function App() {
    return (
        <div>
            <p>Hello World!</p>
        </div>
    );
}
export default App;
Enter fullscreen mode Exit fullscreen mode

Navigate into the src/index.css file and copy the code below. It contains all the CSS required for styling this project.