In addition, there is no comprehensive Python library that I could find for optimizing gifs programmatically.

I did end up figuring out an efficient solution to making gifs programmatically in Python after hours of research and have distilled it into just the necessities. In this process, we’ll use two tools: the Pillow library and Gifsicle.

The Pillow library allows us to manipulate images in various ways and Gifsicle is a CLI tool used for gif manipulation. So let’s dive into how to edit gifs with Python.

Using Pillow

For this example, I’ll take this gif:

And put this hat on him:

To start, we need to install Pillow:

pip install Pillow

and we also need to import the Image class from Pillow:

from PIL import Image

Now, we’ll want to get our images as Image objects. We can run the code below (with your own file path) to show the images and make sure everything is working:

The gif will only show the first frame, that’s normal. If nothing shows up, check your import statement and file path.

For now, we’ll work with just the first frame of the gif and swap over to the full gif once we have everything looking good on the first frame.

Because my hat image has a transparent background, I use the mask argument. If your image doesn’t skip that part.

And here’s what we get:

As you can see, the hat color isn’t correct but that’s okay! That will fix itself later.

Now that we have our images together. We just need to position the hat. We can do this by altering the arguments in the paste method:

gif.paste(hat, (375, -55), mask = hat)

Here I’ve added the amount of offset in the x and y directions as a tuple. The hat image will be offset by 375px on the x-axis and -55px (going up instead of down) in the y-axis.

Now we are set up on our images:

Now we just need to paste the hat in this position over each frame of the gif. We do this by importing the ImageSequence object from PIL and using it to grab all of the frames:

from PIL import Image, ImageSequence

In the code below, I copy each from the iterator which iterates over the gif frames. From there, I paste the hat on the frame and append that frame to the list of created frames.

Once the frames are made, we have to save the file to view the new gif:

new_gif[1].save(
    'gif_creation/new_gif.gif',
    save all = True, 
    appen_images = new_gift[1:],
    loop = 0
)

This method saves the first frame and appends all other frames of the gif. The loop argument allows us to either set a number of loops or use 0 for infinite loops.

Once saved we have a new gif:

And now for the full code:

Not too bad. At this point, the gifs could be generated and used as needed.

However, in my personal project, I was creating very large gifs which could not be uploaded to image hosting platforms.

Optimizing the Gif

So I had to go one step further and optimize the gifs. There are not many good options in Python for gif optimization. There is one excellent CLI tool that does just that called Gifsicle.

If you’re not familiar, CLI (command-line interface) tools are used in the command line. One example of a CLI tool would be Git.

In order to use Gifsicle, you’ll need to download it. All the download links can be found here.

Once downloaded, it’s extremely easy for us to run gifsicle commands in Python. We can simply run a subprocess with all of the items in the command line as a list.

For this particular example, we’ll reduce the size of the gif through lossy compression and limit the number of colors used. There are many other ways Gifsicle can be used to optimize gifs, ezgif.com uses it and shows some great examples of all the possibilities with the tool.

The following code runs optimization after the gif creation:

And we get out this gif:

As you can see, it has been compressed. The overall size of the gif dropped from 8.3MB to 3.1MB.

Conclusion

This is the basics of gif generation and optimization with Python. There is much more that can be done using just the methods shown in this example.

Creating gifs over other gifs and even moving the hat from frame to frame is relatively easy to do which allows for pretty much any type of gif generation.

Hopefully, this has saved you some time reading docs on these tools and gives you a head start on your gif generation journey.

If this helped you out, consider following me on twitter or here on Medium.

Sign up for our Free Weekly Newsletter. Fresh content from developers all over the world, fresh and direct from the field! Don’t forget to follow our publication The Dev Project.

Quick and Easy Gif Creation and Optimization with Python was originally published in The Dev Project on Medium, where people are continuing the conversation by highlighting and responding to this story.

Sentiment analysis can be as hard or as simple as you make it. Just like with any ML project you can pick the algorithm, train the model, and handle all of the issues that come along with it (hello, overfitting 😒).

I’ve been through that and I try to choose the easy road where possible. In the case of sentiment analysis, the work has been done. No need to reinvent the wheel.

So let’s have at it. We’ll only be using two libraries:

• Tweepy
• Flair

Flair is a NLP framework developed by Humboldt University of Berlin. They have models trained on social media data, which is perfect for our use here.

Tweepy is an open-source Python library for accessing the Twitter API.

Setup

We’ll need a few libraries installed to get this project off the ground:

pip install tweepy
pip install flair

Flair takes a while to install, it’s quite a large library. While it’s installing, head over to Twitter’s developer platform and sign up for an account.

Once you have an account, open the developer portal and make a new project and click the key icon next to it to get your API keys and authentication tokens.

We’ll only be using v1.1 endpoints, so no need to worry about getting v2 access or any elevated access. The most basic account will do.

You’ll need four keys for this project, they’ll be labeled:

• API Key
• API Secret
• Access Token
• Access Secret

Once you have those saved somewhere, our setup is done. We can dive into a Python file and start putting everything together.

Getting Tweets

Setting the API Handler

We’ll start by making sure we can get tweets from Twitter’s API. Let’s import tweepy and set those keys and tokens we generated:

import tweepy

API_Key = 'Your API Key'
API_Secret = 'Your API Secret'
Access_Token = 'Your Access Token'
Accesss_Secret = 'Your Access Secret'

Alright, now that we have our variables set, let’s set up an API handler. We can do this by using tweepy’s OAuthHandler class:

auth = tweepy.OAuthHandler(API_Key, Secret_Key)

With that new auth object, we want to use the set_access_token method to set our access tokens:

auth.set_access_token(Access_Token, Access_Secret)

Finally, we can set finish our API handler setup by creating a tweepy API class using that auth object:

api = tweepy.API(auth)

Let’s have a look at what our code should look like at this point:

Searching Tweets

The Twitter API is pretty robust, there are many ways to gather tweets from certain users, followers, etc. In this example, we’ll just search tweets. This works the same as using the search feature on Twitter.

To search tweet’s we’ll use our api object’s search_tweets method. This takes in a few different arguments but we’ll just set a query, language, count, and the tweet_mode:

tweets_data = api.search_tweets(
    q = 'test -RT',
    lang = 'en',
    count = 10,
    tweet_mode = 'extended'
)

If you noticed that our query has “-RT” in it, that is to exclude Retweets. That just makes things slightly simpler to work with.

That should be everything you need to search for tweets. Just print the tweets_data to make sure everything is up and running.

You should get a big list of all kinds of data. The Twitter API will send everything back, including all kinds of information we won’t use.

If you didn’t get that, you likely got an authentication error:

This means you’re having issues with your keys and tokens. Make sure they’re in the right spot. If they are, try regenerating them because there was likely an error when you copied them over.

Getting the Right Tweet Data

So now that we’ve got a bunch of data printed, we need to trim it down to just the data we want. For now, we’ll just grab the text to run sentiment on.

We can do this pretty easily. Our tweet search outputs a list of tweets and we can use an attribute in tweepy’s model to grab the text:

tweet_text = [tweet.full_text for tweet in tweets_data]

That will output only the text associated with the tweet. If we print out tweet_text we’ll get a list of tweets (as strings):

From here we can start analyzing the sentiment of these tweets.

Sentiment Analysis

Flair is a pretty large library. There are only a few pieces of the library we are going to use, so we’ll specify those imports to make our code as readable as possible:

from flair.models import TextClassifier
from flair.data import Sentence

The first thing we’ll need to do with flair is set up our classifier:

classifier = TextClassifier.load('sentiment')

Next, we’ll set up the sentences which will be used for analysis:

tweet_sent = [Sentence(tweet) for tweet in tweets]

Finally, we’ll use the classifier to predict each tweet sentiment:

classifier.predict(tweet_sent, mini_batch_size=32)

So just for a recap on where we are in our program, here’s a screenshot with the tweet_sent printed out:

As you can see, Flair gives us the tokenized sentence. What we are really after though, are the labels. Let’s separate out just the “POSITIVE” and “NEGATIVE” labels:

tweet_sent = [sent.labels[0].value for sent in tweet_sent]

So now we get a printout of just the label given to the tweet:

Now we have the tweets pulled and sentiment analyzed. We can simply combine the text with the sentiment.

analyzed_tweets = list(zip(tweet_text, sent_values))

Printing those out, we can see which tweets ranked positively and negatively.

Conclusion

So this covers a lot of ground, we’ve:

• Generated API keys and authorization tokens for Twitter
• Used tweepy to gather tweets
• Used Flair to analyze the sent

Just one more check on how much code that took:

Right at 33 lines of code. That is an incredibly succinct program.

This is just scraping the surface of what both tweepy and Flair can do. You could build applications to analyze the sentiment of responses to tweets, followers sentiment on a subject, or even analyze sentiment in real time with streamed tweets!

If this helped you out, the best way to support me is by following me on Twitter or here on Medium!

Feedback is my friend, so feel free to reach out and tell me that you liked my story, want some topic covered, or that some part of this could be done better.

Sign up for our Free Weekly Newsletter. Fresh content from developers all over the world, fresh and direct from the field! Don’t forget to follow our publication The Dev Project.

Powerful Twitter Sentiment Analysis in Under 35 Lines of Code was originally published in The Dev Project on Medium, where people are continuing the conversation by highlighting and responding to this story.

JavaScript is everywhere. If you find a website with no JavaScript on the page, you can bet it’s from the 1990’s.

That presents problems for web scraping. Most of the time, the data is right in the HTML of the page. It can be easily seen and scraped. However, there are times when the data is only available after the JavaScript is rendered.

That presents a challenge for web scraping. I’ve written past articles on web scraping which focus on easy-to-use Python libraries.

Unfortunately, that method breaks when you introduce JavaScript rendering into the mix.

So how do we handle websites that trap their data behind a JavaScript rendering? Well, we’re going to have to use some more advanced tools.

The Right Tool for the Job

Usually, I would recommend a couple of go-to libraries for web scraping:

• Requests
• BeautifulSoup

These two tools can do a whole lot, even if you’re going through several pages for data. They can’t be used to render JavaScript though.

There is a whole collection of tools that are built for this type of job. These tools are group into a category known as Browser Automation.

The goal of a browser automation tool is to simulate the web browsing experience but have it automated so that it can be run at intervals or speeds that a person couldn’t achieve.

These are touted more for their website testing capabilities for website owners. They also happen to have everything we need to render JavaScript and scrape the underlying data.

Some of the more popular tools in this category are:

• Scrapy
• Playwright
• Selenium

In this example, we’ll focus on using Selenium. We’ll also use our trusty library BeautifulSoup to parse the response.

Setting up the Workspace

Since we are fully automating a web browser, a little more setup is required than just a few pip installs. We’ll need a few other things installed:

• Chrome (or other web browser, we’ll be using Chrome in this example)
• ChromeDriver (web driver for Chrome)

Go ahead and install Chrome if you would like to follow along. For the ChromeDriver install, we’ll use a handy Python library that will do that for us.

With that said, let’s go ahead and install the libraries we’ll be using:

pip install selenium
pip install bs4
pip install chromedriver-autoinstaller

Once those are all installed, we can start importing:

import chromedriver_autoinstaller
from selenium import webdriver
from bs4 import BeautifulSoup

The chromedriver_autoinstaller library will handle installing ChromeDriver and adding it to PATH if it is not already there, which takes a bit of work off our plate. We can do that with one simple line:

chromedriver_autoinstaller.install()

That’s pretty much it for setting up our environment. Just to recap, we pip installed selenium, bs4, and chromedriver-autoinstaller. Our Python file should now look like this:

Getting The Webpage

With our environment set up, we can begin requesting web pages. To do that we need to set up our webdriver object which selenium will use:

driver = webdriver.Chrome()

And we can go ahead and tell the driver to fetch a web page. In this example, we’ll be scraping Rotten Tomatoes Certified Fresh Movies.

The data we are after (movie titles, ratings, etc.) can be found without rendering the JavaScript. However, parsing the data is much easier when it is rendered.

This page is rendered almost entirely with JavaScript, here is the site with JavaScript enabled:

And with JavaScript disabled:

We can request this web page by using our driver objects “get” method:

driver.get('https://www.rottentomatoes.com/browse/cf-dvd-streaming-all')

And we can get the html output using the page_source attribute:

html = driver.page_source

Just to recap, here is where we are at on our code:

Parsing the HTML

Now, Selenium can parse the data. We’ll use that in certain scenarios. However, BeautifulSoup will be our go-to for parsing the HTML. So let’s make a soup out of the page source:

soup = BeautifulSoup(html, 'html.parser')

Now, we should have something that looks like this:

If we were to print out this soup object that we’ve made. We get the full web page, minus some of the fancy formatting. Luckily, we don’t have to wait for the JavaScript to execute on this page.

In some cases, we will have to wait for JavaScript execution. Which can be done by either Implicitly Waiting or Explicitly Waiting.

Since we don’t have to worry about that, let’s find the information we’re looking for:

It looks like all the movies we are looking for are located inside their own div with the class “mb-movie”.

Each of these hold the information about the individual movie:

We can get at each one of them and get the title, score, and release date easily with BeautifulSoup:

Conclusion

So we have covered quite a lot in a short span of time. Here’s a recap of what we’ve done:

• Installed Chrome
• Installed ChromeDriver using a Python library
• Pulled a JavaScript heavy web page using Selenium
• Parsed and gathered data using BeautifulSoup

Here’s one final look at where we ended up, with the data printed out in the terminal:

If this helped you out, the best way to support me is by following me on Twitter or here on Medium!

Feedback is my friend, so feel free to reach out and tell me that you liked my story, want some topic covered, or that some part of this could be done better.

How to Scrape JavaScript Heavy Sites Like a Pro With Python was originally published in The Dev Project on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of the more difficult tasks when web scraping is dealing with hierarchical data. That is, data that lives on different pages.

If you’re looking for a basic way to get data from each page without manually going through them, look no further. We’ll be using two popular Python libraries to do just that:

• Requests
• BeautifulSoup

In this example, we’ll be using IMDb’s Top 250 Movies. You can follow along in this kaggle notebook!

Getting Each Movie Title

We’ll go over getting the basic information quickly since that is relatively simple. If you’d like more detail, I’ve covered this in a previous Medium article.

We can see that all of the movie information is in the main table body:

Each movie has a row in this table. Inside those rows there are columns. The column that holds the name of the movie has a class of “titleColumn”:

Now, getting the title of the movie is easy from here. We can simply use the requests library to get the page and iterate through the rows. For each row we can navigate to the title column, and grab the movie title inside the <a> tag:

Likewise, we can get the release year in the same column which is inside the <span> tag:

Diving Deeper

Let’s say we want some information about each movie that isn’t on the Top 250 list page. We want to add the genre of movie it is, which can be found on the individual movie page:

In the bottom left corner of the screenshot, we have the genre categories the movie belongs to. So we’ll need to go to each individual movie page to pull that information.

Luckily, the page is linked right in the main top 250 page. Clicking the movie title would send us here. If we open DevTools (click F12 while in the browser), we can see that this link is inside the <a> tag we used to get the movie title:

The link is in the “href” portion of the <a> tag. Luckily BeautifulSoup gives us just the tool to grab that portion of the tag:

link = item.find('a')['href']

It’s very similar to the notation used to slice on strings or lists. In this case, we can grab the data from different elements inside a tag.

Requesting Each Movie’s Page

In our case, we’ll have to get this for each of the movies in the list which will mean iterating over the rows we created earlier:

for row in rows:
    link = column.a['href']

And we can print out the links to each of the movies:

The links don’t have the long query string which we saw in DevTools, however it works the same and will take us to the correct page.

Now we just have to request that page, keep in mind that this isn’t the full link. We have to add on the domain portion:

for row in rows:
    link = column.a['href']
    movie_page = requests.get(f'https://imdb.com/{link}').content

Once we have the contents of the page, we need to parse it to get at the genre section:

for row in rows:
    link = column.a['href']
    movie_page = requests.get(f'https://imdb.com/{link}').content
    movie_soup = BeautifulSoup(movie_page, 'html.parser')

Finding the Movie Genres

Now we just have to figure out where the genre name is located on the page:

So each genre is inside an <a> tag with a long class name. This can be easily added to the row iteration:

for row in rows:
    link = column.a['href']
    movie_page = requests.get(f'https://imdb.com/{link}').content
    movie_soup = BeautifulSoup(movie_page, 'html.parser')
    genre = movie_soup.find('span', 'sc-16ede01-3 bYNgQ ipc-chip ipc-chip--on-baseAlt')

If we were to print out the genre at this point, we get something unexpected:

We only print out one genre per movie. Specifically, the first genre in the html.

Of course, we know a movie can have multiple which is shown in screenshots of The Godfather above. So we need to iterate through the genres. We can do this by using the “find_all” method and using some list comprehension:

And finally, we can attach it to the main movie names and release dates:

This is as deep as we’ll go for this example but you can go much more than two layers deep into the hierarchy using this method.

Conclusion

This has covered quite a lot, you should now be able to:

• Request a web page
• Pull information from that page
• Find other web pages that contain information you want
• Gather information from those pages

On pretty much any website. Let me know if you have any questions in the comments!

If this helped you out, consider following me on twitter for more daily programming tips and articles.

Hierarchical Web Scraping With Python was originally published in The Dev Project on Medium, where people are continuing the conversation by highlighting and responding to this story.

Scraping the internet for data can be one of the most challenging steps in a development project. It doesn’t have to be.

Here we will focus on scraping basic HTML pages using two popular Python libraries:

• Requests
• BeautifulSoup

Both of these libraries have ample documentation and plenty of issues resolved on StackOverflow, which make them easy to use and troubleshoot if problems arise.

In this example, we’ll be using IMDb’s Top 250 Movies. You can follow along in this kaggle notebook!

Installing the Libraries

These libraries will be installed using pip, the package installer for Python. Open your terminal or command prompt and run this:

pip install requests

pip install bs4

Once you’ve run both of those, you should have everything you need to begin web scraping.

Importing the Libraries

Once you have the libraries installed, open up python file in whichever editor you like (or open the kaggle notebook for this article) and run the following imports:

import requests
from bs4 import BeautifulSoup

Running Requests and Making a Soup

Let’s run a request to get the html from the page:

html = requests.get('https://www.imdb.com/chart/top/?ref_=nv_mv_250').content

If we were to print our html, we would see the raw html without any formatting. This is because we grabbed the response as bytes using the “content” attribute.

So we now need to parse the html, which is where BeautifulSoup comes in:

soup = BeautifulSoup(html, 'html.parser')

Now if we print out our soup object, we get parsed html:

Getting the Data

Now that we’ve collected the html page and parsed it, we can grab whatever data we would like to. This is done by using different methods and attributes on the soup object we created (we called this “soup”).

The easiest way to find elements in the html is through the use of DevTools in your browser. You can activate this by opening up the web page and pressing F12.

You can then hover over elements of the page to quickly find the ones that may be most useful. In this case, we can see the movie list is made up of one large table:

Getting Movie Titles

First, let’s get the title of each movie on the list. In our code we can narrow down the html we are looking at by only looking at the <tbody></tbody> portion, since we only care about the movies inside the table.

Now that we’ve narrowed down what we have to look at, it’s easier to tell where the data we want is. Each movie has its own row in that table:

We can simplify what we are looking at further by having our soup return the rows of the table as a list:

rows = soup.tbody.find_all('tr')

Now that we have a list of rows, we can print one row and find the data we want. The title can be found in a column with the class “titleColumn”:

So we will need to find a column with that class for each row. Now that we’ve created a list of rows, we’ll have to iterate over that list:

for row in rows:
    column = row.find('td', 'titleColumn')

If we were to print out the text of “column” which we just created, we would see extra information besides just the movie title:

This is because BeautifulSoup grabs all of the text within the title column. So we’ll have to narrow it down further.

The title is within the only <a> tag in the title column.

So we can easily pull the title by telling BeautifulSoup to get the text from the <a> tag:

for row in rows:
    column = row.find('td', 'titleColumn')
    title = column.a.text

If we print the titles, we get a neat list of all the titles on IMDb’s Top 250:

Getting Movie Release Years

We can also do this with the year by getting the text from the<span> tag.

Since this is in the same column as the title, the only difference is using column.span instead of column.a :

for row in rows:
    column = row.find('td', 'titleColumn')
    title = column.a.text
    year = column.span.text

Now we can print out the title of each movie along with the release year:

Of course, we can do a great deal more with these two tools. The user ratings and rank are easy to grab from here as well.

Conclusion

Just to reiterate, the steps we took to get this information was:

• Request a web page (using the requests library)
• Parse the HTML (using BeautifulSoup)
• Search through the parsed HTML for the piece of information we want

To dive a little deeper, check out how you can go into each movie page and get the genres for each movie using the same tools we used here!

If this helped you out, consider following me on twitter for more daily programming tips and articles.

Web Scraping With Python for Beginners was originally published in The Dev Project on Medium, where people are continuing the conversation by highlighting and responding to this story.