What is Web Scraping?
Before we move forward with this good agenda, let us take a helpful step and get to know what Web scraping is all about. After that, we shall all be in the same space as we look for resources to learn this beautiful technology.
Web Scraping in simple language is the extraction of data from websites. Since there are many websites out there, tools have been developed that go over the websites (web crawling) looking for specific pieces of data and collecting them automatically (Web Scraping). As you can guess, most of this data is being collected in an unstructured fashion in HTML form. This is later converted into structured data for example in a spreadsheet or some form of a database so that it can be utilised in various ways.
Such information in huge amounts can be pretty invaluable for companies that wish to know trends or organizations looking for particular information about something of their interest. This makes Web Scraping such a sought-after skill that the books we will present next intend to instruct. The books are listed in no particular order and they include the following:
1. Python Automation Cookbook
A little information about Jaime Buelta, the author, will suffice as an ice-breaker in this conversation about this resource. He is a full-time Python developer since 2010 and a regular speaker at PyCon Ireland. He has been a professional programmer for over two decades with rich exposure to a lot of different technologies throughout his career. He has accomplished much more including authoring this book. So what is it all about?
Briefly, this edition will enable you to develop a sharp understanding of the fundamentals required to automate business processes through real-world tasks, such as developing your first web scraping application, analyzing information to generate spreadsheet reports with graphs, and communicating with automatically generated emails.
Once you grasp the basics, you will acquire the practical knowledge to create stunning graphs and charts using Matplotlib, generate rich graphics with relevant information, automate marketing campaigns, build machine learning projects, and execute debugging techniques.
What you will learn
- Learn data wrangling with Python and Pandas for your data science and AI projects
- Automate tasks such as text classification, email filtering, and web scraping with Python
- Use Matplotlib to generate a variety of stunning graphs, charts, and maps
- Automate a range of report generation tasks, from sending SMS and email campaigns to creating templates, adding images in Word, and even encrypting PDFs
- Master web scraping and web crawling of popular file formats and directories with tools like Beautiful Soup
- Build cool projects such as a Telegram bot for your marketing campaign, a reader from a news RSS feed, and a machine learning model to classify emails to the correct department based on their content
- Create fire-and-forget automation tasks by writing cron jobs, log files, and regexes with Python scripting
If you are a developer, data enthusiast or anyone who wants to automate monotonous manual tasks related to business processes such as finance, sales, and HR, among others, then you should pick up this book. One thing which will be good to have is a working knowledge of Python so that you can get started smoothly. For you with sparked interest, find out more about Jaime’s work as well as order your copy from Amazon by clicking on the link below.
2. Practical Web Scraping for Data Science
Practical Web Scraping for Data Science was written by Seppe Vanden Broucke and Bart Baesens who are professors in the data area of profession. This book provides a complete and modern guide to web scraping, using Python as the programming language, without glossing over important details or best practices. Written with a data science audience in mind, the book explores both scraping and the larger context of web technologies in which it operates, to ensure full understanding.
Summary of What You Will Learn
- Leverage well-established best practices and commonly-used Python packages
- Handle today’s web, including JavaScript, cookies, and common web scraping mitigation techniques
- Understand the managerial and legal concerns regarding web scraping
This resource welcomes everyone with a strong interest in Web Scraping. Though it is skewed towards a data science-oriented audience that is probably already familiar with Python or another programming language or analytical toolkit, anyone unfamiliar with Python will appreciate a quick Python primer in Chapter 1 to catch up with the basics. Other guides for Python beginners have been pointed out therein as well. As you can see, it is an approachable resource that you can gain a wealth of skills and information from the two professors who have adept expertise in this subject. Click below to head over to Amazon to have a better look at the book. Remember to order yours today.
3. Web Scraping with Python
Ryan Mitchell, the author of Web Scraping with Python, is a Software Engineer at LinkeDrive in Boston, where she develops their API and data analysis tools. She is a graduate of Olin College of Engineering and is a Master’s degree student at Harvard University School of Extension Studies.
That said, the expanded edition of this practical book not only introduces you to web scraping but also serves as a comprehensive guide to scraping almost every type of data from the modern web.
Part I focuses on web scraping mechanics: using Python to request information from a web server, performing basic handling of the server’s response, and interacting with sites in an automated fashion.
Part II explores a variety of more specific tools and applications to fit any web scraping scenario you’re likely to encounter.
Good things you will take home
- Parse complicated HTML pages
- Develop crawlers with the Scrapy framework
- Learn methods to store data you scrape
- Read and extract data from documents
- Clean and normalize badly formatted data
- Read and write natural languages
- Crawl through forms and logins
- Scrape JavaScript and crawl through APIs
- Use and write image-to-text software
- Avoid scraping traps and bot blockers
- Use scrapers to test your website
From Mitchell, if you do not know any Python at all, this book might be a bit of a challenge. Please do not use it as an introductory Python text. Any basic Python knowledge is a good ground to start planting good seeds from this resource. Serve your interest in Web Scraping by clicking on the link below to get more information and to also have a chance of ordering your copy from Amazon.
4. Python for Data Analysis
Written by Wes McKinney, the creator of the Python pandas project, this book is a practical, modern introduction to data science tools in Python. Wes’ goal is to offer a guide to the parts of the Python programming language and its data-oriented library ecosystem and tools that will equip you to become an effective data analyst. It is ideal for analysts new to Python and for Python programmers new to data science and scientific computing. Data files and related material you will find in the book are available on GitHub.
What you will find inside:
- Use the IPython shell and Jupiter notebook for exploratory computing
- Learn basic and advanced features in NumPy (Numerical Python)
- Get started with data analysis tools in the Panda’s library
- Use flexible tools to load, clean, transform, merge, and reshape data
- Create informative visualizations with matplotlib
- Apply the panda’s group by the facility to slice, dice, and summarize datasets
- Analyze and manipulate regular and irregular time series data
- Learn how to solve real-world data analysis problems with thorough, detailed examples.
This hands-on guide is packed with practical case studies that show you how to solve a broad set of data analysis problems effectively. You will learn the latest versions of pandas, NumPy, IPython, and Jupiter in the process of becoming the best Data modeller and designer. Get this spiced-up resource to begin your Data Modelling career with the best.
5. Python Web Scraping Cookbook
Michael Heydt, the author of this resource is an independent consultant and specializes in social, mobile, analytics, and cloud technologies.
This book by Michael is a solution-focused book that will teach you techniques to develop high-performance scrapers and deal with crawlers, sitemaps, forms automation, Ajax-based sites, and caches. You will explore a number of real-world scenarios where every part of the development/product life cycle will be fully covered. You will not only develop the skills to design and develop reliable data flows, but also deploy your codebase to an AWS.
Right from extracting data from the websites to writing a sophisticated web crawler, the book’s independent recipes will be a godsend on the job. This book covers Python libraries, requests, and BeautifulSoup. You will learn about crawling, web spidering, working with AJAX websites, and paginated items.
What you will grasp
- Use a wide variety of tools to scrape any website and data-including BeautifulSoup, Scrapy, Selenium, and many more
- Master expression languages such as XPath, CSS, and regular expressions to extract web data
- Deal with scraping traps such as hidden form fields, throttling, pagination, and different status codes
- Build robust scraping pipelines with SQS and RabbitMQ
- Scrape assets such as images media and know what to do when Scraper fails to run
- Explore ETL techniques for building a customized crawler, and parser, and converting structured and unstructured data from websites
- Deploy and run your scraper as a service in AWS Elastic Container Service
In case you are a Python programmer, web administrator, security professional or someone who wants to perform web analytics, you will find this book relevant and useful. It should be noted that familiarity with Python and a basic understanding of web scraping would be useful to take full advantage of this book. It is all yours and waiting for you to order your copy from Amazon below.
6. Getting Structured Data from the Internet
Getting Structured Data from the Internet has been authored by Jay M. Patel who is a software developer with over 10 years of experience in data mining, web crawling/scraping, machine learning, and natural language processing (NLP) projects.
In this book, Jay teaches you how to use Python scripts to crawl through websites at scale and scrape data from HTML and JavaScript-enabled pages and convert it into structured data formats such as CSV, Excel, JSON, or load it into a SQL database of your choice.
Jay goes beyond the basics of web scraping and covers advanced topics such as natural language processing (NLP) and text analytics to extract names of people, places, email addresses, contact details, etc., from a page at a production scale using distributed big data techniques on an Amazon Web Services (AWS)-based cloud infrastructure. It covers developing a robust data processing and ingestion pipeline on the Common Crawl corpus, containing petabytes of data publicly available and a web crawl data set available on AWS’s registry of open data.
As you go through this book, you will:
- Understand web scraping, its applications/uses, and how to avoid web scraping by hitting publicly available rest API endpoints to directly get data
- Develop a web scraper and crawler from scratch using lxml and BeautifulSoup library, and learn about scraping from JavaScript-enabled pages using Selenium
- Use AWS-based cloud computing with EC2, S3, Athena, SQS, and SNS to analyze, extract, and store useful insights from crawled pages
- Use SQL language on PostgreSQL running on Amazon Relational Database Service (RDS) and SQLite using sqlalchemy
- Review sci-kit learn, Gensim, and spaCy to perform NLP tasks on scraped web pages such as name entity recognition, topic clustering (Kmeans, Agglomerative Clustering), topic modelling (LDA, NMF, LSI), topic classification (naive Bayes, Gradient Boosting Classifier) and text similarity (cosine distance-based nearest neighbours)
- Handle web archival file formats and explore Common Crawl open data on AWS
- Illustrate practical applications for web crawl data by building a similar website tool and a technology profiler similar to builtwith.com
- Write scripts to create a backlinks database on a web scale similar to Ahrefs.com, Moz.com, Majestic.com, etc., for search engine optimization (SEO), competitor research, and determining website domain authority and ranking