Introduction to HTML

6.1. Introduction to HTML#

6.1.1. Introduction#

In this chapter, we will learn about web scraping, one of the more vital skills of a digital humanist. Web scraping is process by which we automate the calling of a server (which hosts a website) and parsing that request which is an HTML file. HTML stands for HyperText Markup Language. It is the way in which websites are structured. When we scrape a website, we write rules for extracting pieces of information from it based on how that data is structured within the HTML. To be competent at web scraping, therefore, one must be able to understand and parse HTML.

In this section, we will break down HTML and you will learn the most common tags, such as div, p, strong, and span. You will also learn about attributes within these tags, such as href, class, and id. It is vital that you understand this before moving onto the next chapter in which we learn how to web scrape with Python.

6.1.2. Diving into HTML#

So why is HTML useful? HTML, like other markup languages, such as XML, or eXstensible Markup Language, allows users to structure data within data. This is achieved by what are known as tags. I think It is best to see what this looks like in practice. Let’s examine a simple HTML file.

<div>
    <p>This is a paragraph</p>
</div>

Above, we see a very simple HTML file structure. In the first line of this HTML, we see <div>. Note the use of < and >. The opening < indicates the start of a tag in HTML. A tag is a way of denoting structure within an HTML file. It is a way of saying that what comes after this nested bit of code is this type of data. After the <, we see the word div. This word denotes the type of tag that we are using. In this case, we are creating a div tag. This is one of the most common types of tags in HTML. After the tag’s name, we see >. This identifies the end of the tag creation.

In line 2, we see a nested, or indented bit of HTML. In HTML, unlike Python, indentation is optional. It is, however, good practice to use line breaks and indentation in HTML to make the document easier for humans to parse. Line two begins with a p tag. The p tag in HTML is used to denote the start of a paragraph.

After the creation of the p tag, we see This is a paragraph. In HTML, text that lies outside of the tags is text that appears on the web page. In this case, the HTML file would display the text This is a paragraph. Immediately after this bit of text we encounter our first close tag. A close tag in HTML indicates that this nested bit of structure is over. In our case, the first close tag is </p>. We know that it is a close tag because of the </, as opposed to <.

In our final line, we see a close div tag.

6.1.3. Understanding Attributes#

Let’s take a look at another block of HTML. This time, we will make one small change. Can you spot it?

<div class="content">
    <p>This is a paragraph</p>
</div>

If you said the class="content" portion of the open div tag, then you would be right. This bit is nested within the tag and is known as an attribute. In our case, the special attribute used is a class attribute (a very common attribute type). This attribute has a value of content.

When scraping websites, you can use these attributes to specify which div tag to grab. There are several common attributes, specifically class and id.

6.1.4. Parsing HTML with BeautifulSoup#

Now that we understand a bit about HTML, let’s start trying to parse it in Python. When we parse HTML, we attempt to automate the identification of HTML’s structure and systematically interpret it. This is the basis for web scraping. To begin, let’s create a simple HTML file in memory.

html = """
<html>
    <body>
        <div class="content">
            <p>This is our first content</p>
        </div>
        <div class="other">
            <p>This is is another piece of content</p>
        </div>
    </body>
</html>
"""

There are many Python libraries available to you for parsing HTML data. For more robust problems, Selinium is the industry standard. While Selinium is powerful, it has a steep learning curve and can be challenging for those new to Python, especially for those programming on Windows. Additionally, most web scraping problems can be solved without Selinium.

For those reasons, we will not be using Selinium in this textbook, rather BeautifulSoup. BeautifulSoup is a light-weight Python library for parsing HTML. It is quick and effective. The most challenging thing about BeautifulSoup is remembering how to install it and import it in Python.

To install BeautifulSoup, you must run the following command in your terminal:

pip install beautifulsoup4

Once it is installed, you can then import BeautifulSoup in the following way:

from bs4 import BeautifulSoup

With BeautifulSoup imported, we can use the BeautifulSoup class. This class allows us to parse HTML. As we will see throughout this chapter, you will rarely have HTML as a string within your Python script, but for now, since we are starting, let’s try to parse the above html string by passing it to the BeautifulSoup class. It is Pythonic to name your BeautifulSoup variable soup. If you have a more complex script that is parsing soup objects from multiple websites, you may want to be more original in your naming conventions, but for our purposes,soup will serve us well.

soup = BeautifulSoup(html)

With our soup object created in memory, we can now begin to examine it. If we print it off, we won’t notice anything special about it. It appears as a regular string.

print(soup)

<html>
<body>
<div class="content">
<p>This is our first content</p>
</div>
<div class="other">
<p>This is is another piece of content</p>
</div>
</body>
</html>

While it may look like a string, it is not. We can observe this by asking Python what type of object it is with the type function

type(soup)

bs4.BeautifulSoup

As we can see, this is a bs4.BeautifulSoup class. That means that while it may look like a string, it actually contains more data that can be accessed. For example, We can use the .find method to find specific the first occurrence of a specific tag. The find method has one mandatory argument, a string that will be the tag name that you wish to extract. Let’s grab the first div tag.

first_div = soup.find("div")
print(first_div)

<div class="content">
<p>This is our first content</p>
</div>

As you can see, we were able to grab the first div tag with .find, but we know that there are multiple div tags in our HTML string. What if we wanted to grab all of them? For that, we can use the .find_all method. Like .find, .find_all takes a single mandatory argument. Again, it is the string of the tag name you wish to extract.

all_divs = soup.find_all("div")
print(all_divs)

[<div class="content">
<p>This is our first content</p>
</div>, <div class="other">
<p>This is is another piece of content</p>
</div>]

Unlike .find which returns a single item to us, the .find_all method returns a list of tags. What if we did not want all tags? What if we only wanted to grab the div tags with a specific class attribute. BeautifulSoup allows us to do this by passing a second argument to .find or .find_all. This argument will be a dictionary whose keys will be the attributes and whose values will be the attribute names of the tags you wish to extract.

div_other = soup.find_all("div", {"class": "other"})
print(div_other)

[<div class="other">
<p>This is is another piece of content</p>
</div>]

Once we have obtained the specific tag that we want to extract from the HTML, we can then access its nested components. Each bs4.BeautifulSoup object functions the same way as the original soup. It contains all children (nested) tags. We can, therefore, grab the p tag embedded within the div tag that has a class attribute of other by using .find("p").

paragraph = div_other[0].find("p")
print(paragraph)

<p>This is is another piece of content</p>

If we are working with textual data, though, we rarely want to retain the HTML, rather we want to extract the text within the HTML. For this, we can access the raw text that falls within the HTML with .text.

print(paragraph.text)

This is is another piece of content

Being able to do this programatically means that we can automate the scraping of HTML files via Python. We can, for example, extract all the text from each div tag in our file via the following two lines of code.

for div in all_divs:
    print(div.text)

This is our first content

This is is another piece of content

6.1.5. How to Find a Website’s HTML#

Now that you are familiar generally with HTML and how it works, let’s dive in and take a look at some real-world HTML from a real website. In the next chapter, we will web scrape Wikipedia, so let’s go ahead and focus on Wikipedia here as well. If you are using a web browser that supports it (Chrome and Firefox), you can enter developer mode. Each operating system and browser has a different set of hotkeys to do this, but on all you can right click the webpage and click “inspect”. This will open up developer mode. At this point in the chapter, I highly recommend switching over to the video as it will be a bit easier to follow along.

For this chapter (and the next), we will be working specifically with this page: https://en.wikipedia.org/wiki/List_of_French_monarchs

Go ahead and open it up on another screen or in a new tab. This Wikipedia article contains some text, but primarily it hosts a list of all French monarchs, from the Carolingians up through the mid-19th century with Napoleon III.

When you inspect this page, you will see all the nested HTML within it. Spend some time and go through these tags. In the next section, we will learn how to scrape this page, but for now take a look at what we can do with a few basic commands in Python.

import requests
from bs4 import BeautifulSoup

url = "https://en.wikipedia.org/wiki/List_of_French_monarchs"
s = requests.get(url)

soup = BeautifulSoup(s.content)
body = soup.find("div", {"id": "mw-content-text"})
for paragraph in body.find_all("p")[:5]:
    if paragraph.text.strip() != "":
        print (paragraph.text)

The monarchs of the Kingdom of France ruled from the establishment of the Kingdom of the West Franks in 843 until the fall of the Second French Empire in 1870, with several interruptions. Between the period from King Charles the Bald in 843 to King Louis XVI in 1792, France had 45 kings. Adding the 7 emperors and kings after the French Revolution, this comes to a total of 52 monarchs of France.

In August 843 the Treaty of Verdun divided the Frankish realm into three kingdoms, one of which (Middle Francia) was short-lived; the other two evolved into France (West Francia) and, eventually, Germany (East Francia). By this time, the eastern and western parts of the land had already developed different languages and cultures.

Initially, the kingdom was ruled primarily by two dynasties, the Carolingians and the Robertians, who alternated rule from 843 until 987, when Hugh Capet, the progenitor of the Capetian dynasty, took the throne.  The kings use the title "King of the Franks" until the late twelfth century; the first to adopt the title of "King of France" was Philip II (r. 1180–1223).  The Capetians ruled continuously from 987 to 1792 and again from 1814 to 1848. The branches of the dynasty which ruled after 1328, however, are generally given the specific branch names of Valois (until 1589), Bourbon (from 1589 until 1792 and from 1814 until 1830), and the Orléans (from 1830 until 1848).

In the cell above, we used two libraries, requests and BeautifulSoup to call the Wikipedia server that hosts that particular page. We then searched for the div tag that contains the main body of the page. In this case, it was a div tag whose attribute “id” corresponded to “mw-content-text”. I thin searched for all of the “p” tags, or paragraphs within that body and printed off the the text if the text was not blank. By the end of the next chapter, you will not only understand the code above, but you will be able to write it and code it out yourself. For now, inspect that page and see if you can find where the div tag whose id corresponds to “mw-content-text” is located in the HTML. It is okay if this is hard! It is not something that you naturally can do quickly. It takes practice. A trick to help get you started, however, is to right click the area that you want to scrape and then click inspect.

Once you feel comfortable with this, feel free to move on to the next chapter to start learning how to web scrape!