Web scraping with JS

If you’ll try to google “web scraping tutorial” you’ll get a bunch of tech articles on the subject that tells you how to achieve the result using python. The toolkit is pretty standard for these posts: python 3 (hopefully not second) as an engine, requests library for fetching, and Beautiful Soup 4 (which is 6 years old) for web parsing.

I’ve also seen few articles where they teach you how to parse HTML content with regular expressions, spoiler: don’t do this.

The problem is that I’ve seen articles like this 5 years ago and this stack hasn’t mostly changed. And more importantly, the solution is not native to javascript developers. If you would like to use technologies you are more familiar with, like ES2020, node, and browser APIs you will miss the direct guidance.

I’ve tried to fill the spot and create ‘the missing doc’.

Overview

Check if data is available in request

Before you will start to do any of the programming, always check for the easiest available way. In our case, it would be a direct network request for data.

Open developer tools - F12 in most browsers - then switch to the Network tab and reload the page

If data is not baked in the HTML like it is in half of the modern web applications, there is a good chance that you don’t need to scrape and parse at all.

If you are not so lucky and still need to do the scraping, here is the general overview of the process:

1. fetch the page with the required data
2. extract the data from the page markup to some in-language structure (Object, Array, Set)
3. process the data: filter it, transform it to your needs, prepare it for the future usage
4. save the data: write it to the database or dump it to the filesystem

That would be the easiest case for parsing, in sophisticated ones you can bump into some pagination, link navigation, dealing with bot protection (captcha), and even real-time site interaction. But all this wouldn’t be covered in the current guide, sorry.

Fetching

As an example of this guide, we will scrape a goal data for Messi from Transfermarkt. You can check his stats on the site. To load the page from the node environment you will need to use your favorite request library. You can also use the raw HTTP/S module, but it doesn’t even support async, so I’ve picked node-fetch for this task. Your code will look something like:

const fetch = require('node-fetch')

const DATA_URL =
  'https://www.transfermarkt.com/lionel-messi/alletore/spieler/28003/plus/1'
const loadMessiGoals = async () => {
  const response = await fetch(DATA_URL)
  return response.text()
}

module.exports = {loadMessiGoals}

Tools for parsing

There are two major alternatives for this task which are conveniently represented with two high-quality most-starred and alive libraries.

The first approach is just to build a syntax tree from a markup text and then navigate it with familiar browser-like syntax. This one is fully covered with cheerio that declared as jQuery for server (IMO, they need to revise their marketing vibes for 2020).

The second way is to build the whole browser DOM but without a browser itself. We can do this with wonderful jsdom which is node.js implementation of many web standards.

Let’s take a closer look at both of them.

cheerio

Despite these analogies cheerio doesn’t have jQuery in dependencies, it just tries to reimplement most known methods from scratch:

 "dependencies": {
  "@types/node": "^14.11.2",
  "css-select": "~3.1.0",
  "dom-serializer": "~1.1.0",
  "entities": "~2.1.0",
  "htmlparser2": "^5.0.0",
  "lodash": "^4.17.20",
  "parse5": "^6.0.0",
  "parse5-htmlparser2-tree-adapter": "^6.0.0"
}

Basic usage is really easy:

• you load a HTML

  const $ = cheerio.load('<ul id="steps">...</ul>')

• done, you’re great, now you can use JQ selectors/methods

  $('li .main').text()

Probably you can pick this one if you need to save on size (cheerio is lightweight and fast) or you are really familiar with jQuery syntax and for some reason want to bring it to your new project. Cheerio is a nice way to do any kind of work with HTML you need in your application.

jsdom

This one is a bit more complicated: it tries to emulate part of the whole browser that is working with HTML and JS (apart from rendering the result). It’s used heavily for testing and … well scraping.

Let’s spin up jsdom:

• you need to use a constructor with your HTML

  const dom = new JSDOM('<!DOCTYPE html><p>Hello world</p>')

• then you can access standard browser API

  dom.window.document.querySelector("p").textContent

jsdom is a lot heavier and it does a lot more job. You should understand, why to choose it over other options.

Parsing

In our example, I want to stick to the jsdom. It will help us to show one last approach at the end of the article. The parsing part is really vital but very short.

So we’ll start with building a dom from the fetched HTML:

const dom = new JSDOM(goalsHTML)

Then you can select table content with css selector and browser API. Don’t forget to create a real array from NodeList that querySelectorAll returns.

const rows = Array.from(
  dom.window.document.querySelectorAll('.responsive-table tbody tr')
)

Now you have a two-dimensional array to work with. This part is finished, now you need to process this data to get clean and ready-to-work-with stats.

Processing

First, let’s check the lengths of our rows. Each row is the stat about goal and we mostly don’t care how many do we have. But each row can contain numbers in a different format so we have to deal with them.

We map over rows and get the length. Then we deduplicate results to see what options do we have here.

const lengths = rows.map(row => row.length)
const result = [...new Set(lengths)]
// [1, 5, 14, 15]

Not that bad, only 4 different shapes: with 1, 5, 14, and 15 cells.

Since we don’t need rank data from extra cell in 15-cells case it is safe to delete it.

const omitRank = (row) =>
  row.length === 15 ? [...row.slice(0, 5), ...row.slice(6)] : row

Row with only one cell is actually useless: it is just a name of the season, so we will skip it.

if (row.length === 1) return null

For the 5-cells case (when player scored few goals in one match) we need to find previous full row and use it’s data for empty stats.

const getLastMatch = (idx, goals) =>
  goals[idx].length === 14 ? goals[idx] : getLastMatch(idx - 1, goals)

const match = getLastMatch(idx, goals)
const isSameMatch = row.length === 14

Now we just have to manually map data to keys, nothing scientific here and no smart way to avoid it.

return {
  competition: match[1],
  matchday: match[2],
  date: match[3],
  venue: match[4],
  opponent: match[7],
  result: match[8],
  position: match[9],
  minute: row[1 + isSameMatch * 9],
  atScore: row[2 + isSameMatch * 9],
  goalType: row[3 + isSameMatch * 9],
  assist: row[4 + isSameMatch * 9],
}

Saving

We would just dump our result to a file, converting it to a string first with the JSON.stringify method.

const fs = require('fs').promises

const saveStats = async (data) => {
  fs.writeFile('./data.json', JSON.stringify(data))
}

Bonus: One-time parsing with a snippet

Since we used jsdom with the browser-compatible API we actually don’t need any node environment to parse the data. If we just need it once from a particular page we can just run some code at the Console tab in the developers tools of your browser. Try to open any player stats on Transfermarkt and paste this giant non-readable snippet to the console:

let data = Array.from(
  document.querySelectorAll('.responsive-table table tbody tr')
).map(
  row => Array.from(row.children).map(node => node.textContent.trim())
).map(
  (row) => row.length === 15 ? [...row.slice(0, 5), ...row.slice(6)] : row
).map(
  (row, idx, goals) => {
  if (row.length === 1) return nul

  const getLastMatch = (idx, goals) =>
    goals[idx].length === 14 ? goals[idx] : getLastMatch(idx - 1, goals
  const match = getLastMatch(idx, goals
  const isSameMatch = row.length === 14

  return {
    competition: match[1],
    matchday: match[2],
    date: match[3],
    venue: match[4],
    opponent: match[7],
    result: match[8],
    position: match[9],
    minute: row[1 + isSameMatch * 9],
    atScore: row[2 + isSameMatch * 9],
    goalType: row[3 + isSameMatch * 9],
    assist: row[4 + isSameMatch * 9],
  }}
).filter(Boolean)

And now just apply this magic copy function that is integrated into the browser devtools. It would copy data to your clipboard.

copy(data)

Not that hard, right? And no need to deal with pip anymore. I hope you found this article useful. Stay tuned, next time we will visualize this scraped data with modern JS libs.

You can find whole script for this article in the following codesandbox:

by Pavel Prokudin. I write about web-development and modern technologies. Follow me on Twitter