What’s New in Firefox 147?

Firefoxbrowserweb scraping       / 4 min read

Firefox 147 is a fairly quiet release for web scraping, but there are a few changes that affect how automated browsing behaves, how servers see your requests and how Firefox behaves on Linux.

The official documentation for version 147 can be found in the release notes and release notes for developers.

Network Access Restrictions (ETP Strict)

Users with Enhanced Tracking Protection (ETP) set to Strict will have local network access restrictions enabled by default. Firefox will now require users to explicitly allow public websites to access local network resources.

This change affects how public websites interact with your private (local) network. With strict ETP Firefox now blocks websites from scanning or connecting to devices on your local network unless you give them explicit permission. Local network resources include any device connected to your Wi-Fi or Ethernet that isn’t on the public internet. This might be

  • smart home gear (security cameras, smart bulbs and thermostats)
  • office equipment (printers and scanners) or
  • media devices (smart TVs, Sonos speakers and Chromecast).

In the past a public website could try to communicate with your local devices. From the perspective of web scraping this was mostly a concern because it could be used for fingerprinting.

Dialog for selecting ETP (Enhanced Tracking Protection) level in Firefox.

Now with strict ETP when a site wants to talk to a local device (for example, a web-based setup tool for a new printer), Firefox will show a popup asking for permission.

This new security feature may have other implications for web scraping. If you’re driving Firefox interactively or via automation and your target site tries to reach private network resources, you may see permission prompts or hard failures. Of course, this will only be a thing if you are using strict ETP.

Accept-Language Changes and Fingerprinting

Firefox now uses the same quality values (q-values) in Accept-Language headers as other major browsers. The second language preference is now sent as q=0.9 instead of q=0.5, with subsequent language preferences decreasing by 0.1 each (minimum 0.1). This change fixes compatibility issues with some servers that incorrectly rejected requests with lower quality values.

For scrapers, this has two implications:

  • if you set Accept-Language, align your header with the new values to remain below the radar; and
  • if you rely on language negotiation for content variants then the priority ordering may shift.

Linux and Desktop Behavior

Firefox now supports the Freedesktop.org XDG Base Directory Specification.

Perhaps not directly relevant to web scraping but it’s useful to understand how Firefox persists user data. Suppose, for illustration, that I were using a profile called datawookie then I’d launch Firefox with

firefox -P "datawookie"

Old Directory Structure

In older versions of Firefox profile was stored under ~/.mozilla/firefox/.

Each profile had its own sub-direcrory. For example, the datawookie profile data might be found in ~/.mozilla/firefox/1xjxywwb.datawookie/. The random 8 character prefix to the directory name is “salt”, making it slightly harder for malicious code to locate. If there were other profiles then each would have its own directory named using the same convention.

The profile directory would contain files like

  • prefs.js — user preferences;
  • cookies.sqlite — cookie database; and
  • places.sqlite — browsing history and bookmarks database.

There would be a bunch of other files too, all dumped into the same location.

If you’re curious then the cookies.sqlite and places.sqlite are quite fun to explore. You will likely find that the cookies database is locked while the browser is running, so shut down Firefox if you want to explore that.

The browser also maintains cached data, which would be found under ~/.cache/mozilla/firefox/. Again there would be a separate sub-directory for each profile, using the same profile directory name. For example, the cache for the datawookie profile would be found in ~/.cache/mozilla/firefox/1xjxywwb.datawookie/.

Thunderbird, the email client from Mozilla, stores its configuration data under ~/.thunderbird/. It uses the same approach to naming sub-directories for individual profiles. The problem with this approach (individual applications each having a separate configuration folder in your home directory) is that things get cluttered and it can be hard to find what you’re looking for among the profusion of files and directories.

New Directory Structure

The goal of the XDG Base Directory Specification is to provide a clean and consistent directory structure for application files. Files should be stored under application specific sub-directories in three logical locations:

  • ~/.config/ — configuration (settings and preferences);
  • ~/.local/share/ — application data (databases and user content);
  • ~/.local/state/ — state data (logs, history, undo data and recently used files); and
  • ~/.cache/ — cache (temporary data that can be regenerated).

These locations can be changed using the XDG_CONFIG_HOME, XDG_DATA_HOME, XDG_STATE_HOME and XDG_CACHE_HOME environment variables respectively. But you’d need a good reason to mess with the conventional locations.

The latest version of Firefox will use this directory structure for storing data in newly created profiles. For example, configuration data would be stored under ~/.config/mozilla/firefox/ and application data under ~/.local/share/mozilla/firefox/. Each of these would have sub-directories for individual profiles.