My previous post about the Massive proxy service looked at the geographic distribution of their residential proxy exit nodes. The nodes were selected according to Massive’s rotation algorithm and I found a broad distribution of exit IPs across many countries, regions and cities.
Now we’re looking at this from a different perspective: rather than granting the algorithm carte blanche to select exit locations, we’re going to ask it to restrict itself to a specific spatial region. This is geotargeting.
I’ll be introducing a few key principles from the official documentation and then kicking the tyres.
How Geotargeting Works
Massive allows you to narrow down the pool of potential proxy exit locations using these parameters:
country— using the two-letter country code (mandatory);subdivision— a first-level subdivision inside that country;city; andzipcode— for a postal code.
If you specify more than one parameter then Massive looks for an exit node that strictly matches all of the specified parameters. However, zipcode is an exception: if it’s present then city is ignored. Another quirk for postal codes is that for some countries (like US and UK) partial matching is supported. For example, if you specify SW1E for the UK, then any exit node with a postal code starting with SW1E will be considered a match.
TipWhat is a Subdivision?
A subdivision is a first-level administrative division of a country. The standard for subdivision codes is ISO 3166-2. The codes consist of two parts: the country code, a separator and then a subdivision code.
The UK’s top-level subdivisions are England (GB-ENG), Scotland (GB-SCT), Wales (GB-WLS) and Northern Ireland (GB-NIR). Each state in the US is a subdivision. For example, California (US-CA), Florida (US-FL), New York (US-NY), Texas (US-TX) and Washington (US-WA). The ISO 3166-2 subdivision codes are broken down by country in this JSON file.
To specify one or more parameters you append them to your proxy username. This is an interesting implementation choice. I would have assumed that headers or query parameters would be the obvious way to go. But it works. And it works well! For further details see the Massive geotargeting documentation.
Here are some examples (****** is a placeholder for the actual username):
******-country-GB # Locations in Great Britain.
******-country-GB-subdivision-ENG # Locations in England.
******-country-GB-city-London # Locations in London.
******-country-GB-zipcode-SW1E # Locations with SW1E postal code prefix.So, for example, a request might look like this:
USERNAME="******-country-GB"
PASSWORD="******"
curl -x "http://network.joinmassive.com:65534" -U "$USERNAME:$PASSWORD" https://lnkd.in
Warning
If you have a city name with spaces in it, then you need to URL-encode the spaces.
For example, you’d use ******-country-GB-city-New%20York for New York.
It’s a good principle to start geotargeting on a broad area and then narrow down if necessary. As a web scraper I generally find that geographic restrictions are made at the country or continental level. So, for example, start by restricting to the US. If you need more precision then select a specific state (as a subdivision). Need to be even more granular? For example, a retailer might localise stock availability by city. Then select a city. And if you really need to be specific, then try a postal code.
The narrower the target, the smaller the pool of eligible exit nodes. Avoid unnecessary constraints. If you demand a small target area, you should expect either reduced IP rotation, more capacity misses or both.
Smoke Tests
To evaluate the performance of geotargeting I generated samples using geotargeting filters for the UK and US. I started by filtering at the country level and then proceeded to become progressively more granular by using the subdivision, city and postal code filters. At each level of granularity I collected a sample and used https://ip-api.com/ to record the observed exit locations.
UK Geotargeting
The UK sample data are saved in country, subdivision, city and postal code JSONL files. The country run produced 1,000 geolocated responses. The map below plots the locations of the exit points, which are all correctly located in the UK.
Next I narrowed my focus on England (subdivision GB-ENG). The locations of the 1,000 geolocated exit points are plotted on the map below. They are mostly located in England. The sample breaks down into 975 in England, 11 in Scotland, 11 in Wales and 3 in Northern Ireland. Not too bad!
When targeting London only 88.0% of the 1,000 geolocated responses were in the requested city, with 16 specific locations, 124 distinct IP addresses and 18 providers.
Finally for the SW1E postal code prefix, 95.5% of the 1,000 geolocated responses were in the requested postal code. The exit locations collapsed to just 4 coordinate pairs and 57 distinct IP addresses across 8 providers.
WarningGeolocation
I’ve been using http://ip-api.com/ to get the geolocation data for the exit nodes. The results presented above are predicated on the accuracy of that service.
A geolocation service looks up an IP address in an extensive database of IP-to-location mappings. The quality of the location data is only as good as the underlying database. The database is built using data from various sources (IP ownership records, routing data, ISP metadata and active network measurements), none of which are perfect. Typically they are very accurate at the country level, reasonably good at the city level and much less reliable when you drill down to postal codes. The accuracy also varies by country, with some being better covered than others.
The geolocation results should thus be treated as indicative rather than definitive. They are a useful proxy for the actual exit node locations, but they are not a perfect record.
US Geotargeting
I then repeated the same exercise with US geotargeting. The samples are saved in country, subdivision, city and postcode JSONL files. The country run produced 1,000 geolocated responses. The map below plots the locations of the exit points, which are all located in the US.
Focusing on Illinois (subdivision US-IL), the locations of the 1,000 geolocated exit points are plotted on the map below. They are mostly located in Illinois. The sample breaks down into 981 in Illinois, 6 in Indiana, 4 in Missouri, 4 in Wisconsin, 3 in Michigan, 1 in South Dakota and 1 in New York. Also not too bad!
Targeting Chicago, 95.9% of the 1,000 geolocated responses were reported in the requested city. There were 78 coordinate pairs, 638 distinct IP addresses and 12 providers, which seems reasonable for city-level targeting.
ZIP code 60657 is located in Cook County in the Chicago metropolitan area. According to https://zipcodes.org/ this area has a population of around 66,000 people distributed across about 41,000 households, with a median age of 35. The sample targeting this ZIP code had 71.6% of the 1,000 responses in the requested postcode, with only 4 coordinate pairs and 10 distinct IP addresses across 5 providers. This doesn’t seem as good as the UK postal code targeting, but the comparison is not entirely fair because SW1E is a postal code prefix. Massive uses partial matching, so any exit node with a postal code starting with SW1E is considered a match. The 60657 ZIP code is more specific, so it doesn’t benefit from partial matching and the pool of options is naturally smaller.
Summary
Massive’s geotargeting seems solid. It doesn’t promise geographic control and then quietly wave in the general direction of a map. It delivers results that are generally consistent with what you ask for.
At the country level it does exactly what you’d hope: broad, reliable placement with a healthy-looking pool of exits. It’s also good at the subdivision level. City targeting is still useful, giving results that are concentrated enough to be meaningful without collapsing into a tiny, brittle set of exit points. The shine dulls a little for postal code targeting though. That’s not a disaster, and it is not unusual, but it’s a reminder that geotargeting (and geolocation) gets less convincing the more precisely you ask it to behave.
The geolocation data for the exit nodes is not perfect, but it is good enough to establish that geotargeting is working as intended. Massive is very good at getting you into the right country, good at getting you into roughly the right city, but occasionally a bit optimistic about your exact patch of pavement.