Unpacking cURL Commands

cURL is the ultimate Swiss Army Knife for interacting with network protocols. But to be honest, I really only scratch the surface of what’s possible. Usually my workflow is something like this:

1. Copy a cURL command from my browser’s Developer Tools.
2. Test out the cURL command in a terminal.
3. Convert the cURL command into a programming language (normally Python or R).
4. Prosper.

I’m going to take a look at my favourite online tool for converting a cURL command to code and then see what other tools there, focusing on Python and R as target languages.

Web Tools

The quickest way to convert a cURL command into code is using a tool like curlconverter, which is a transpiler that translates a cURL command in a variety of other languages (Ansible, C, C#, ColdFusion, Clojure, Dart, Elixir, Go, HAR, HTTP, HTTPie, Java, JavaScript, Julia, JSON, Kotlin, Lua, MATLAB, Node.js, Objective-C, OCaml, Perl, PHP, PowerShell, Python, R, Ruby, Rust, Swift and Wget).

The curlconverter web tool with a cURL input box and generated Python requests code.

It’s an Open Source project and you can dig around in the repository.

There are other similar services (like this) but IMHO curlconverter is the best.

R Tools

The {httr2} package has a curl_translate() function that will convert cURL to R. First let’s load the package.

library(httr2)

Now try it out with the simplest cURL command.

curl_translate("curl https://example.com")

request("https://example.com") |> 
  req_perform()

A little underwhelming, right? But it should do precisely what we expect: get the content from that URL. Let’s check.

response <- request("https://example.com") |> 
  req_perform()

response$status_code

[1] 200

Nice. Now a somewhat more complicated cURL command. Stash the command in a variable.

CURL <- "curl 'https://example.com/' -H 'User-Agent: Mozilla/5.0' -H 'Accept: text/html,application/xml;q=0.9,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br, zstd' -H 'Connection: keep-alive' -H 'Upgrade-Insecure-Requests: 1' -H 'Sec-Fetch-Dest: document' -H 'Sec-Fetch-Mode: navigate' -H 'Sec-Fetch-Site: none' -H 'Sec-Fetch-User: ?1' -H 'Priority: u=0, i'"

Now translate.

curl_translate(CURL)

request("https://example.com/") |> 
  req_headers(
    `User-Agent` = "Mozilla/5.0",
    Accept = "text/html,application/xml;q=0.9,*/*;q=0.8",
    `Accept-Language` = "en-US,en;q=0.5",
    `Accept-Encoding` = "gzip, deflate, br, zstd",
    `Upgrade-Insecure-Requests` = "1",
    Priority = "u=0, i",
  ) |> 
  req_perform()

All of the headers in the original command have been translated to work with {httr2}.

Python Tools

We’re going to look at two Python packages: uncurl and curlify. Install them first.

pip3 install uncurl curlify

The uncurl package can be used translate a cURL command to Python. The package comes with a command line executable.

uncurl "curl 'https://example.com/' -H 'User-Agent: Mozilla/5.0'"

requests.get("https://example.com/",
    headers={
        "User-Agent": "Mozilla/5.0"
    },
    cookies={},
    auth=(),
)

It can also be used in code. First import the package.

import uncurl

Now use the parse() function.

uncurl.parse("curl 'https://example.com/' -H 'User-Agent: Mozilla/5.0'")

requests.get("https://example.com/",
    headers={
        "User-Agent": "Mozilla/5.0"
    },
    cookies={},
    auth=(),
)

Magical!

The curlify package allows you to go in the opposite direction, translating a requests object into a cURL command.

First create a requests object.

import requests

response = requests.get("https://example.com")

Now translate.

import curlify

curlify.to_curl(response.request)

That returns a string with the following content (manually split across lines for readability):

"curl -X GET -H 'Accept: */*' -H 'Accept-Encoding: gzip, deflate' \
  -H 'Connection: keep-alive' -H 'User-Agent: python-requests/2.31.0' \
  https://example.com/"

There’s also a curlify2 package, which handles httpx requests.