just enough curl for api devs
Tags: php, curl
Published: 2022-04-28 17:26:15

sure, [postman](https://www.postman.com/) is a great tool for poking at an api, but before postman ever existed there was [`curl`](https://curl.se/).

`curl` is a command-line tool that allows you to do basically anything you want to with an url. it runs on just about every operating system imaginable and has north of ten billion installs. 

if you open up the `man` page for `curl` it's terrifying; there are a dizzying array of options and features. it's easy to get discouraged.

however, `curl` is a useful and powerful tool, and learning the subset of features needed to test and develop your restful(ish) api is worthwhile. this article is designed to give us just enough `curl` skills to do precisely that.

## why curl
if postman already exists, why bother with `curl`? there are a lot of good reasons:

### curl is universal(ish)
if you use a linux-or-similar type operating system, there's an excellent chance that `curl` is already installed. and if, for some reason, it isn't, it's a fast `apt` or `yum` command away. `curl` can also be installed on just about every operating system imaginable. if you're using [beOS](https://en.wikipedia.org/wiki/BeOS) or [plan9](https://en.wikipedia.org/wiki/Plan_9_from_Bell_Labs), there's a `curl` for you. right click on an api call in the firefox console 'network' tab, and there's 'Copy as cURL'. `curl` is everywhere.

### curl is copy-pastable
a lot of tutorials for things like apis want to show you lo-res screenshots of postman under their 'trying it out' section. it's... not great. 

but `curl` is just text. if we give examples using `curl`, users can copy, modify and paste them to experiment and explore.

### curl is composable
because `curl` is just text, we can also use it in things like shell scripts or even pipelines if we add a little `sed` or `jq` here and there. being able to incorporate http calls into our scripts can be useful!

### curl is cli-friendly
`curl` is also useful for those times when you don't have a graphical environment, like when you're ssh'd into that server.

## the flyover
we can do a *lot* of stuff with `curl`, but this article is going to focus on the tools we need for api development. specifically:

* basic `GET`, `POST`, `PUT` and `DELETE` commands
* sending headers
* receiving headers
* sending json bodies, both as strings and from files
* uploading files
* a little bit of composition with `jq`

## the basic 'get'
let's start with the simplest `curl` statement: getting the content of an url. it looks like this:

```bash
curl https://api.example.ca/stuff
```

this command only has two parts: the `curl` command itself, and the url we are calling. 

if you run this against the url of a website, you'll probably get a bunch of html and javascript and the like. if you hit an api endpoint with it, you'll probably get a pile of json. precisely what we would expect!

when you run `curl` like this, it's *assuming* that you're calling HTTP `GET`. however, we can explicitly *tell* `curl` to use that method like so:

```bash
curl -X GET https://api.example.ca/stuff
```

the `-X GET` argument here is what tells `curl` that we want to use `GET` (instead of, say, `POST`). although it's not strictly necessary, it may be a good idea to explicitly state the HTTP method like this to improve readability.


### a pro tip: silent output
when we run `curl` we notice that it prints out some status outputs; stuff that looks like this:

```
* Trying 127.0.0.1:80...
* Connected to api.example.ca (127.0.0.1) port 80 (#0)
```

if we want to, we can suppress that noise by passing `-s` (for 'silent mode') to our command like so:

```bash
curl -s -X GET https://api.example.ca/stuff
```

## all the other methods
once we have `GET` down pat, using all the other http methods is just a matter of changing what we put after the `-X`. for instance, we can `POST` to our endpoint like so:

```bash
curl -s -X POST https://api.example.ca/stuff
```

this works nicely for `POST`, `PUT`, `DELETE` and the like.

## headers, sending 
sending headers with your request is a fairly straightforward affair. here, we add two headers using the `-H` (for 'header') argument with the header in a quote-enclosed string:

```bash
curl -s -X GET \
-H 'Accept: application/json' \
-H 'Origin: http://web.example.ca' \
https://api.example.ca/stuff
```

we can send as many headers as we like this way, or none at all.

## headers, receiving 
a big part of debugging url calls is inspecting the headers the server sends back to us. we can do that with the `-I` switch.

```bash
curl -I -s -X GET \
-H 'Accept: application/json' \
-H 'Origin: http://web.example.ca' \
https://api.example.ca/stuff
```

the `-I` switch we've added here means we get the headers *only*. our output will look something like.

```bash
HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
Cache-Control: no-cache, private
Date: Wed, 27 Apr 2022 15:03:42 GMT
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 59
Access-Control-Allow-Origin: *
```

### a pro tip: being verbose
if you want to see the headers sent as well as received *and* the returned body all at once, there's the 'verbose' argument `-v`.

## POSTing and PUTting json in the body
of course, if we're calling `POST` or `PUT`, we will probably want to send a json body of some sort. we can do this with the `-d` argument (for 'data'!) thusly:

```bash
curl -s -X POST \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
https://api.example.ca/stuff \
-d '{"name": "gbhorwood", "os": "popos21"}'
```

we pass the `-d` argument a quote-enclosed string of json here. 

there are a couple of important 'gotchas' to watch out for with this, though.

### escaping double quotes
first, be careful to escape quotes as needed and to do it properly. for this we simply use the the `\` character:

```bash
-d '{"name": "gb \"horwood\"", "os": "popos21"}'
```

### escaping single quotes
this is a bit more complex. in older versions of bash, escaping a single quote in a string enclosed by single quotes was challenging. more recently, though, it is possible by pre-pending a `$` to the string, ie

```bash
-d $'{"name": "gb \'horwood\'", "os": "popos21"}'
```

### building json bodies with variables
constructing a json body with a variable requires a bit more work on our part. let's start with an example:

```bash
name="gbhorwood"
os="popos21"
printf -v data '{"name": "%s", "os": "%s"}' "$name" "$os"
curl -s -X POST \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
https://api.example.ca/stuff \
-d "$data"
```

here we see that we've created two variables, `name` and `os`, that we would like to use in our json body. 

on line 3, we use those variables to build a string of json using `printf`. we use the `-v` switch on `printf` to assign the json string we build with our variables to the variable `data` and then, in our `curl` call, pass the `data` variable as the body. note that we need to refer to that variable as `"$data"` as the double quotes expand it to its value.

this is certainly a bit more complex, but with some practice it becomes easier.

### using a file as a json body
if we decide we want to keep our json body in a separate file, we can certainly do that.

first, let's create a file called `/tmp/body.json`

```json
{"name": "gbhorwood", "os": "popos21"}
```

note, that we don't need to have the enclosing single quotes here. this is just straight json!

we can then call `curl` and pass it the path to our file using the `@` operator like so:

```bash
curl -s -X POST \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
https://api.example.ca/stuff \
-d "@/tmp/body.json"
```

and the contents of `body.json` file is used as the request body.

## uploading a file
uploading files to api endpoints is a handy thing. people want their avatars, after all.

building on what we know so far about `curl`, writing a file upload command is fairly straightforward. 

```bash
curl -s -X POST \
-F 'file=@/path/to/some/file.jpg' \
'http://api.example.ca/avatar' 
```

the magic here is the `-F` argument that takes a path to the file we want to upload and assigns it to a variable name (`file` in this example). we note that the file path here is indicated with an `@`, in the same way it was when we used a file as our json body.

## composing a bit with `jq` 
one of the big advantages of `curl` is that, because it's just a command, it can be used in a composition like a shell script or pipeline.

to do this effectively, it's helpful to combine our `curl` calls with tools that select and format its output like [`jq`](https://stedolan.github.io/jq/) and [`sed`](https://www.gnu.org/software/sed/manual/sed.html).

let's take a look at an example where we make a `curl` call to a login endpoint and then extract the returned bearer token for use later.

```bash
token=`curl -s -X POST 'http://api.example.ca/login' \
-H 'Content-Type: application/json' \
-H 'Accept: application/json' \
-d '{"email": "gbhorwood@example.ca", "password": "<secretpassword>"}' | jq .data.token | sed -e 's/"//g'`

echo "our token is"
echo $token
```

there's a lot going on here, so let's go over it one step at a time.

first, we assign the output of our `curl` call to the variable `token` by enclosing our `curl` call in backtick quotes. this is a technique called [command substitution](https://www.gnu.org/software/bash/manual/html_node/Command-Substitution.html) and it's a powerful way to take the results of any `bash` command, not just `curl`, and put it in a variable.

the `curl` command itself should be readable to us by now; it's a `POST` call to login that sends the username and password in a json body.

the next bit of interest is the [`jq`](https://stedolan.github.io/jq/) command after the pipe. 

`jq` stands for 'json query' and it is a tool that allows us to format and query json text. it is easily installable via the package manager of your choice and is an important utility to have if you work with json.

here we pass `jq` the arg `.data.token`. this allows us to query the value keyed by `token` in the object keyed by `data`. an example will help here, so let's look at the json our login endpoint returns.

```json
{
    "data": {
        "name": "gbhorwood",
        "token": "<mybearertoken>"
    }
}
```

to get the value "<mybearertoken>", we use `jq` to query the top level, indicated by a `.`, then the object at `data`, then the value at `token`. the resulting command is `jq .data.token`.

of course, json encloses strings in double quotes and we would like to remove those; for that we use [`sed`](https://www.gnu.org/software/sed/manual/sed.html).

`sed` stands for 'stream editor' and, like the name implies, it allows us to programatically edit a stream of text. it is an incredibly powerful tool, but for our purposes here we will be using it simply to find all instances of the `"` character and replace those with null. the syntax is `sed -e 's/<what to find>/<what to replace with>/g'`. since we want to remove double quotes, we do `sed -e 's/"//g'`.

the result of all this is our bearer token assigned to a variable that we can use for anything we want, ie. `curl` calls to endpoints that require authorization using the header argument 

```bash
-H "Authorization: Bearer $token"
```

as we become more familiar with `jq`, we will be able to leverage it with `curl` to do all sorts of great things.

## conclusion
`curl` is an incredible powerful tool that has been under development for over twenty-five years and what we have covered here represents only a small fraction of what it is capable of. having a basic level of comfort with command allows us to better experiment with, document and communicate the functionality of our apis.