By the end of this tutorial, you should be able to:
Identify the base URL, path, and query parameters in an API request.
Construct and send an API request in R using httr2.
Write a function that changes selected parts of an API request.
Use map() and map2() to repeat API requests across multiple inputs.
The Business Challenge
Suppose you work as an analyst at the Australian Treasury. You have been asked to put together a dataset that describes how the Australian economy has changed over time.
You are interested in three measures: life expectancy, population, and GDP per capita. The World Bank collects each of these measures and makes them available through its API.
For this analysis, we want observations from 1960 to 2020, measured every five years. We could download the data by hand, but that would mean repeatedly finding and downloading the same kinds of data for different years and indicators.
Instead, we will use the World Bank API to request the data directly. We will start with one indicator in one year, then build up our code until we can collect all three indicators across the full period.
By the end of the tutorial, we will have created a dataset that we can use to study how Australia has changed between 1960 and 2020.
Libraries
We will need the following libraries:
library(httr2)library(tidyverse)
── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr 1.2.1 ✔ readr 2.2.0
✔ forcats 1.0.1 ✔ stringr 1.6.0
✔ ggplot2 4.0.3 ✔ tibble 3.3.1
✔ lubridate 1.9.5 ✔ tidyr 1.3.2
✔ purrr 1.2.2
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag() masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
Prepare these Exercises before Class
Prepare these exercises before coming to class. Plan to spend 30 minutes on these exercises.
Exercise 1: Breaking down an API URL
The World Bank API uses URLs to specify the data we want to request.
For example, the following URL requests Australia’s life expectancy in 2020:
(d) In your own words, explain what each of the following functions does in the request:
request()
req_url_path_append()
req_url_query()
req_perform()
Exercise 3: Reading the response
We have sent the request and saved the result as response.
(a) Print response.
response
What does this tell you about whether the request was successful?
(b) Use resp_body_json() to extract the body of the response and save it as data_raw.
data_raw <- response |> YOUR_CODE
(c) Inspect data_raw.
What do you notice about the structure of the object returned by the World Bank API?
(d) The response contains the data we requested, but it is stored in a nested structure. Cleaning this structure requires several steps that are specific to the World Bank API.
We have provided the following function to do this work for us:
Use clean_world_bank_data() to create a cleaned dataset called data by running this code:
data <- data_raw |>clean_world_bank_data()data
In-Class Exercises
You will discuss these exercises in class with your peers in small groups and with your tutor. These exercises build from the exercises you have prepared above, you will get the most value from the class if you have completed those above before coming to class.
Exercise 4: Turning the request into a function
Suppose we want to request data for multiple combinations of years and indicators.
(a) Looking back at the request from the pre-class exercises, which parts of the request would need to change when we move from one year–indicator pair to another?
Name the two pieces of information that should become arguments to our function.
(b) Complete the function below.
Call the function get_indicator() and give it two arguments: year and indicator.
(d) Now use the function to request Australia’s population in 2020. The World Bank indicator code for population is "SP.POP.TOTL".
YOUR_CODE
Exercise 5: Requesting data for many years
We now have a function that can request one indicator for one year. Suppose we want life expectancy for Australia every five years from 1960 to 2020.
(a) Create a vector called years containing the years from 1960 to 2020 in steps of five.
years <-seq(from = YOUR_CODE,to = YOUR_CODE,by =5 )
(b) Without using map(), use get_indicator() to request life expectancy for the first three years in years. What is changing across these three calls of the function? What is staying the same?
YOUR_CODE_HERE
(c) Use map() to repeat the request for every value in years.
(e) Inspect indicator_data. How does this dataset differ from the life_expectancy dataset we created earlier?
Exercise 8: Extending the request
So far, our function assumes that every request is for Australia.
(a) Suppose we also wanted the country to vary across requests. What would need to change in get_indicator()?
(b) We used map() when one input varied and map2() when two inputs varied. If year, indicator, and country all varied, what kind of change would we need to make to our mapping approach?