Pulling Data from an API using Python Requests

In this tutorial, we will learn how to pull data from an API using Python. We will start by providing an overview of HTTP methods such as GET, POST, PUT, and DELETE, and explain their purposes. Then, we will introduce the requests library in Python, which is commonly used for making HTTP requests. Finally, we will provide an example of making a basic GET request to retrieve data from an API.

Let’s get started!

Overview of HTTP Methods

HTTP methods are the actions that can be performed on a particular resource. The most commonly used HTTP methods are:

GET: Retrieves data from the server. It is used to request a representation of a specified resource.
POST: Sends data to the server to create a new resource.
PUT: Updates a resource with new data sent to the server.
DELETE: Deletes a specific resource from the server.

Understanding these methods is crucial as they determine how data is requested, sent, updated, or deleted in an API. Most public APIs will use GET to interact with the data. Once you start developing your own APIs, you’ll use the other methods, as well.

Introduction to the requests library

Python provides various libraries for making HTTP requests, but one of the most popular ones is the requests library. It simplifies the process of making HTTP requests and handling responses. To use the requests library, you need to install it first using pip:

$ pip install requests

Once installed, you can import the library in your Python script:

import requests

Get Our Python Developer Kit for Free

I put together a Python Developer Kit with over 100 pre-built Python scripts covering data structures, Pandas, NumPy, Seaborn, machine learning, file processing, web scraping and a whole lot more - and I want you to have it for free. Enter your email address below and I'll send a copy your way.

Making a basic GET request using requests

To pull data from an API, we can use the GET method with the requests library. The library provides a get() function that makes it easy to send GET requests. Here’s how you can make a basic GET request to retrieve data from an API:

import requests
import json

# The URL of the API endpoint you want to retrieve data from
url = "https://api.example.com/data"

# Send a GET request to the API endpoint
response = requests.get(url)

# Check if the request was successful (status code 200)
if response.status_code == 200:
    # Access the retrieved data
    data = response.json()
    # Process the data as necessary
    json_formatted_str = json.dumps(data, indent=2)
    print(json_formatted_str)
else:
    print("Error:", response.status_code, response.reason)

Sample Code Explanation

We import the requests module to make the HTTP request and the json module to display the results in a nicer format.
Set the url variable to the API endpoint you want to retrieve data from. Replace https://api.example.com/data with the actual URL of the API endpoint you are targeting.
Use the get() function from the requests module to send a GET request to the specified URL.
Check if the request was successful by verifying the status code of the response. A status code of 200 indicates a successful request.
If the request was successful, we can access the retrieved data using the json() method provided by response. We can then process the data as needed. To help you, we made a tutorial showing how to read JSON files in Python.
In case of an error, we print the error status code.

It’s important to remember that not all APIs will follow the same convention of presenting results in JSON format, so if you’re testing a specific API url, you may encounter a JSON decode error. Always refer to the documentation for the API you’re interacting with to understand the output format. It’s not uncommon to have to replace data = response.json() with:

data = response.content

data = response.text

Another point to keep in mind is that most APIs require authentication of some sort to prevent people from abusing access to free APIs. For APIs that require authentication, you’ll need to follow the documentation to learn how to apply or pass your unique API key to your request.

Full Python API GET Demo

To demonstrate an API GET request, let’s process US wages by occupational category for recent years. In the example below, we updated the url to match a valid API request URL (at the time of publication). We also updated the section that processes the JSON output so it’s presented in a suitable format. In this example, we plot the data.

import requests
import json
import pandas as pd
import matplotlib.pyplot as plt

# The URL of the API endpoint you want to retrieve data from
url = "https://xenotime.datausa.io/api/data?drilldowns=Year,Broad Occupation&measures=Average Wage&Workforce Status=true&Nation=01000US&Broad Occupation=230000,110000,150000,170000,130000&Record Count>=5"

# Send a GET request to the API endpoint
response = requests.get(url)

# Check if the request was successful (status code 200)
if response.status_code == 200:
    # Access the retrieved data
    data = response.json()
    # Process the data as necessary
    data = response.json()["data"]
    df = pd.json_normalize(data) 
    df = df.sort_values(by=['Year'])
    for occupation in df["Broad Occupation"].unique(): 
       subset_df = df[df["Broad Occupation"]==occupation]
       plt.plot(subset_df["Year"], subset_df["Average Wage"], label=occupation) 
    plt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.2), fancybox=True, shadow=True) 
    plt.xlabel("Year") 
    plt.ylabel("Average Wage") 
    plt.subplots_adjust(bottom=0.38)
    plt.grid(True)
    # display the y-axis labels as currency
    plt.gca().get_yaxis().set_major_formatter(plt.FuncFormatter(lambda x, loc: "${:,}".format(int(x))))
    plt.show()

else:
    print("Error:", response.status_code, response.reason)

The JSON response for this example is formatted like this:

{
  "data": [
    {
      "ID Year": 2020,
      "Year": "2020",
      "ID Broad Occupation": "230000",
      "Broad Occupation": "Legal Occupations",
      "ID Workforce Status": true,
      "Workforce Status": "true",
      "ID Nation": "01000US",
      "Nation": "United States",
      "Average Wage": 127425.80165454857,
      "Slug Broad Occupation": "legal-occupations"
    },
    ...
    ...
    ...
  ]
}

Instead of viewing the raw output, we processed this JSON response using Pandas and Matplotlib to display wages over time:

Python requests API output processing

In this tutorial, we learned how to pull data from an API using Python. We started by understanding the HTTP methods and their purposes. Then, we introduced the requests library and demonstrated how to use it to make a basic GET request to retrieve data from an API. This is just the beginning, as you can explore more advanced features of the requests library and make use of other HTTP methods to interact with various APIs. You can even make your own APIs to interact with a database you’ve created yourself. Happy coding!