Linux cURL - How to use the Curl Command in Linux

The free software cURL is one of the oldest and most popular open source projects. The program is written in C and is used to transfer data into computer networks. The name cURL stands for “client URL.” Because it has an open license, programmers can use the program for a wide range of purposes. Currently, cURL is being used in countless devices.

What is cURL?

The cURL software consists of two components. The libcurl software library acts as the backbone for data transfer and supports the following protocols:

  • DICT
  • FILE
  • FTP
  • FTPS
  • GOPHER
  • HTTP
  • HTTPS
  • IMAP
  • IMAPS
  • LDAP
  • LDAPS
  • POP3
  • POP3S
  • RTMP
  • RTSP
  • SCP
  • SFTP
  • SMB
  • SMBS
  • SMTP
  • SMTPS
  • TELNET
  • TFTP

The cURL command-line program, on the other hand, acts as a text-based interface and interacts with libcurl via the command line.

This program is an important tool for web development. It allows developers to communicate with servers directly instead of having to go through a browser. Scripts based on cURL commands are used to automate processes, testing, and debugging.

How does cURL work?

The two components in cURL work in different ways.

What is libcurl used for?

The libcurl software library provides functions for transferring data in computer networks. There are language bindings for dozens of popular programming languages. These make it easy for libcurl functions to be used in a wide variety of software programs that communicate with servers.

What is cURL used for?

The cURL command-line program is used for web development. The simplest method is to enter cURL commands into the command line. With the right know-how, it can be used to test and debug servers and APIs.

Instead of typing commands into the command line manually, they can be bundled in scripts. This allows complex operations to be standardized and automated. This includes both uploading and downloading of data and programmatically filling out forms and mirroring entire websites.

The following is the general syntax of a cURL command:

# General syntax of a cURL command
curl [options] <url>

We will use the following form in the examples below:

# Curling the following URL
url="www.example.com"
curl [options] "$url"

We have turned the actual URL into a variable. This allows the focus to be placed on the options. The options determine how the cURL request works.

Practical examples of the cURL command in Linux

To use the cURL examples below, you will need the following:

  1. A computer with Linux or a Unix-like operating system — including macOS
  2. Access to the command-line window or terminal
  3. A code editor/plain-text editor for writing commands
Note

You should only use a code editor/plain-text editor to write commands. Under no circumstances should you use a word processing program (e.g. Word, OpenOffice, or LibreOffice) to prepare text for the command line.

An initial test with cURL

First, test whether cURL is installed on your system. To do so, open a command-line window and enter the following command there. You may need to press Enter to execute the code.

# Check whether cURL is installed
curl --version

If you now see text starting with “curl” and a version number, this means cURL is installed. If cURL is not installed, please follow the instructions in “Everything curl” to install the program.

Tip

Use the command “curl --help” and “curl --manual” to learn more about the cURL command.

You should also open a new document in a code editor. You can copy commands to prepare them for use in the command-line window. Once the command is ready, copy it to the command-line window and execute it there.

Note

Commands used in the command-line window are quite powerful. In principle, an incorrectly entered command can cripple the entire system. Therefore, you should not just copy commands you find on the internet and execute them in your home command-line window. Instead, you should have a blank document open in a code editor where you can copy the commands first. This intermediate step allows you to check each command and modify it when necessary before executing it.

Downloading files with cURL

Retrieving data from a server with cURL

In principle, you can use cURL to retrieve any URL. In this context, the term “curl” is used as a verb to describe requesting a URL with cURL. Test the following example:

# Curling the following website
site="www.google.com"
curl "$site"

If you execute the code like written, it will output “data salad.” This is because the returned response to the cURL access will be the Google homepage in HTML. The HTML source code will be directly displayed in the command-line window without any formatting. A browser will be required to display it correctly. It is usually more useful to save the file locally.

Tip

Use the “clear” command in the command-line window to clear the screen. This will get rid of undesired data in the command-line window.

Retrieving data with cURL and saving it locally

Let us download a logo from the English version of Wikipedia. The “-O” option (upper case o, not a zero) tells cURL to use the name of the file at the end of the URL. When you use this option, cURL will save the downloaded file locally under the same name.

# An image from the English version of Wikipedia
file="https://en.wikipedia.org/static/images/project-logos/enwiki-2x.png"
# Retrieving the image and saving it locally under the same name
curl “$file" -O

But what if the URL does not contain a file name? Test the following code:

# Google homepage
homepage="www.google.com"
# Retrieving the homepage with the -O option
curl "$homepage" -O

As you can see, an error is displayed because the URL of the Google homepage does not contain a file name. In this case, you need to use the “-o” option (lower case o) to assign the file a name yourself:

# Google homepage
homepage="www.google.com"
# Name for the file to be created
name="homepage-google.html"
# Retrieving the homepage and saving it locally under the chosen name
curl "$homepage" -o "$name"

Resuming a download with cURL if it is interrupted

You may have run into this annoying situation before: you are performing a large download which has already taken hours when it is suddenly interrupted forcing you to start all over again. In this instance, cURL can help with the “-C-” option:

# A large file (human genome) to be downloaded
file="https://ftp.ncbi.nih.gov/genomes/refseq/vertebrate_mammalian/Homo_sapiens/reference/GCF_000001405.39_GRCh38.p13/GCF_000001405.39_GRCh38.p13_genomic.gbff.gz"
# Resuming the download if it is interrupted
curl -C- -O “$file"
Note

Technically, the option is “-C -” with a space after the C, but the similar version “-C-” is easier to read and remember, so we are using that here.

You can also manually cancel a download that was initiated with cURL if you need to. This is useful for larger files when, for instance, you have to leave the house and take your laptop with you.

Tip

You can cancel the current cURL request. To do so, press Ctrl+C. You may need to do this multiple times.

Communicating with the server using cURL

The download functionality of cURL is similar to the popular command-line tool wget. However, cURL is not specifically designed for performing downloads. It is designed for general communication in networks. cURL can thus do much more.

Using cURL to test whether a server is available

You can use cURL to test whether a server is available. In this regard, the way cURL works is similar to the ping command. However, cURL is more versatile due to the number of protocols and options available. In addition, ping works on an internet layer while cURL runs on the application layer. This means that ping tests whether the machine is on the network. In contrast, the following cURL command checks whether and how a server responds:

# Testing whether a web server is available
server="google.com"
curl -I "$server"

If the status information is displayed after executing the code, this means that the server is available.

Outputting a header using cURL

In each HTTP request, metadata is exchanged in addition to the actual document. This data is called the HTTP Header and describes both the document and the status of the HTTP request. You have probably already encountered the 404 Not Found error message at one time or another. The error is a type of metadata – the requested document was not found.

Take a look at the output of the header for the URL “google.com.” To do so, use cURL with the “--head” option:

# Outputting the header
url="google.com"
curl --head "$url"

You should see “301 Moved Permanently,” among other information. This is the HTTP status code for a permanent redirect. What you are seeing is a redirect from [google.com] (without the “www”) to [www.google.com].

Tip

Instead of using the “--head” option, you can use the equivalent version “-I” (upper case i). It is easy to remember using a mnemonic: you use it to retrieve information for a URL instead of its actual content.

Analysing redirect chains using cURL

HTTP redirects can be linked together in a sequence. We call this a “redirect chain.” For example, take a website with a homepage that displays the address [https://www.example.com/]. If you try to access the website using the address [http://example.com], this may result in the following redirects:

  • [http://example.com] → [https://example.com]
  • [https://example.com] → [https://www.example.com]
  • [https://www.example.com] → [https://www.example.com/]

Chaining redirects results in unnecessarily long load times and should be avoided. Unfortunately, this is not an easy problem to deal with. When the website is loaded into the browser, the redirects occur without a user seeing much of it. In this instance, cURL can help with the “--location” option: this tells cURL to follow the redirects to the last one. Here we are using the “--head” option because we are not interested in the content of the websites.

# Checking redirects
url="google.com"
curl --location --head  "$url"
Tip

Instead of using the “--location” option, you can use the equivalent version “-L” (upper case l).

Transferring data to a server with cURL

In addition to retrieving data, you can also use cURL to transfer data to a server. This may be practical if you want to automate the process of filling out an online form, for example. cURL supports the POST request method as well as GET.

Since transferring data is more complex than retrieving it, we can only provide a rough example here. For more detailed information, read about the HTTP POST method in Everything curl.

# Transferring data to a server
url="example.com"
# Providing the data as key value pairs separated by the “&” symbol
data="firstname=Peter&lastname=Mustermann&age=42"
# This cURL request transfers the data via POST
curl --data "$data" "$url"
# Alternatively, tell cURL to transfer the data via GET
curl --data "$data" "$url" --get

Accessing cookies with cURL

HTTP cookies are small pieces of data that are stored locally on your device when you visit most websites. You can view the cookies from websites you visited in your browser and delete them from there. However, this way is usually too tedious for testing purposes. You can also use the cURL option “--cookie-jar” to access the cookies more directly.

# Accessing cookies
url="www.google.com"
cookies="cookies.txt"
curl --head --cookie-jar "$cookies" "$url"
# Displaying cookies with the cat command
cat "$cookies"
Tip

You can also tell cURL to transfer cookies using the “--cookie” option.

General options

Some of the cURL options available can be combined with the ones we have already gone over.

Displaying additional information

Sometimes, the information displayed when you execute a cURL command is not quite sufficient. In such a case, you can use the “--verbose” option. When you execute a cURL command with this option, it will provide more information.

# Displaying additional information
url="www.google.com"
curl --verbose -I "$url"

Entering a username and password with cURL

Some URLs are protected against unauthorised access with HTTP authentication. But what if you want to access this kind of URL with cURL? If you do not provide a username and password, you will receive the HTTP 401 error message. In this situation, use the following syntax:

# Entering a username and password with cURL
# Password-protected URL 
url="www.example.com/secure/"
# User
user=""
# Password
password=""
curl --user "${user}:${password}" "$url"
Tip

You can also use cURL to retrieve data from an FTP server. To do so, you will once again use the --user option to enter the username and password.

Using a proxy with cURL

A proxy is an intermediary server. Proxies can be useful for a number of reasons. For example, some security and performance requirements can be more easily met by using a proxy.

You need to tell cURL if you are connecting to the internet using a proxy. To do so, use the “--proxy” option:

# Using a proxy with cURL
url="www.google.com"
proxy="proxy.example.com"
port="8080"
curl --proxy "${proxy}:${port}" "$url"
Note

If you need to enter a username and password for the proxy server, you can do so by using the “--proxy-user” option as well. If you do, enter the username and password in the following format: “user:password”.

Was this article helpful?