cURL over HTTPS - web mailbox authentication and data access

Created:01 Aug 2020 16:35:27 , in  Host development

CURL is one of these killer command line tools, which are as good today, in the age of ever more ubiquitous JavaScript and APIs as they were years ago, in gone-by times when a simple GET or POST HTTP request would have done what it takes. It is a transport tool, which facilitates exchange of data in a network, in its simplest form between two nodes - a client and a server.

CURL makes possible highly effective scripted communication between machines over a network, and it is fluent in a good number of protocols, HTTP (with TLS) being just one of them.

This article, first in this series, shows how to use cURL for interactions with a webmail service. In particular the focus is on how to log in to the service and obtain any of your data from it over HTTPS successfully.

It is not about how to hack a web mailbox. The goal is to show you how to use cURL to access YOUR OWN webmail account and data it holds using a CLI network client.

In general, an idea behind this, and other articles in this series is to look under the hood of what your browser or email client does as they access a service on the internet.

Requirements

In order to be able to replicate commands presented in this article you will need:

• - a webmail account with a webmail company. For the sake of this article I'm using a Polish webmail services provider called o2.pl. They provide webmail accounts free of charge. However, when you want to use one, you are likely to be served advertisements throughout the process (some less some more intrusive). So, the webmail account with the company is a much better place to visit with cURL than with a standard browser.

• - you will also need curl command compiled with TLS support and able to use certificates stored on your computer system. In my case the system is Debian\GNU Linux. Certificates on Debian are stored in /usr/share/ca-certificates directory.

• -last but not least, if you want to analyze HTTPS requests you will need a browser with Developer Tools like Firefox or Chrome installed. Trusted certificates used by Firefox are stored in /usr/share/ca-certificates/mozilla directory on Debian.

How to logging in to webmail mailbox with cURL

Let's start using cURL with an attempt to log in to a webmail account first. In order to get a feel of how the process looks, probably it is best to see it in your browser first. Open Developer Tools -> Network and try to log in while following what's going on during both request and response steps. Once the process is finished, check what headers and data were sent with the request and received with the response. Finally check what response code you got back (Anything between 200 and below 400 is ok).

Here is a log in attempt using a POST request using cURL:

# request is to https://poczta.o2.pl/login/v1/token, 
# -L option makes curl re-send request if the page has moved to different URL 
curl -L 'https://poczta.o2.pl/login/v1/token' \
# --cookie-jar option makes curl store cookie in a file, in this case the file is called poczta_jar 
--cookie-jar poczta_jar \
# if there are any cookies for this domain, send them in a header as part of the request
--cookie poczta_jar \
# set user agent (cURL as a user agent often causes request to be rejected, so use something different)
-H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0' \
# set acceptable response content type
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \
# set acceptable laguage
-H 'Accept-Language: en-US,en;q=0.5' \
# keep this connection alive for further request
-H 'Connection: keep-alive' \
# ask for encrypted and authenticated response
-H 'Upgrade-Insecure-Requests: 1' \
# what page the request is coming from
-H 'Referer: https://poczta.o2.pl/zaloguj' \
# send username and password to the server, this option sets request type to POST
--data-raw "username=YOUR_USERNAME_HERE" \
--data-raw "password=YOUR_PASSWORD_HERE" \
# require TLS connection
--ssl-reqd > /dev/null

Replace YOUR_USERNAME_HERE and YOUR_PASSWORD_HERE with your username and password and strip comments before running the code.

Adding -v option to the first line of the command gives a lot of detail about how a communication session with a server is progressing and whether it is successful one or not. Eventually, you want to see server returning back to cURL code like this ( visible -v option is on):

< HTTP/2 200 

If you do, it will mean, that as a result of the request a persistent secure connection has been established and all necessary cookies have been captured by curl and can be exchanged in further requests. That's all for the authentication step.

Obtaining data through imitating asynchronous requests with cURL

Pretty much all the data obtainable through my webmail account on successful login is fetched using asynchronous JavaScript requests. So, sending a request for it to the address in the address bar of the browser will return an error. In other words, that's not the endpoint you can obtain your data from.

If you try a request like the one below you will get a message, that your client does not support JavaScript. So, this is what not to do:

curl -L 'https://poczta.o2.pl/d/' \
--cookie-jar poczta_jar \
--cookie poczta_jar \
-H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0' \
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \
-H 'Accept-Language: en-US,en;q=0.5' \
-H 'Upgrade-Insecure-Requests: 1' \
-H 'Referer: https://poczta.o2.pl/' \
--ssl-reqd > /dev/null

To find out what the correct endpoint for xhr requests used by the interface is, visit your browser's Developer Console -> Networks tab again, click a link to resource you are interested in and scrutinize where browser sends your request.

I wanted data about number of read / unread e-mails in my mailbox first. It is loaded using a GET JavaScript asynchronous request. Response is a JSON object passed to the client as a string. Here is how to get this object.

Firstly, the endpoint for this request is: https://poczta.o2.pl/api/v1/listing/info.

The request is sent like this:

# there is no -L option this time, no redirection expected
curl 'https://poczta.o2.pl/api/v1/listing/info' \
--cookie-jar poczta_jar \
--cookie poczta_jar \
# make this request a GET request
--get \
# append url encoded data as a query string
--data-urlencode "seg=152,154,153" \
-H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0'  \
-H 'Accept: application/json, text/plain, */*' \
-H 'Accept-Language: en-US,en;q=0.5' \
# ask for a compressed response data
--compressed \
-H 'Referer: https://poczta.o2.pl/'   \
-H 'Cache-Control: no-cache' \
# ask for JSON encoded content
-H 'Content-Type: application/json; charset=utf-8' \
-H 'Connection: keep-alive' \
-H 'TE: Trailers' \
--ssl-reqd 

And here is the JSON object in response:

{"data":
  {"labels":
    [
      {
        "id":1,
        "count":3,
        "unread":3
      },
      {
        "id":148,
        "count":1,
        "unread":1
      },
      {
        "id":152,
        "count":1,
        "unread":1
      },
      {
        "id":2,
        "count":43,
        "unread":12
      },
      {
        "id":5,
        "count":276,
        "unread":276
      },
      {
        "id":3,
        "count":395,
        "unread":394
      }
    ],
    "size":74006373,
    "flags":0
  }
}

One cool thing about requests, which return JSON encoded data is that the output is very clean and easy to consume away. There is no trouble with tags or other extraneous bits which need removing. You get something that can be sent away to some other location and used immediately and in any modern language, be it PHP, JavaScript or anything else.

Let's look at few similar, but not identical, request attempts now.

More cURL examples which return JSON object

Here is how to obtain data about messages stored in the webmail mailbox. The request endpoint was found using Developer Tools / Network tab again.

curl 'https://poczta.o2.pl/api/v1/listing/messages' \
# save and use cookies collected earlier
--cookie-jar poczta_jar \
--cookie poczta_jar \
# make it a get request
--get \
# attach a query string with data to the URL
--data-urlencode "seg=152,154,153" \
--data-urlencode "label=1" \
--data-urlencode "conv=true" \
--data-urlencode "count=100" \
--data-urlencode "binder=true" \
# set usual HTTP request headers
-H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0' \
# expect a JSON object as a respons
-H 'Accept: application/json, text/plain, */*' \
-H 'Accept-Language: en-US,en;q=0.5' \
--compressed \
-H 'Referer: https://poczta.o2.pl/' \
-H 'Cache-Control: no-cache' \
-H 'Content-Type: application/json; charset=utf-8' \
-H 'Connection: keep-alive' \
-H 'TE: Trailers'

If everything went well you should get another nice and tidy JSON object which contains e-mail subjects, e-mail sender, email addresses, delivery times etc. .

Obtaining email addresses from the contact book

This time, the goal is to get all the email addresses together with names of their holders. The data is available from https://poczta.o2.pl/api/v1/contacts

curl 'https://poczta.o2.pl/api/v1/contacts'  \
# user cookies
--cookie-jar poczta_jar \
--cookie poczta_jar \
# make it a GET request
--get \
# attach a query string with configuration of what you want
--data-urlencode "marker=0" \
--data-urlencode "per_page=100" \
--data-urlencode "with_categories=1" \
--data-urlencode "count=100" \
--data-urlencode "source=1" \
# ask for compressed response
--compressed \
# add usual headers
-H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0'  \
-H 'Accept: application/json, text/plain, */*' \
-H 'Accept-Language: en-US,en;q=0.5' \
-H 'Referer: https://poczta.o2.pl/d/' \
-H 'Cache-Control: no-cache' \
-H 'Content-Type: application/json; charset=utf-8' \
-H 'Connection: keep-alive' \
-H 'TE: Trailers'

Here is an edited JSON object I got:

{"data":
  [
    {"categories":null,"source":1,"email":[
      {"value":"******@poczta.onet.pl"}
    ],
      "id":"6ef165f3-3528-8d80-1a80-*****",
      "fn":"Agnieszka ***"
    },
    .
    .
    .
  ],
  "_meta":{
    "status":"success",
    "all":25,
    "marker":"",
    "perpage":100,
    "took":71517195,
    "success":true
  }
}

I think, at this point you should have a fairly good idea about how to handle authentication and repilicate asynchronous JavaScript requests while logged in with cURL.

One point worth noting here is, Developer Tools -> Networks context menu option 'copy as cURL' allows you to copy the whole command used by the browser. However, often those are not commands you can use straight away. They need some adjustment to exit or obtain data successfully . So, they are a good starting, sometimes a middle point, but not the final solution.

Conclusion

CURL can be seen as an alternative interface for interacting with enpoints of a network. One which is so versatile and powerful at communication and facilitating data transportation, that many service owners just do not want to see it at their doorstep. They have a good reason for this, among other things, the tool can easily bypass all the spam they inject in their service frontends.

I hope you have found this tutorial useful. If you have something to add,leave a comment below.

This post was updated on 03 Aug 2020 00:51:26

Tags:  cURL