A network protocol specifies how two devices, or more precisely processes, communicate with each other.
A network protocol is a pre-defined set of rules and processes to determine how data is transmitted between devices, such as end-user devices, networking devices, and servers. The fundamental objective of all protocols is to allow machines to connect and communicate seamlessly, regardless of any difference in their internal design, structure, logic, or operation. In analogy, a networking protocol is like a “common language” that helps make communication possible among people with different native languages and from various parts of the globe.
In this room, we will learn primary protocols essential for network security at each OSI model layer.
Understanding of following topics is recommended before starting the course:
In this part, we will learn the core technical concepts of various protocols. However, to proceed further, we must be very clear about the following two ideas, which are the hallmark of any protocol regardless of its functionality:
- Each protocol represents specific layers of the OSI or TCP/IP model and operates as per the functionality of that layer.
- TCP and UDP-based protocols operate on specific network ports.
Technical Overview – HTTPS
Hypertext Transfer Protocol Secure (HTTPS) is a client-server protocol; responsible for securely sending data between a web server (website) and a web browser (client side). It is an encrypted variant of HTTP which sends data in an unencrypted format.
HTTP would be enough to browse a website to learn about a company product; however, HTTPS is a must if you want to provide your credit card details to place an online order. HTTPS was developed to securely share sensitive information, including passwords, contact information and financial information, between web browsers and websites. Without HTTPS, secure online banking and online payment wouldn’t have been possible.
Workflow – HTTPS
HTTPS uses its unencrypted counterpart, i.e., HTTP, and adds a layer of encryption. In this case, it is SSL/TLS (Secure Sockets Layer/Transport Layer Security); the rest of the workflow remains the same. So before proceeding forward, we will review how HTTP requests and responses operate in a typical client and server environment.
Request and Response – HTTP
An HTTP request is made by a user agent (a browser or any other application sending requests through a web API (Application Programming Interface)). It is vice versa in the case of response. This request aims to access some resources on the remote web server, which is then responded to by the web server. The figure below shows a web browser sending an HTTP request to a web server, listening at TCP port 80.
The request might be
GET to request a web page, an image, or a file. Other HTTP requests include
POST, which send data to the web server, such as a value or a file. You can read more about HTTP in the HTTP in Detail room.
If an attacker can capture the network packets between the client and the server communicating over HTTP, they will be able to read their content as it is sent in cleartext.
Request and Response – HTTPS
After our quick review of the HTTP request and response workflow, it is convenient to learn about HTTPS. Remember that the “S” in HTTPS is for the extra SSL/TLS layer of encryption added over HTTP.
Even if an attacker can capture the network packets between the client and the server communicating over HTTPS, they will fail to read the contents of the TCP data due to encryption.
Encryption Mechanism of HTTPS
As already mentioned, SSL/TLS provides the encryption layer of HTTPS. It relies on asymmetric encryption (public key cryptography) and symmetric encryption. Asymmetric encryption uses two keys, i.e., public key and private key; its rule is to negotiate the symmetric encryption algorithm and the secret key. The default port of HTTPS is 443. Encryption protects against interception and alteration of data, maintaining the confidentiality and integrity of exchanged traffic.
Technical Overview – FTPS
File Transfer Protocol Secure (FTPS) is a communication protocol which is a refined and secure version of File Transfer Protocol (FTP). Initially, FTP was developed in 1971 and published as RFC 114. Additional improvements and various changes were published in RFC 765 and RFC 959.
FTP was designed as a client-server model; separate control/command and data connections between a client and a server are used, along with a username and password. In FTP, both authentication and data transfer take place in an unencrypted form between the client and the server; however, in FTPS, an encrypted channel is established.
FTPS is an extension of FTP, which adds TLS security to commands and data connections. It is necessary to get an overview of FTP to understand FTPS.
Request and Response – FTP
As described earlier, FTP is based on the client-server model. It utilizes the following two communication channels between the client and the server.
- Control Connection: In this connection, an FTP client (such as Filezilla and CuteFTP) sends a connection request (authentication) to the remote FTP server at the default FTP port, TCP port 21. As the name implies, a control connection is used for sending and receiving commands and responses.
- Data Connection: After authentication, this connection is used for transferring data (files and folders).
FTP Connection Types
FTP has two modes:
- Active modes
- Passive mode
In active connection mode, the client establishes the control connection to send commands/authentication parameters to the server. After authentication and upon the client’s request to initiate data transfer, the server establishes the data connection to the client to transfer the data. In brief:
- The FTP client connects to the FTP server at TCP port 21 to establish a command connection.
- The FTP server connects to the FTP client at TCP port 20 to establish a data connection.
It is worth mentioning that this type of connection is unsuitable in an environment where the client is behind a firewall, as it will block incoming connections to the client. In the case of a client behind a firewall, a passive connection would be necessary.
In passive connection mode, the client establishes the control and data connections. The client sends the
PASV command to the server over the command channel; the server sends a random port to the client. As soon as the client receives the port number, the client establishes a connection to the provided port number so that the server can initiate the data transfer to the client.
This type of connection works well when the client is behind the firewall.
FTP Data Types
When data exchange between client and server takes place, the following type of data types are used:
- ASCII/Type A: This is the default type and is used for text file transfers. If necessary, data is converted into 8-bit ASCII before transmission and then converted back upon reception.
- Image/Type I: This is commonly referred to as the binary mode. It uses byte-by-byte transmission. The recipient stores the received bytes upon reception.
- EBCDIC/Type E: It is suitable for text communication using the EBCDIC character set.
- Local Type L n: It is typically used for file transfer among machines that do not support 8-bit bytes transfer. Here
nis a second parameter that represents byte size.
Request and Response – FTPS
As the name implies, FTPS is an extension of FTP. It adds an encryption layer to transmit command and data channels between client and server securely. The following two methods are used to invoke security:
- Implicit Connection: In this connection, FTPS client and server establish a link in which both command and data channels are secured automatically with SSL encryption.
- Explicit Connection: The FTP client explicitly requests the server to invoke an SSL/TLS secured session on port 21 and then continue data transfer based on a mutually agreed authentication mechanism. With explicit connection, you can choose which channel to encrypt by choosing among three modes of communication for control and data channel, i.e., control only encrypted, data only encrypted and both control and data encrypted.
The standard port for FTP and Explicit FTPS is 21, whereas it is 990 in the case of Implicit FTPS. Adding FTPS protects against sniffing attacks against login information and data.
Simple Mail Transfer Protocol Secure (SMTPS) is an extension of SMTP, which is used for email communication. We should not confuse SMTP with POP3. Although both are used for email communication, SMTP is an “Email Push Protocol” used to transfer email messages from the client to the server. In contrast, POP3 is used to download email messages from the server to the client. SMTPS is an extension of SMTP; it uses TLS/SSL to provide authentication, integrity, and confidentiality for transferred data. First, let’s review the SMTP protocol.
As described earlier, SMTP is an “Email Push Protocol” commonly used to transfer emails from an SMTP client to an SMTP server. SMTP is implemented in the following two models:
- SMTP End-to-End: This model is used for email communication between organizations. In this model, the sender-side SMTP client initiates an SMTP connection to the recipient’s SMTP server.
- SMTP Store-and-Forward: This model is used for email communication within an organization. In this model, the SMTP server will maintain the copy of the mail within itself (i.e., store) until the copy is forwarded to the receiver.
To understand the workflow of SMTP, we will study the following essential components of SMTP:
- User Agent (UA): UA is responsible for creating the email message and sending it to the Mail Transfer Agent (MTA).
- Mail Transfer Agent (MTA): MTA will transfer the email from the UA to the recipient MTA across the Internet (often, the MTA and Mail Delivery Agent are hosted on the same server).
TLS Process in SMTPS
SMTPS is not a proprietary protocol; instead, it wraps SMTP inside TLS. You can say that SMTPS is similar to SMTP on the application layer, with an extension of TLS encryption at the transport layer. For encryption, the
STARTTLS command is used between the email client and the email server.
Port 587 and 465 are both frequently used for SMTPS traffic. Mails transmitted using SMTP are not encrypted, so they are prone to sniffing attacks. Therefore, SMTPS is used to encrypt emails through TLS before transmission. In addition, SMTPS also forbid attackers from sending spam messages from compromised/vulnerable domains, exfiltration sensitive information, and conducting phishing attacks.
Technical Overview – POP3S
Post Office Protocol Secure (POP3S) is an extension of the POP3 protocol; it is used for the encrypted retrieval of email messages from the email server to the email client. So first, let’s review the POP3 protocol.
In the previous section, we explored how SMTPS is used for secure email transmission. SMTP is not responsible for retrieving email messages; here, POP3 comes into play. POP3 is the latest POP version; it retrieves email messages from a Mail Delivery Agent (MDA) to a Mail User Agent (MUA).
POP3 Components and Workflow
Like SMTP, POP3 has two components: client (MUA) and server (MDA). The steps are the following:
- The email client establishes a connection to the email server.
- The email client downloads all the queued emails from the email server. (This is a default option; however, the client can select only particular email messages to download.)
- All emails are saved on the device that initiated the connection.
- The email server deletes the email copy. (This is a default option; a client can choose not to download an email after it is retrieved.)
Limitations of POP3
- Emails are Processed Locally: No synchronization of email messages across multiple devices. Protocol downloads the emails on the currently logged-in device and usually deletes them from the server.
- Transmission in clear text: The username and password, along with the email messages, are sent in cleartext, which makes them vulnerable to sniffing attacks.
As we conclude, POP3 is considered weak from a security point of view. This requires an added layer of security; hence, POP3S comes into play. POP3S is an extension of POP3, which wraps the communications related to email messages within TLS. For this purpose, the client and server initiate the
STARTTLS command, as shown in the figure below. After the
EHLO, the POP3S server will trigger the switch to TLS. Note that
EHLO stands for Extended HELO, where
HELO is the command used to identify to the server.
The POP3S Protocol uses port 995, while POP3 uses port 110. In the next task, we will learn a few other secure protocols at the Application layer.
As you would already know, DNS stands for Domain Name System. The DNS protocol is responsible mainly for resolving domain names. Instead of remembering the IP address, you need to focus on the domain name. For instance, at the time of this writing,
example.com resolves to
18.104.22.168; it is clear which one is easier for the human mind to remember.
DNS works by sending a DNS query. For instance, when browsing the web, your web browser might send a query for DNS record type
AAAA, i.e., IPv4 or IPv6 addresses. In the following console output, we can see the host with IP address
192.168.0.102 sending two DNS queries to the DNS server
22.214.171.124 regarding the domain name
example.com. We can see the responses for the
user@TryHackMe$ sudo tshark port 53 1 0.000000000 192.168.0.102 → 126.96.36.199 DNS 82 Standard query 0x2717 A example.com OPT 2 0.012241216 188.8.131.52 → 192.168.0.102 DNS 98 Standard query response 0x2717 A example.com A 184.108.40.206 OPT 3 0.013454645 192.168.0.102 → 220.127.116.11 DNS 82 Standard query 0xac05 AAAA example.com OPT 4 0.018705620 18.104.22.168 → 192.168.0.102 DNS 110 Standard query response 0xac05 AAAA example.com AAAA 2606:2800:220:1:248:1893:25c8:1946 OPT
This name-to-IP address resolution is very convenient; however, anyone on the network could have responded with a forged response. Furthermore, the host that sent the query would have accepted “any” response. In other words, the host would connect to a rogue server. One way to avoid such a situation would be by using DNSSEC.
DNSSEC makes it possible to ensure that the DNS response we receive is from the domain owner. To achieve this, DNSSEC requires two main things:
- The DNS zone owner should sign all DNS records using their private key.
- The DNS zone publishes its public key so users can check the validity of the DNS records signatures.
In other words, the data to our DNS query is signed to ensure its integrity and authenticity; moreover, we can efficiently check the signature.
With signed records, DNSSEC provides the following:
- Authenticity: You can confirm that a certain DNS owner has authored and sent the record. Authenticity is possible because the received record is signed by the DNS owner’s private key.
- Integrity: You can ensure that no changes have been made to the record on its way. Any changes to the record will render its signature invalid.
When the first email was sent in 1971, we had a different cyber security landscape. Email protocols such as SMTP and POP3 are designed to send emails in cleartext. The same applies to IMAP, which allows synchronizing your mailbox with that on the server. All these protocols make your email no different than an exposed postcard open for everyone to see as it is handed from one server to another.
The image below shows a simplified example where a user uses an email client to send their email over SMTP and receive new email messages over POP3 or IMAP. The mail server uses SMTP to deliver the user’s email messages to the intended recipient. Since all these protocols use clear text, an intruder can read the email messages as they travel across the Internet.
With the increased popularity of web-based email, users started to connect to a web server to read and compose their email messages. The image below shows an email message as it is written using a web browser. The web server, in turn, uses a mail server to send composed email messages and receive incoming ones. The connection was over HTTP, which meant that the same security issues related to confidentiality and integrity persisted.
However, as service providers realized the need for SSL/TLS to secure web traffic, HTTPS became the new standard. Consequently, most web-based email systems migrated to HTTPS, causing the traffic between the web browser and the web server to be encrypted. However, the email traffic is not necessarily encrypted between the web server and the mail server(s). The web server and mail servers can read the contents of the messages; moreover, mail servers might use SMTP to transfer the messages, which means that email messages will traverse the Internet in cleartext.
Eventually, SSL/TLS started to find their way into all email protocols. SMTP, POP3, and IMAP became SMTPS, POP3S, and IMAPS, respectively. The “S” added to the protocol name refers to secure, indicating the addition of SSL/TLS on top of the existing protocol.
The image below shows a simplified example where a mail client uses SMTPS to send an email and uses POP3S or IMAPS to receive an email. The result is that email is sent encrypted between the client and the server; however, the mail server can read the email message contents.
The addition of SSL/TLS has dramatically enhanced the security of email messages. However, we must still trust the mail servers across the way. If this is not something you are comfortable with, you need to consider a standard such as OpenPGP. PGP (Pretty Good Privacy) is an encryption program created by Phil Zimmerman. OpenPGP is an open standard for signing and encrypting files and email messages and is detailed in RFC 4880. GnuPG (Gnu Privacy Guard), or simply GPG, is a free and open-source implementation of the OpenPGP standard. In brief, GnuPG allows you to sign and encrypt your data and communications.
GnuPG can easily integrate with your mail client to seamlessly sign, encrypt and decrypt email messages. Email messages encrypted using GnuPG (i.e., following OpenPGP standard) be only readable by the intended recipient. In other words, no one, including the mail servers, can read the contents of the messages except the intended recipient.
When used with email, GnuPG requires each user to generate a key pair: a private key and a public key. In simple terms, the sender’s private key is used for signing, while the recipient’s public key is used for encryption. From the recipient’s perspective, the sender’s public key is used to check the signature, while the recipient’s private key is used for decryption. For more information about asymmetric encryption, we recommend you check the Introduction to Cryptography room.
OpenPGP implementations, such as GnuPG, offer a great solution to protect the confidentiality and integrity of email message contents. However, this does not include email message headers.
Below is an example of a message before and after being encrypted using OpenPGP. The original message is shown below.
user@TryHackMe$ cat message.txt Hello, Please proceed with the transaction. Best, Strategos
To use OpenPGP, both parties need to generate a key pair using the command
gpg --gen-key. This command will ask the user to provide their name and email address and create a private and public key. The private key should be stored securely, while the public key should be shared with the other parties we wish to communicate securely with.
Using an email client that supports OpenPGP will encrypt the message using the key of the recipient; however, if we want to accomplish this via the command line, the command would be something like the following:
gpg --encrypt --sign --armor -r firstname.lastname@example.org message.txt
Notice the following options:
--encrypt -r email@example.com encrypt
message.txtusing the public key associated with the recipient’s email. This will provide confidentiality.
--signwill sign our message (using our private key). This will prove authenticity.
--armoris to produce the output using ASCII instead of binary.
gpg created the following message that can be sent seamlessly with an email client.
user@TryHackMe$ cat message.txt.asc -----BEGIN PGP MESSAGE----- hF4Dt1Jduipo/LESAQdAmqgCLQRRQCFNOBWSF+dY64suA8xtty7ysfolfF7+fnUw crwR2ioRTcXTe6c0dZl/sdmtjDJPZWGHI3XcD7XWA2hPDb+w4P46e9FJGsCE/JaO 1I4BCQIQZc91A79Ebli/41D0aVkBmDpjIgvpwjHdmomT7dghTcB+Qp80WbYDnV20 4qTgdgdAnLtQp3fnJCXlZ0BfecPB+ZfECdD1IAleBB3o14v5v/ntfPKfXPZwODUm ELY7piC2GclBWbirrZsnzTLWeYCrABiKJ3Rb75VgJXdM1uKNBY0HLN06VdEuDy+L =nRQE -----END PGP MESSAGE-----
With the beginning of networked systems, there was a need to connect to a system over a network. For instance, a user needs to execute processes and administer the system. The aim was to do this remotely instead of being physically present at the computer.
Two of the earliest protocols were Telnet and remote login, with the clients
rlogin, respectively. Both protocols made it possible to log in to a system over a network; however, neither focused on the security aspects. The confidentiality of the exchanged traffic, especially the login credentials, was not protected. Moreover, the integrity of the traffic and commands sent was not ensured. In other words, it was easy for an attacker monitoring the network to read the login credentials and modify the commands sent over the network.
The screenshot below is taken from Wireshark after using “Follow TCP Stream” against a Telnet session. We can see the user typing his username
michael and pasting his password
RJ9wn^t3T%gC. Although the password is secure by modern standards, it is sent in cleartext for any properly located network packet-capturing software to read. Note that in the image below, the text in red is sent by the Telnet client, while the text in blue is sent by the Telnet server.
The Secure Shell Protocol (SSH) provided the security requirements lacking in Telnet and remote login. With SSH, it is no longer feasible for the attacker to read the login credentials or modify the traffic. If we attempt to “Follow TCP Stream” on Wireshark after capturing the packets, we won’t get any information regarding the username, password, or issued commands. The screenshot below shows that the only visible information is the version numbers and the supported protocols. (Please note that the red characters are sent by the client, while the blue characters are sent by the server.)
Technical Overview – SSL/TLS
Secure Socket Shell (SSL) and Transport Layer Security (TLS) are protocols used to encrypt data exchanged between a client, such as a web browser, and a server. Consider SSL/TLS as a wrapper that encrypts various communication protocols, such as HTTP and FTP, to create HTTPS and FTPS. SSL is not commonly used nowadays as TLS has been gradually replacing it.
SSL/TLS handshake is performed to encrypt the communication between client and server through the following steps:
- Client Hello Message: The client sends a hello message to the server; it includes the client TLS version and the cypher suite that the client supports, in addition to random bytes.
- Server Hello Message: The server responds with a hello message, highlighting its certificate, chosen cypher suite and random bytes.
- Authentication: The client authenticates the server’s certificate through the certificate authority that issued it. For example, when we visit Google, Google shares its certificate. The received certificate is verified by our browser, which is pre-installed with the certificates of various certificate authorities.
- Premaster Secret: The client encrypts random bytes with the server’s public key. (The client retrieves the public key from the server’s certificate.)
- Decryption of Premaster: The server decrypts the premaster with its private key.
- Session Keys Generated: The client and the server generate session keys based on client random bytes, random server bytes and premaster secret. Both will arrive at the same results; this session key is not transmitted, and encryption and decryption are based on this key.
- Ready Messages: The client and server send a “finished” message using the session key to indicate that the session is ready for transmission. The client and server are now ready to exchange messages over SSL/TLS encrypted connection.
TLS is a wrapper that encrypts communication of communication protocols. It has port numbers for various protocols, such as 443 for HTTPS and 990 for FTPS.
Technical Overview – SOCKS5
Socket Secure (SOCKS) is a proxy protocol for data exchange through a delegate server (SOCKS5 proxy). It is used to secure application layer protocols. For example, the Squid server implements the SOCKS5 protocol to transfer data via the HTTP protocol.
Consider a scenario when user A wants to connect with client B over the Internet, but a firewall is between them. The following handshake steps are involved:
- Client Initiation
- Client A connects with the SOCKS5 proxy and sends the first byte (0x05) to the proxy where “5” is the SOCKS version.
- Client A sends a second byte (0x01). One means authentication is supported.
- Client A sends the third byte (0x00, 0x01, 0x02, or 0x03); these bytes denote the supported authentication methods and can be of variable length.
- SOCKS5 Proxy Reply
- The proxy sends back a second byte, which is the chosen authentication method by the proxy server.
- After the initiation packet, client A sends the request packet, which includes BHOST & BPORT numbers.
- The successful session is established between client A and the proxy. The same steps are involved in the association of client B with the proxy.
- Data Transfer
- After successfully associating both clients with a proxy server, both clients can exchange data and share information that will be routed through the proxy server.
Benefits of SOCKS5
- In direct communication via the proxy server, hide the internal details from routing over the Internet.
- A proxy acts as a relay server, bypassing Internet censorship based on the client’s IP address.
IPsec stands for Internet Protocol Security. In this room, we use IPsec to refer to IPsec-v3. IPsec provides security by adding authentication and protecting the integrity and confidentiality of the network traffic. IPsec uses the following protocols:
- Authentication Header (AH): Provides authentication and integrity.
- Encapsulating Security Payload (ESP): Provides authentication, integrity, and confidentiality.
- Security Association (SA): Is responsible for negotiating the encryption keys and algorithms. One example is Internet Key Exchange (IKE). Discussing SA in more detail is outside the scope of this room.
In the following sections, we discuss AH and ESP in more detail.
Authentication Header (AH)
Authentication Header (AH): The AH protocol is responsible for the authentication and the integrity of the traffic; however, it cannot protect the confidentiality of the data.
The AH protocol works in two modes, as shown in the figure below:
- Transport Mode: Provides authentication for the TCP/UDP header and data.
- Tunnel Mode: Provides authentication for the IP header, TCP/UDP header, and data.
The AH protocol is suitable if providing authentication and integrity is enough without confidentiality. It is worth mentioning that AH is optional in IPsec-v3; however, it is mandatory to implement in IPsec-v2.
Encapsulating Security Payload (ESP)
Encapsulating Security Payload (ESP) provides encryption in addition to authentication and integrity. It works in two modes:
- Transport Mode: Provides security (confidentiality and integrity) for the TCP/UDP header and data.
- Tunnel Mode: Provides security (confidentiality and integrity) for the IP header, TCP/UDP header, and data.
When the TCP/IP protocol was designed, security requirements such as confidentiality and integrity were not a design target. In contrast, availability was the priority as one of the purposes of the Internet is to withstand a nuclear attack, as is evident by the routing protocols adapting quickly when a link goes down. But we need to allow a corporation to use the existing Internet infrastructure to connect its offices securely. The answer lies in setting up a VPN.
A Virtual Private Network (VPN) makes it possible to establish a private connection over a public network. In other words, we can establish a secure connection over an insecure infrastructure.
For instance, in the figure below, we can see a remote office and a remote user connected over a VPN to the main office. A VPN connection requires a VPN client and a VPN server or concentrator. All the traffic between the VPN client and server is encrypted.
The two most common protocols used to establish VPN connections are:
IPsec’s ESP is a perfect protocol for setting up secure tunnels between different networks or a computer and a network. ESP can provide security and integrity of all data transmitted between two points; moreover, even the IP address can be hidden in tunnel mode. Note that the system must be behind a VPN concentrator for the IP address to be hidden. Cisco VPN systems offer IPsec.
Although SSL was created to secure HTTP traffic, SSL/TLS has found its way to establish secure VPN connections with OpenVPN. Using various tools and libraries built around TLS, OpenVPN offers different authentication and encryption mechanisms to establish VPN connections.
Some older protocols that can be used to establish VPN connections are no longer considered secure. One example is Point to Point Tunneling Protocol (PPTP), which is no longer considered secure.
In this room, we have covered various network security protocols essential for the security of transferred data. The most widely used mechanism for securing data over an insecure channel is to add an SSL/TLS wrapper, as we saw in HTTPS, FTPS, POP3S, and SMTPS. The security protocols protect against Man In the Middle (MITM), replay, and eavesdropping attacks.