TLS Handshake

The process by which a client and server negotiate encryption parameters, authenticate each other, and establish session keys for secure communication.

Table of Contents#

1. Overview
2. TLS 1.2 Handshake
3. TLS 1.3 Handshake
4. Cipher Suite Negotiation
5. Certificate Chain Validation
6. Session Resumption
7. TLS Fingerprinting (JA3/JA4)
8. Troubleshooting
9. See Also
10. Sources

1. Overview#

In a TLS handshake, clients and servers exchange certificates, cipher suite requirements, and randomly generated data to create session keys. TLS handshakes are a foundational part of how HTTPS works.

SSL (Secure Sockets Layer) was the original encryption protocol developed for HTTP. SSL was replaced by TLS (Transport Layer Security). SSL handshakes are now called TLS handshakes, although the "SSL" name is still in wide use.

Note: TLS 1.0 and 1.1 are deprecated (RFC 8996, 2021). TLS 1.2 remains supported but TLS 1.3 is preferred for all new deployments.

Key terminology:

• ClientHello - the client's opening message containing supported cipher suites, TLS versions, and extensions
• ServerHello - the server's response selecting cipher suite and TLS version
• Certificate - the server's X.509 certificate (and optionally the client's)
• Key Exchange - the process of deriving shared secret keys
• Finished - verification message confirming the handshake completed successfully
• Forward Secrecy - property ensuring past sessions cannot be decrypted even if the server's long-term key is compromised

2. TLS 1.2 Handshake#

The TLS 1.2 handshake requires two round trips (2-RTT) before application data can be exchanged:

┌───────────┐                ┌───────────┐
│  Client   │                │  Server   │
└─────┬─────┘                └─────┬─────┘
      │                            │
      │                            │
      │ ─────────────────────────> │  ──┐
      │ 1. SYN                     │    │
      │                            │    │
      │                            │    │ TCP
      │ <───────────────────────── │    │
      │ 3. ACK          2. SYN ACK│  ──┘
      │                            │
      │ -------------------------- │
      │                            │
      │ ─────────────────────────> │  ──┐
      │ 4. ClientHello             │    │
      │                            │    │
      │ <───────────────────────── │    │
      │             5. ServerHello │    │
      │                Certificate │    │
      │            ServerHelloDone │    │
      │                            │    │ TLS
      │ ─────────────────────────> │    │
      │ 6. ClientKeyExchange       │    │
      │    ChangeCipherSpec        │    │
      │    Finished                │    │
      │                            │    │
      │ <───────────────────────── │    │
      │        7. ChangeCipherSpec │    │
      │           Finished         │  ──┘

Step-by-step:

1. TCP handshake - standard SYN, SYN-ACK, ACK (1 RTT)
2. ClientHello - client sends supported TLS versions, cipher suites, random bytes, SNI (Server Name Indication), and extensions
3. ServerHello - server selects TLS version and cipher suite, sends its random bytes
4. Certificate - server sends its X.509 certificate chain
5. ServerKeyExchange - server sends Diffie-Hellman parameters (for DHE/ECDHE key exchange)
6. ClientKeyExchange - client sends its key exchange parameters; both sides derive the shared secret
7. ChangeCipherSpec + Finished - both sides switch to encrypted communication and verify with a hash of the handshake

3. TLS 1.3 Handshake#

TLS 1.3 (RFC 8446, 2018) is a major revision that simplifies and secures the handshake process.

Key changes from TLS 1.2:

• 1-RTT handshake - the handshake completes in a single round trip instead of two. The client sends its key share in the ClientHello, and the server responds with its key share, certificate, and Finished message in a single flight.
• 0-RTT resumption - clients that have previously connected to a server can send encrypted application data in the first message (early data), reducing latency for repeat connections. This comes with replay risk and should be used carefully.
• RSA key exchange removed - static RSA key exchange is no longer supported. All key exchanges use ephemeral Diffie-Hellman (DHE or ECDHE), providing mandatory forward secrecy.
• Simplified cipher suites - legacy algorithms (RC4, DES, 3DES, MD5, SHA-1) and compression are removed. Only AEAD cipher suites (AES-GCM, ChaCha20-Poly1305) are supported.
• Encrypted extensions - most handshake messages after ServerHello are encrypted, reducing metadata leakage.

┌───────────┐                ┌───────────┐
│  Client   │                │  Server   │
└─────┬─────┘                └─────┬─────┘
      │                            │
      │ ─────────────────────────> │  ──┐
      │ ClientHello                │    │
      │ + key_share                │    │ 1-RTT
      │                            │    │
      │ <───────────────────────── │    │
      │             ServerHello    │    │
      │             + key_share    │    │
      │             {Certificate}  │    │
      │             {Finished}     │  ──┘
      │                            │
      │ ─────────────────────────> │
      │ {Finished}                 │
      │                            │
      │ <═════════════════════════>│
      │     Application Data       │

3.1. 0-RTT Early Data#

┌───────────┐                ┌───────────┐
│  Client   │                │  Server   │
└─────┬─────┘                └─────┬─────┘
      │                            │
      │ ─────────────────────────> │
      │ ClientHello                │
      │ + key_share                │
      │ + early_data               │  0-RTT
      │ (encrypted app data)       │
      │                            │
      │ <───────────────────────── │
      │             ServerHello    │
      │             + key_share    │
      │             {Finished}     │

Warning: 0-RTT data is vulnerable to replay attacks. Only use it for idempotent requests (e.g., GET, not POST). Servers should implement replay protection or limit 0-RTT to safe operations.

4. Cipher Suite Negotiation#

4.1. How Negotiation Works#

1. The client sends a list of supported cipher suites in the ClientHello, ordered by preference
2. The server selects the strongest cipher suite it supports from the client's list
3. If no common cipher suite exists, the handshake fails with a handshake_failure alert

4.2. TLS 1.2 Cipher Suite Format#

TLS 1.2 cipher suites specify four components: key exchange, authentication, encryption, and MAC.

TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
│     │     │        │    │    │   │
│     │     │        │    │    │   └── MAC/PRF hash
│     │     │        │    │    └────── Mode (GCM = AEAD)
│     │     │        │    └─────────── Key size
│     │     │        └──────────────── Symmetric cipher
│     │     └───────────────────────── Authentication
│     └─────────────────────────────── Key exchange
└───────────────────────────────────── Protocol

Recommended TLS 1.2 cipher suites:

TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

Avoid: anything with CBC mode, RC4, DES, 3DES, MD5, SHA-1 MAC, or static RSA key exchange.

4.3. TLS 1.3 Cipher Suites#

TLS 1.3 separates the cipher suite from the key exchange. Cipher suites only specify the AEAD algorithm and hash:

TLS_AES_256_GCM_SHA384
TLS_CHACHA20_POLY1305_SHA256
TLS_AES_128_GCM_SHA256

Key exchange is negotiated separately via the supported_groups extension (X25519, P-256, P-384).

4.4. Checking Cipher Suites#

# Check which cipher suites a server supports
nmap --script ssl-enum-ciphers -p 443 <target>

# Using OpenSSL
openssl s_client -connect <target>:443 -tls1_3

# Show the negotiated cipher
openssl s_client -connect <target>:443 2>/dev/null | grep 'Cipher is'

5. Certificate Chain Validation#

5.1. Chain of Trust#

When a server presents its certificate, the client validates the entire certificate chain:

Root CA (self-signed, pre-installed in OS/browser trust store)
  └── Intermediate CA (signed by Root CA)
        └── Server Certificate (signed by Intermediate CA)

5.2. Validation Steps#

The client performs these checks in order:

1. Chain building - construct a chain from the server certificate to a trusted root CA
2. Signature verification - verify each certificate's signature using the issuer's public key
3. Validity period - check that each certificate's notBefore and notAfter dates cover the current time
4. Revocation check - verify the certificate has not been revoked via CRL (Certificate Revocation List) or OCSP (Online Certificate Status Protocol)
5. Name matching - verify the server's hostname matches the certificate's Subject Alternative Name (SAN) or Common Name (CN)
6. Key usage - verify the certificate's key usage extensions permit TLS server authentication
7. Path constraints - verify intermediate certificates have the CA:TRUE basic constraint and appropriate path length limits

5.3. Common Validation Failures#

# Test certificate chain
openssl s_client -connect <target>:443 -showcerts

# Verify a certificate chain manually
openssl verify -CAfile ca-bundle.crt -untrusted intermediate.crt server.crt

5.4. Certificate Transparency (CT)#

Certificate Transparency is a public logging system that records all issued certificates. Browsers require certificates to include Signed Certificate Timestamps (SCTs) from CT logs, preventing CAs from issuing certificates secretly.

# Check CT logs for a domain
# Use crt.sh: https://crt.sh/?q=example.com

5.5. OCSP Stapling#

Instead of the client querying the CA's OCSP responder directly, the server can "staple" a signed OCSP response to the TLS handshake:

# Check if a server supports OCSP stapling
openssl s_client -connect <target>:443 -status 2>/dev/null | grep "OCSP Response"

Benefits: reduces handshake latency, improves privacy (the CA does not see which sites clients visit).

6. Session Resumption#

Session resumption allows clients and servers to skip the full handshake for subsequent connections, reducing latency.

6.1. TLS 1.2 Session Resumption#

Session IDs:

1. After a full handshake, the server assigns a session ID
2. On reconnection, the client sends the session ID in the ClientHello
3. If the server still has the session cached, both sides skip key exchange (abbreviated handshake, 1 RTT)

Session Tickets (RFC 5077):

1. After a full handshake, the server encrypts the session state into a ticket
2. The server sends the ticket to the client
3. On reconnection, the client sends the ticket back
4. The server decrypts the ticket to restore the session (no server-side state needed)

6.2. TLS 1.3 Session Resumption (PSK)#

TLS 1.3 uses Pre-Shared Keys (PSK) for session resumption:

1. After a successful handshake, the server sends a NewSessionTicket message containing a PSK identity
2. On reconnection, the client includes the PSK in the ClientHello
3. The server validates the PSK and uses it for key derivation
4. Combined with key_share, this provides both resumption and forward secrecy (PSK-DHE mode)

6.3. Security Considerations#

• Session tickets in TLS 1.2 break forward secrecy if the ticket encryption key is compromised; rotate ticket keys frequently
• 0-RTT in TLS 1.3 is vulnerable to replay attacks; servers must implement anti-replay mechanisms
• PSK-only mode (without DHE) does not provide forward secrecy; always prefer PSK-DHE

7. TLS Fingerprinting (JA3/JA4)#

The ClientHello message contains fields that vary between TLS implementations: supported cipher suites, extensions, elliptic curves, signature algorithms, and their order. This creates a fingerprint unique to each TLS library.

JA3 (2017) hashes five fields from the ClientHello into an MD5 digest:

• TLS version, cipher suites, extensions, elliptic curves, elliptic curve point formats

JA4 (2023) extends this with additional signals:

• ALPN, signature algorithms, TLS version negotiation, SNI presence, extension ordering

These fingerprints allow network monitors (firewalls, DPI, CDNs) to identify the software behind a TLS connection without decrypting it:

Censorship implications: DPI systems use JA3/JA4 to detect non-browser TLS clients (proxies, VPN tools, custom tunnel software). A Go-based proxy connecting to port 443 has a JA3 fingerprint that looks nothing like Chrome, which creates an immediate flag.

Countermeasure: uTLS is a library that mimics the ClientHello of real browsers. Tools like Xray and sing-box use uTLS with fingerprint: "chrome" to produce a ClientHello identical to Chrome's, defeating JA3/JA4 classification.

# Capture and analyze a TLS fingerprint
sudo tcpdump -i eth0 -c 1 -w /tmp/hello.pcap 'tcp port 443 and tcp[((tcp[12:1] & 0xf0) >> 2):1] = 0x16'
# Analyze with Wireshark or ja4 tools

Troubleshooting#

See Also#

• OpenSSL
• DNS Encryption

Sources#

• RFC 8446 - TLS 1.3
• RFC 5246 - TLS 1.2
• RFC 8996 - Deprecating TLS 1.0 and 1.1
• RFC 5077 - TLS Session Resumption
• Mozilla SSL Configuration Generator
• JA3 Fingerprinting (Salesforce)
• JA4 Fingerprinting (FoxIO)