TLS Basics and a Real Traefik “Not secure” Incident

TL;DR

Symptom: https://traefik.example.local/dashboard/ shows Not secure in Chrome/Safari; Firefox works fine.
Root cause: macOS Trust validator rejects wildcard cert *.example.local for subdomain traefik.example.local (OpenSSL accepts it, Apple Trust does not).
Fix: Issue explicit hostname certificate for traefik.example.local instead of betting on wildcard.
In one sentence: Validator rules determine everything, not "the cert is broken."

This was not "the certificate is broken." It was "the certificate is validated by a different engine." OpenSSL said OK. Apple Trust said no. Chrome/Safari follow Apple Trust.

This post has two parts: TLS basics + the real troubleshooting chain.

TLS basics (only the essentials)

Let’s keep the core pieces:

Certificate chain: leaf → intermediate (optional) → root CA. Browsers trust roots, not leaves.
SAN (Subject Alternative Name): hostname matching is done here; CN is legacy.
Trust store: macOS uses System Keychain; Firefox has its own store; Chrome/Safari use system trust.
Validation: the same cert can be accepted by one validator and rejected by another.

One sentence: “Valid” does not mean “accepted by your current validator.”

Problem Symptoms and Impact Scope

Accessing https://traefik.example.local/dashboard/#/ showed Not secure in Chrome/Safari.

Impact scope:

❌ Chrome (relies on macOS System Keychain)
❌ Safari (relies on macOS System Keychain)
✅ Firefox (uses its own cert store, unaffected)

Reproduction prerequisites:

macOS version: Sonoma 14.x+ (other versions may behave differently)
CA already installed in macOS Keychain and set to "Always Trust"
Certificate issued by mkcert (version 1.4.4)
Service running on 10.0.0.100

I needed three answers:

Does DNS point to the right server?
Is the cert chain complete and SAN correct?
Does the system trust engine accept the hostname?

Hypothesis Checklist (in elimination order)

Before troubleshooting, list possible root causes:

H1: CA not properly installed in system trust store
- Validation: Check macOS Keychain, confirm CA cert exists and is "Always Trust"
- Expected: If CA missing, all browsers would fail (but Firefox works, so low probability)
H2: Incomplete certificate chain (missing intermediate cert)
- Validation: openssl s_client -showcerts to check chain depth
- Expected: Self-signed CA usually has only two levels (leaf + root), mkcert default is complete
H3: SAN does not include target domain
- Validation: openssl x509 -text to check Subject Alternative Name field
- Expected: If SAN missing or mismatched, OpenSSL would also fail (but curl passed, so low probability)
H4: Validator rule differences (OpenSSL vs Apple Trust)
- Validation: Use security verify-cert to simulate macOS system validation
- Expected: This is the most likely root cause — different validators have different hostname matching rules for wildcards

Validate each hypothesis in order.

Validation Flow Visualization

This diagram shows the 4-layer validation progression (no execution, conceptual only):

bash
#!/usr/bin/env bash
# Validation flow: 4-layer check to identify which layer rejects

cat <<'EOF'

┌─────────────────────────────────────────────────────────┐
│  Layer 1: DNS Resolution                                │
│  dig traefik.example.local +short                       │
│  ✅ Result: 10.0.0.100                                   │
└─────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────┐
│  Layer 2: Certificate Content (SAN)                     │
│  openssl s_client + grep "Subject Alternative Name"     │
│  ✅ Result: DNS:*.example.local, DNS:example.local      │
└─────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────┐
│  Layer 3: OpenSSL Validator (Linux/curl)                │
│  curl -Iv https://traefik.example.local/                │
│  ✅ Result: SSL certificate verify ok                   │
└─────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────┐
│  Layer 4: Apple Trust Validator (macOS)                 │
│  security verify-cert -n traefik.example.local          │
│  ❌ Result: Host name mismatch                          │
└─────────────────────────────────────────────────────────┘

Key finding: Layer 3 ✅ but Layer 4 ❌
→ Different validators have different hostname matching rules for wildcards
→ Chrome/Safari rely on Apple Trust (Layer 4)

EOF

The troubleshooting chain (in order)

1) DNS check

bash
dig traefik.example.local +short

Expected output:

text
10.0.0.100

Observation: DNS points correctly, network layer is fine.

Conclusion: Network connectivity ruled out, proceed to certificate layer.

2) Inspect the cert chain and SAN

bash
openssl s_client -connect traefik.example.local:443 -servername traefik.example.local -showcerts 2>/dev/null | openssl x509 -text -noout | grep -A2 "Subject Alternative Name"

Expected output (key part):

text
X509v3 Subject Alternative Name:
    DNS:*.example.local, DNS:example.local

Observation: SAN contains wildcard *.example.local, which should theoretically match traefik.example.local per RFC 6125.

Conclusion: Certificate fields are correct, H3 (SAN missing) eliminated.

3) OpenSSL validation (passes)

bash
curl -Iv https://traefik.example.local/ 2>&1 | grep -E "SSL certificate verify|subject:"

Expected output:

text
* SSL certificate verify ok.
* subject: CN=*.example.local

Observation: OpenSSL validator considers cert valid, hostname matching passed.

Conclusion: OpenSSL finds no issues, H2 (incomplete chain) eliminated.

4) macOS system validation (fails)

bash
# Export cert to temp file first (if not already done)
echo | openssl s_client -connect traefik.example.local:443 -servername traefik.example.local 2>/dev/null | \
  openssl x509 > /tmp/traefik-leaf.pem

# macOS system-level validation
security verify-cert -c /tmp/traefik-leaf.pem -p ssl -n traefik.example.local -L -v

Expected output (key error):

text
...certificate verification failed
...Host name mismatch

Observation: Apple Trust explicitly rejects this wildcard match for traefik.example.local.

Conclusion: Apple Trust considers this SAN unable to match the target domain. This is the pivotal point: OpenSSL OK, Apple Trust not OK. Chrome/Safari rely on Apple Trust. H4 (validator rule differences) confirmed.

Root cause

Not a missing CA, not a broken chain. The real cause:

macOS hostname validation did not accept the wildcard cert for traefik.example.local.

This is not theory; it is the system validator's explicit result.

Minimal, reliable fix

Stop relying on the wildcard. Issue an explicit hostname certificate.

bash
mkcert -cert-file traefik.example.local.pem -key-file traefik.example.local-key.pem "traefik.example.local"

If you want to keep wildcard + apex too:

bash
mkcert -cert-file traefik.example.local.pem -key-file traefik.example.local-key.pem \
  "traefik.example.local" "*.example.local" "example.local"

Then replace the Traefik TLS cert and restart the service.

A reusable checklist

DNS resolution is correct (dig)
SAN includes the target hostname (openssl x509 -text)
OpenSSL validation (curl -Iv)
macOS validation (security verify-cert)
Browser trust store alignment

Retrospective and Prevention

One-Sentence Lesson

The most valuable evidence: The Host name mismatch output from security verify-cert — it directly proved the issue was not "CA trust" but "hostname validation rules."

If you remember one thing: Don't judge cert correctness by "it opens on my machine." Different validators (OpenSSL / Apple Trust / Firefox NSS) can reach different conclusions on the same cert.

Prevention Checklist (for next time you issue certs)

For internal self-signed CA (mkcert/self-built CA):

Prefer explicit hostnames over betting on wildcard cross-platform compatibility
If you must use wildcards, pre-validate with security verify-cert on macOS
Keep issuance command records (with complete SAN list) for easy re-issuance

For public certs (Let's Encrypt, etc.):

Request both example.com and *.example.com if you need to cover apex and subdomains
Validate on at least 3 platforms: Linux (OpenSSL) / macOS (Apple Trust) / Windows (Schannel)

Validation template (reusable script):

bash
#!/usr/bin/env bash
# Quick validation of cert behavior across validators

DOMAIN="traefik.example.local"

echo "==> OpenSSL validation"
curl -Iv https://${DOMAIN}/ 2>&1 | grep -E "SSL certificate verify|subject:"

echo "==> macOS validation (must run on macOS)"
# Export cert first
echo | openssl s_client -connect ${DOMAIN}:443 -servername ${DOMAIN} 2>/dev/null | \
  openssl x509 > /tmp/cert-to-verify.pem
# System validation
security verify-cert -c /tmp/cert-to-verify.pem -p ssl -n ${DOMAIN} -L -v

echo "==> Check SAN"
openssl s_client -connect ${DOMAIN}:443 -servername ${DOMAIN} </dev/null 2>/dev/null | \
  openssl x509 -text -noout | grep -A2 "Subject Alternative Name"

Final Conclusion

"I already installed the CA" is not the end of the story. What matters is which validator you are actually using.

This time the fix is simple and clean: issue an explicit cert for the host and the warning disappears.