I stumbled across what I assume is a performance optimisation being used by Twitter and wondered how much I could optimise my site using a similar principle. The changes would result in no difference to functionality but yield a slightly smaller payload for the page meaning faster page load times!


The Twitter optimisation

Whilst putting some new functionality through its paces on the test site for my HTTP header analysis service, https://securityheaders.io, I noticed something odd about the Twitter score. They'd dropped from an A grade to a B grade, but only on the test site. After some quick investigation I realised I hadn't broken everything (phew!) and this was actually caused by their CSP header not being issued and reducing their score. This was really odd at first because the test site runs on the same infrastructure as the live site and I couldn't figure out why. After much head scratching I realised the only difference with the request was that I'd changed the User Agent (UA) string. Previously I was using the latest Chrome browser UA string so that my scanner looked like a browser but after senior security researcher Ofer Gayer reached out to me from Incapsula regarding how that looked, I decided to try a change on the test site. Using a browser UA string will 'trick' sites into thinking my scanner is a browser and treat me like one, which is what I want, but this looks bad from the perspective of people like Incapsula when I'm effectively running a bot that's trying to pretend to be a browser. The change was simple, I went from this:

curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.97 Safari/537.36');

To this:

curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (compatible; SecurityHeaders/1.0; +https://securityheaders.io/about/)');

This is beneficial because sites that get scanned by me and pick it up in their logs have some opportunity to understand what my service is and why it's connecting to them and issuing HTTP requests. Right now I'm testing a hybrid of these headers to look enough like a browser to have the CSP issued but still also link to my About page so that sites can have some information to go on rather than just ban me. It did get me thinking though and the only reason that I could think of for Twitter to do this is to save some bytes on the wire as their CSP header is pretty big! The response headers for the request with my custom UA string were only 1.23KB, you can see them here:

HTTP/1.1 200 OK
cache-control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0
content-length: 253735
content-type: text/html;charset=utf-8
date: Sun, 31 Jan 2016 17:08:57 GMT
expires: Tue, 31 Mar 1981 05:00:00 GMT
last-modified: Sun, 31 Jan 2016 17:08:57 GMT
pragma: no-cache
server: tsa_a
set-cookie: _twitter_sess=BAh7CSIKZmxhc2hJQzonQWN0aW9uQ29udHJvbGxlcjo6Rmxhc2g6OkZsYXNo%250ASGFzaHsABjoKQHVzZWR7ADoPY3JlYXRlZF9hdGwrCAKtqJhSAToMY3NyZl9p%250AZCIlNjhiZjdmNWFkN2ViNjRiMGM2NmMxMzE4ZTlmOTZlY2U6B2lkIiUyZTcz%250AOTk2OGJlOTFiZDQyNDQzMGY4ZjNkODIzZjk1Mw%253D%253D--cc35ace2184c6e4b3787ca7af2811c7bb0aa8115; Path=/; Domain=.twitter.com; Secure; HTTPOnly
set-cookie: ua="m2,msw"; Expires=Sun, 31 Jan 2016 18:08:57 UTC; Path=/; Domain=.twitter.com; Secure; HTTPOnly
set-cookie: guest_id=v1%3A145426013717532576; Domain=.twitter.com; Path=/; Expires=Tue, 30-Jan-2018 17:08:57 UTC
status: 200 OK
strict-transport-security: max-age=631138519
x-connection-hash: 25ab8fd22ca280589690ab5a168960a0
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-response-time: 228
x-transaction: dec22fc2ea127fd8
x-twitter-response-tags: BouncerCompliant
x-ua-compatible: IE=edge,chrome=1
x-xss-protection: 1; mode=block

When using the browser UA string though the size of the response headers jumped up to 3.54KB, a 287% increase in size meaning their CSP is 2.31KB in size!

HTTP/1.1 200 OK
cache-control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0
content-length: 251409
content-security-policy: script-src https://connect.facebook.net https://cm.g.doubleclick.net https://ssl.google-analytics.com https://graph.facebook.com https://twitter.com 'unsafe-eval' https://*.twimg.com https://api.twitter.com https://analytics.twitter.com https://ton.twitter.com https://syndication.twitter.com https://www.google.com https://t.tellapart.com https://platform.twitter.com 'nonce-IIFzv/dLKvTapuEQcsrU/A==' https://www.google-analytics.com 'self'; frame-ancestors 'self'; font-src https://twitter.com https://*.twimg.com data: https://ton.twitter.com https://fonts.gstatic.com https://maxcdn.bootstrapcdn.com https://netdna.bootstrapcdn.com 'self'; media-src https://twitter.com https://*.twimg.com https://ton.twitter.com blob: 'self'; connect-src https://graph.facebook.com https://media4.giphy.com https://media0.giphy.com https://pay.twitter.com https://analytics.twitter.com https://media.riffsy.com https://media.giphy.com https://media3.giphy.com https://upload.twitter.com https://media2.giphy.com https://media1.giphy.com 'self'; style-src https://fonts.googleapis.com https://twitter.com https://*.twimg.com https://translate.googleapis.com https://ton.twitter.com 'unsafe-inline' https://platform.twitter.com https://maxcdn.bootstrapcdn.com https://netdna.bootstrapcdn.com 'self'; object-src https://twitter.com https://pbs.twimg.com; default-src 'self'; frame-src https://staticxx.facebook.com https://twitter.com https://*.twimg.com https://player.vimeo.com https://pay.twitter.com https://www.facebook.com https://ton.twitter.com https://syndication.twitter.com https://vine.co twitter: https://www.youtube.com https://platform.twitter.com https://upload.twitter.com https://s-static.ak.facebook.com 'self' https://donate.twitter.com; img-src https://graph.facebook.com https://twitter.com https://*.twimg.com https://media4.giphy.com data: https://media0.giphy.com https://fbcdn-profile-a.akamaihd.net https://www.facebook.com https://ton.twitter.com https://*.fbcdn.net https://syndication.twitter.com https://media.riffsy.com https://www.google.com https://media.giphy.com https://stats.g.doubleclick.net https://media3.giphy.com https://www.google-analytics.com blob: https://media2.giphy.com https://media1.giphy.com 'self'; report-uri https://twitter.com/i/csp_report?a=NVQWGYLXFVZXO2LGOQ%3D%3D%3D%3D%3D%3D&ro=false;
content-type: text/html;charset=utf-8
date: Sun, 31 Jan 2016 19:40:53 GMT
expires: Tue, 31 Mar 1981 05:00:00 GMT
last-modified: Sun, 31 Jan 2016 19:40:53 GMT
pragma: no-cache
server: tsa_a
set-cookie: _twitter_sess=BAh7CSIKZmxhc2hJQzonQWN0aW9uQ29udHJvbGxlcjo6Rmxhc2g6OkZsYXNo%250ASGFzaHsABjoKQHVzZWR7ADoPY3JlYXRlZF9hdGwrCBvFM5lSAToMY3NyZl9p%250AZCIlZDc0MjZhYTVhYWU3M2U2MjBhZGYzMzE5NmExMWUzNjU6B2lkIiU3MmFl%250AZWJjZTVkMTBiN2M3YzM0ZTkyNmZhZDU4YjljNw%253D%253D--c745f41139763bd70368f80e7c78f4680038553e; Path=/; Domain=.twitter.com; Secure; HTTPOnly
set-cookie: ua="f5,m2,m5,rweb,msw"; Expires=Sun, 31 Jan 2016 20:40:53 UTC; Path=/; Domain=.twitter.com; Secure; HTTPOnly
set-cookie: guest_id=v1%3A145426925289063837; Domain=.twitter.com; Path=/; Expires=Tue, 30-Jan-2018 19:40:53 UTC
status: 200 OK
strict-transport-security: max-age=631138519
x-connection-hash: 025dfdae11e2f1f18699e2f8380e1c81
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-response-time: 161
x-transaction: 53730aefd0093e95
x-twitter-response-tags: BouncerCompliant
x-ua-compatible: IE=edge,chrome=1
x-xss-protection: 1; mode=block

This is a thought shared by Neil Matatall who worked on the original deployment of CSP at Twitter.


As an interesting side note Dropbox also selectively issue their CSP based on the browser version for compatibility reasons!


So, that basically got me thinking. It's not a huge amount to save per connection, 2.31KB, but it's a saving nonetheless and when you get the kind of traffic that I imagine Twitter or Dropbox get, that's probably going to add up to something quite significant in a very short amount of time.


Dropping headers when they're not needed

After thinking about the saving of not issuing headers where they aren't needed I realised that I issue all of my security based headers at a global level in my config. They get set on every single asset that gets loaded be that images, scripts, style sheets or pages themselves. Considering the headers only have any impact on a page load that seemed like a total waste of precious bandwidth and it just had to be resolved. My home page was weighing in at 186KB for a full load so I decided to drop the unnecessary headers and see what impact that would have.

homepage with headers


There weren't too many assets being loaded due to my already minimal homepage but there were still a few there and removing the headers from them saved me some bandwidth.

homepage headers removed


That's a 21KB saving in bandwidth by removing headers that served no purpose! You can see the page remained the same size at 6.5KB but the assets loaded, like Aio.min.css reduced in size due to the removal of the headers from the response. A really good example is my ga.min.js file which loads my Google Analytics. This file dropped from 4.1KB to 532B by removing the headers which means it was ~8x larger than it needed to be because of the headers. I probably should have done that right from the beginning but hey, live and learn! On a page with a lot of other images or other assets this saving would have been ever bigger. Even still, on a HTTP/1.0 or HTTP/1.1 connection there is no compression of the response headers so 1,000,000 hits on my homepage with these improvements would save 21GB of bandwidth. Not too bad.


Remove redundant assets

Whilst doing my optimisations I noticed that there were a few files being loaded on the homepage that weren't needed. One was a library for the theme that I based my blog off that I simply didn't use or need and the others were the JS and CSS files for my social sharing buttons that were included by default but never used. Off with their heads!

redundant assets removed


That's another 52KB shaved off the page and I still haven't sacrificed anything, this was just wasted bandwidth! Not bad for a quick 10 minutes of work.


Results

In the end I cut the size of my homepage down by a quite impressive 39%! There was no real need for me to do this but sometimes these little interesting things pop up and they result in you learning something new or thinking about something you hadn't thought of before. If something grabs your interest then go look into it and do some digging around. For me that 39% wasn't too significant but who knows what you might find.