I've just pushed the next major update to https://report-uri.io and there are some great new features that I'm really excited to be launching! The service has come a long way in the 12 months it's been running and usage has soared. Here is your next batch of new features, upgrades and improvements!
Subdomains for everyone!
The use of report-uri.io has exploded in recent months and with thousands of sites reporting through the service, sending reports that number in the hundreds of millions a month, it's fair to say that it handles a pretty hefty amount of traffic. The existing structure of the address that reports are sent to started to cause some problems. This is how it used to look:
https://report-uri.io/report/ScottHelme
As all of the traffic is being sent to the bare domain for the site, report-uri.io, the only opportunity I have to redirect or re-route traffic is when it hits my load balancers. These same load balancers also handle traffic for serving the site itself as well as reporting so when they get busy, the page load times start creeping up. I needed more flexibility to route traffic and I needed to do it before it even hit my infrastructure. I needed to do it at the DNS level:
https://scotthelme.report-uri.io/r/default/csp/enforce
By moving each reporting address to a subdomain I can now control the flow of traffic at the DNS level. Each subdomain can resolve to a shared group of servers used for handling reports and the site itself can be hosted from a different set of servers altogether. Now, adverse load on one will not affect the other. Not only that but it allows me to handle 'noisy neighbours' a lot more efficiently. If a site joins and they're sending an overwhelming amount of reports, either due to misconfiguration or just the sheer amount of traffic they get, I can control the flow of traffic using DNS (this happened and I have a big blog coming soon). To start with, I can take them away from the shared reporting infrastructure and put them onto dedicated hardware. If they overwhelm that there is no detriment to the rest of my users. This also means that for really big sites I can offer a better level of service by segregating them from the main infrastructure. On top of that, if someone is receiving an abusive level of reports, I can simply stop resolving their subdomain and the traffic stops making it through to me. At present the only way I can handle this, and do handle this, is by dropping their reports at my load balancers. At this point though it's already too late as they're using up resources in the form of network bandwidth and CPU cycles just to drop the inbound report.
The new subdomains offer me a great deal of flexibility and require a simple change in your policy to paste in the new reporting address from the Setup page in your account. I will continue to support the old reporting addresses for several weeks before I drop incoming reports in favour of the new addresses. Because subdomains aren't case sensitive, if you have upper case characters in yours you will need to change it to the lower case version when you switch to using the subdomain. Simply use the form on the Setup page and submit your desired value there, even if it is only a lower case version of your existing one.
View only the reports you want to see
The existing method of viewing reports was ok and gave a basic table that allowed you to view, and paginate through where needed, your reports.
The time filters across the top allowed for some control of the data you were looking at but they just weren't enough. I wanted better. I wanted much better.
With the updated report table you now have considerably more options when it comes to filtering your reports!
- You can filter reports based on whether the policy was enforced or a report-only policy.
- The time filter can now be customised from as small a unit as a specific hour right up to a whole, specific month.
- If you collect reports for multiple domains or subdomains, you can now filter reports for a specific one.
- You can even specify a particular path on that domain that you want to view reports for!
- Only want to see reports for the violation of a specific directive, say script-src? We've got you covered.
- Want to see only specific blocked URIs? Yep, that's there.
- That also includes specific paths for the blocked URI too.
- Lastly, you can filter down to only reports from a specific browser.
With this level of granularity you can really start to drill down into actual, specific problems on your site and hopefully debug issues far easier than before. Unfortunately, some of these filters will only work on data collected from the date of the update onwards, but the time based filters will work historically. This is due to the new properties I'm now adding to each report when I save it in Table Storage for me to then filter on.
Cutting out more of the noise
The new filters for refining the reports we see are great and really help to focus in on specific problems but something that's just as important is filtering out the noise on the way in. Because the inbound filter list has grown, and because I feel it's important enough to deserve it's own menu item now, it has one!
In the filters menu there are all of the same filters you had before with 2 new ones. The ability to filter out reports with a blocked-uri of 'data' and 'about'. These seem to be the next most common noisy reports but filtering these will depend on your individual use case so they will default to off. If these reports are noisy for you, then you can choose to filter them.
The other thing worth noting here is that you must now define the list of hosts that you expect to collect reports for and the method to do this has changed slightly. This field is still a space separated list of hosts but you need to specify all domains/subdomains that you wish to collect reports for. As an example, if I wanted to collect reports from scotthelme.co.uk, blog.scotthelme.co.uk and shop.scotthelme.co.uk then I'd specify:
scotthelme.co.uk blog.scotthelme.co.uk shop.scotthelme.co.uk
The main change here is that defining scotthelme.co.uk will no longer allow the collection of reports for all subdomains, each of them have to be specified. You can of course still specify multiple different domains like I do.
scotthelme.co.uk report-uri.io securityheaders.io hpkp.scotthelme.co.uk
The main reason for this is that you can now filter on each of these entries when viewing reports to allow you to refine the data you're viewing even further.
New report-uri format
The format of the address that you use as your report-uri directive is changing with the new update. The main change is the shift to subdomains as mentioned above but I've also made a few other changes to provide far more flexibility later too.
As you can see, I now have my own custom subdomain in there and there are some updates to the path too. We now have:
- /r - Shorter than /report.
- /default - This is the application name and is 'default' at present.
- /csp or /hpkp - These provide information on the expected report type.
- /enforce or /reportOnly - These provides details on the whether the header was enforced or not.
In a subsequent update you will be able to define custom app names for filtering your reports on, say you want to group multiple domains together, but for now 'default' is the only accepted value. As we all have our own subdomain there is also a new option to check that the DNS entry for your subdomain exists, and if not it will be created.
The old address format will be supported for 4 weeks from the date of this blog post and then any reports sent to those addresses will no longer be collected. At your earliest possibility please switch over to your new subdomain address and hit the Check button shown above to check your DNS record is ready to go!
Coming up next
These are the biggest changes that I wanted to announce and cover the new features that users will need and want to be aware of. There are quite a few technical changes behind the scenes that will result in a considerable performance increase whilst using the site and also a reduction in cost on my side due to the lower resource utilisation. These will be covered in a separate blog with a more technical focus and possibly be broken out into a small series. As the service has grown so fast in its first year I've encountered problems in almost all aspects of operation. The number of inbound reports is now in the tens of millions every single week and that number is only heading in one direction, up! As a result of this explosive growth I've had to make technical changes in order to accommodate levels of traffic I simply hadn't ever expected. The service is constantly growing and gaining strength and this latest update offers me a huge amount of flexibility to adapt going forwards. I'm really looking forwards to the next 12 months!
Scott.
Short URL: https://scotthel.me/ruu