I've continued to see an incredible amount of growth for report-uri.io and it is still exceeding all of my expectations. So that I can keep the service up and running I need to make a few changes that I will outline in this post.
Still growing!
The site is continuing to get an awesome level of attention and is steadily growing its user base despite the fact that I don't advertise it anywhere. This shows that there is a real need for a CSP and/or HPKP reporting service and is why I want to keep it running. To do that though, I need to start introducing some sensible limits on usage now that things are really taking off. None of these limits should impact the usefulness of reporting, that's the last thing I want to do, but there are some options available to reduce the burden of running the service whilst still ensuring it remains useful. This is what I have in mind.
Rate limiting inbound reports
Previously there was no rate limit on inbound reports so it seemed like the first and most obvious aspect to address. Even if this were a commercial service, it seems unlikely that there wouldn't be some kind of rate limit in place. To start out, I introduced the limit at 1,000,000 reports per month. This allows for a little over 33,333 reports to be collected per day and I think this is more than enough to get the necessary information about what's happening on your site. The allowance will also be broken down into a rate limit rather than a total so that you can't burn through your entire monthly allowance in a day because of something like a misconfiguration or a genuine problem. I want the service to continue to be useful and this seems like a good way to tackle it. Just to show the sheer scale of traffic I'm currently handling, even after slowly introducing the rate limit, I'm still handling close to 100,000,000 transactions in an average week!
No more infinite data retention
Right now there is no maximum retention on data and earlier this year I passed through 1TB of data on disk and I'm quickly approaching 2TB. It's also time to do something about that! To tackle this, I will be introducing some time limits on how granular the report data will be going back over time.
On the Reports pages:
- The hourly view will be retained for 7 days.
- The daily view will be retained for 30 days.
- The weekly view will be retained for 8 weeks.
- The monthly view will be retained for 3 months.
There won't be any changes to the graphs page as the storage requirements for that are minimal. The main aim here is to reduce the granularity over time and to ease away the burden of storing so much data. I could also do something like remove the hourly view, but I'd like to know how useful people find it to be able to inspect a single hour compared to an entire day, for example. Please do let me know your thoughts in the comments below.
Help support the service
As much as I've built this service to be as efficient as possible, recently optimising it even more, and as cheap as cloud hosting is, it's still costing money! The above changes are intended to reduce the cost of hosting whilst not having a negative impact on the usefulness of the service. In honesty I don't think they will have any real negative impact at all but will help to reduce costs in a big way. That said, the rate at which they're climbing right now is what I need to address, not necessarily the current monthly fees.
As a result, I'd like to outline a few ways that others can help to support the service by making contributions. If you use the service or find it useful, please consider helping out!
PayPal: https://www.paypal.me/scotthelme
BitCoin: 1ScottkY3kbPi6fQEgAaGQoRfz4MxJLGZ
DigitalOcean: https://m.do.co/c/6a2e0aeeb5bd
The DigitalOcean referral link will also give you $10 of free credit when you open your account so if you're planning on using them, give that link a click and help me out. Any amount that you can donate would be greatly appreciated and go directly to covering the costs of running the service, be that a one off donation or continued support. Thanks in advance to anyone that can help out!