The continued growth of report-uri.io has been amazing and every week I'm genuinely surprised at the sites I see signing up and enabling reporting. I've added some pretty big features to the site and now I need to address the continued growth so that the site remains sustainable.
Growth like I never imagined
It seems that each time I write about report-uri.io things have not only continued to grow and get better but the rate at which they're doing so is also increasing too. The site has had some pretty large successes and it hasn't even been around for 2 years yet. Here's a quick timeline of the major events.
May 20, 2015 Launch day, looking back the site had a rather crude UI but the basic features were there and it was launched as a beta.
July 22, 2015 Major update, came out of beta, new branding, all of the tools, CSP and HPKP split, filters, data deletion and much more.
September 01, 2015 3 months overview, published data on the first 3 months including performance/cost breakdowns. ~250,000 reports per week.
October 18, 2015 Further imrpovements, largely around the GUI with better aesthetics, filters, pagination, reporting sorting, querying and more.
March 25, 2016 Next major update, subdomains for users, more advanced filters, querying on reports page, new reporting addresses.
April 11, 2016 Performance optimisations, query projection, memcached, cached queries and gzip to boost performance.
September 19, 2016 Limits introduced, rate limiting on inbound reports and data retention. ~25 million reports per week.
To go from launching the service to having ~250,000 reports per week in 3 months was quite the feat I thought. Here is my database activity per week.
Then I went on to ~25,000,000 reports per week over the next 12 months and I was amazed, especially considering this is reports processed after rate-limiting. The received amount is considerably higher.
The constant improvements and optimisations in the service have been to cope with the ever increasing load and the new features are because I want it to be as useful as possible. Even with all of the optimisations in the world though, the traffic keeps on coming!
The numbers kept on growing and then one day the Azure dashboard cut off my weekly graph and would only allow me to view data in an hourly window. Yes, the number looks a lot smaller here but that's actually because we're now looking at transactions per hour. I took this screenshot during my quieter times in the UK during the day as I have a largely American traffic base, but even at this level that still represents over 300,000,000 transactions per week. With my recent optimisations that means I'm now processing over 80,000,000 reports per week! As I mentioned earlier I do have rate-limiting in place, and I now have some of that taking place upstream (more on that in a subsequent blog) but I'm now receiving an average of ~1,300 reports per second that need some kind of action, be that processing or dropping. Here is a slightly busier time period for the same graph.
The increasing financial burden
I have built the service to be as efficient as possible, making some pretty big improvements along the way too, but even with the best I can do at present, the costs do keep increasing with the traffic. The number of servers I have in DigitalOcean is rising to keep a reasonable response time and to be able to absorb small spikes in traffic and my Azure bill for Table Storage increases month on month as I use more bandwidth, disk space and transactions. As much as I don't want to, the time has come to ask the community and the wider industry for support. I built report-uri.io as a free tool, and it will remain free for as long as it can, but I can no longer do that alone. After much thought and deliberation with various people from different walks of life, I've chosen a path that many don't think will work, but I'm placing some faith in the good will of my users, their visitors and anyone else who may be able to help. I'm introducing optional, paid subscriptions to the site.
Optional Subscriptions
The idea behind this is pretty simple for me, and it can be broken down into two distinct parts. First, I started report-uri.io as a free service because I wanted it to be as much us as possible to anyone who could use it with no financial barriers standing in the way of better security. I will try and keep it that way for as long as I can. Second, implementing a billing system is a pretty complicated task. I'd have to tie it to account levels, their features, rate limits and all kinds of other metrics. I'd have to handle monthly payments and hooking those in to the back end to make it all run smoothly would be a lot of development and testing work. A lot of work that would take a lot of time away from actually building new features that would increase the usefulness of the service. I don't want to spend time on building a mechanism to reliably charge people money when I can spend that time on refining a new feature to improve their security and that of their visitors. To realise that ambition however, I need help.
These are the subscription levels that I'm planning to introduce to report-uri.io during the account creation process. I've done research on similar services with similar offerings and looked at the costs involved with hosting and maintaining the service and these numbers seem fair and reasonable. I'd like to stress again though that these subscriptions are 100% optional. There is a button at the bottom that says 'Continue Without Paying' and you're entirely welcome to use that. What I'm hoping for is that enough people who can pay will choose to support me that I don't have to require people to pay to use the service. Even if you can't support me at one of the higher levels, supporting me on a lower tier would be better than nothing. The 'Sign Up' buttons simply link out to a credit card subscription form on PayPal and have no bearing on your account whatsoever. None of the limits will be imposed or enforced, you will still receive exactly the same service whether you pay £0 or £99.99 a month, but you will be ensuring the continued survival and reliability of the service.
Commercial options
As the service has grown and larger sites start using it, I've started to meet the needs of those larger, usually commercial, sites. If your organisation would like to support the service and requires invoicing, please contact me directly and I can arrange that for you. I also offer a Service Level Agreement and Privacy Agreement if you'd like assurances about the operation of the service and for enterprise level deployments you can be isolated on your own reporting infrastructure. If you feel you have specific requirements not covered here then please email me to see if we can cater for your needs.
Looking forwards
I have some truly amazing ideas for how to improve the service and I'm looking forward to blogging and talking about all of them. With the service continuing to grow as it is, and with such a large existing user base, I'm hopeful that enough people will choose to support the service that it remains free so I can dedicate what time I do have to spend on it on these features. If you do a quick scan of the Alexa Top 1,000 sites and look at how many of them are using report-uri.io for their CSP and/or HPKP reporting, the service clearly has potential and is providing genuine benefits to those that use it right now. To give you an idea of some of the features I'm thinking about, here's a list of those I'm prepared to share publicly so early on, in no particular order:
- Alerts, via email/slack
- Weekly reports, via email/slack
- 2 Factor Authentication
- Team access
- CSP Builder using collected reports
- Embedded dashboards
- Custom filtering
There are many out there who tell me this won't work and that I'm perhaps being naive in hoping people will pay when they don't have to. I'm hoping that there are those who will see the value in supporting the service and want it to remain free for all to use. I have a lot of big things planned for 2017 and this is the first step!