2018 in Review: How Our Bug Bounty Program Guided Prioritizing Work

Apr 10, 2019

2018 was the year when HackerOne expanded its security team and scope. Today, the team consists of four security engineers and one product manager who dedicate their time to supporting our company to move faster while reducing security risks, both for HackerOne and our customers. The team is named Dark Matter, because we believe that, similar to dark matter, security is always there even though it can’t be observed (yet) and is entangled with everything else. From supporting teams across HackerOne to running the company’s own bug bounty program, here’s an overview of the past year and a preview of what’s to come in 2019.

The Inception of Dark Matter

One of Dark Matter’s guiding principles is that security should be an enablement function. At a company where security is central to our very existence, we built HackerOne from the ground up with security as our top priority. Even so, we also know that no technology is perfect and all technology contains bugs, so support for the program and its processes were overwhelmingly positive from its inception. The challenge, we quickly discovered, would be finding our focus and establishing our unique value as a standalone team separate from and working alongside engineering, IT, and the hacker community.

We initially focused on making sure employees knew they had a team at their disposal for their security needs. However, without a known adversary, we struggled to prioritize real work and trackable OKRs from the get-go. For the first month, we focussed mainly on understanding the company’s risk appetite, working through lingering security risks and identifying projects that were aligned with our mission like products focused on improving the security of our community and customer data. Many projects were derived from exercises we ran with a number of HackerOne team members, hackers, and customers throughout this experimentation phase, leading us towards our two main charges: running HackerOne’s own bug bounty program and spearheading and troubleshooting security products, the most pressing of which being the former.

Bug Bounty

The Numbers

In April 2018, Dark Matter took HackerOne’s bug bounty program under its wing. Up until this point, the program was a shared responsibility of the engineering team — the on-call engineer would respond to incoming reports, triage it, and find an owner. With HackerOne’s growth skyrocketing year over year — in terms of the number of hackers on the platform, customers being served, and Hacktivity on our platform — it was clear we needed to expand our own security practices, so Dark Matter stepped up to take on this responsibility.

Over the course of the year, a total of 75 security vulnerabilities were reported through the bug bounty program (up from 58 in 2017) and reviewed and triaged by our team. As a result, the hackers who found them were rewarded. The majority of these security vulnerabilities got a CVSS of Low and Medium. A total of six high and critical severity security vulnerabilities were identified. Two of the critical security vulnerabilities were identified by the community: #438306 and #489146 and a third was identified by our own team: #435066. Based on our core value of “Default to Disclosure,” our team then took it upon itself to publicly disclose each and every valid vulnerability once confirmed and resolved.

Figure 1: severity breakdown of 2018 security vulnerabilities

As we triaged each request, we also worked towards improving upon a few key metrics we use to track the strength and development of our program, including time to first response and hacker participation. One notable improvement is the time it took us to fully resolve a security vulnerability — a testament to our incident response capabilities, as well as our team’s investment in acting like owners, a core HackerOne value — and individuals taking initiative to personally resolve the security vulnerability. A notable regression is our Time to First Response, which went up 91% year over ear (YoY). We believe this is caused by fewer people looking at incoming reports.

Metric 2017 2018 YoY Δ
Valid Vulnerability Volume 58 75 +29%
Hackers Participated 45 57 +26%
Recurring Hackers 35
Assets Under Test 4 11 +175%
Total Bounties Paid $54,600 $90,708 +66%
Average Bounty $1,400 $1,163 -17%
Time to First Response
Goal: sub 48 hours
12 hours 23 hours +91%
Time to Triage
Goal: sub 72 hours
6 days 2 days -66%
Time to Bounty
Goal: sub 168 hours
21 days 5 days -77%
Time to Resolution
Goal: see below
20 days 14 days -30%
Table 1: Annual metrics comparison

Our Resolution

With the momentum from 2018 at our heels, we have ambitious goals for 2019. Dark Matter’s internal Time to Resolution SLA depends on the severity of the submitted security vulnerability determined by CVSS or Common Vulnerability Scoring System — a way to capture the principal characteristics of a vulnerability and produce a numerical score to characterize it. Below is an overview of our internal SLAs, including how we did in 2018 and our goals. All days are business days (M-F). The findings show that the team responds well to high and critical severity reports, whereas there’s room for improvement for low and medium severity reports, representative of our prioritization.

Time to Resolution Goal Time to Resolution Actual
Critical 1 day 3.5 hours top 5th percentile
High 5 days 4 days top 21st percentile
Medium 10 days 14 days top 40th percentile
Low 20 days 23 days top 55th percentile
None
Table 2: Time to Resolution goals and actuals

Our Biggest Incident of the Year

On November 11, a code change was deployed to a production environment that caused program members to erroneously be added to a number of other programs on the platform. Although our engineers are alerted when something seems peculiar, it was also submitted to our bug bounty program within two hours after deploying the code. The incident showed a number of gaps in our incident response capabilities that the team uncovered through blameless retrospectives and a thorough root cause analysis, which can be found here: #438306.

Two of the exercises we regularly run are tabletops and forecasting. Tabletops help us identify gaps in our processes and product without a real incident. Forecasting helps us to align on risk and known defenses and helps us measure progress towards risk reduction. In this case, the forecasting exercise after the November 11 code change was ran multiple times amongst a similar group of people for the team to measure progress as we worked to improve our capabilities to respond to similar incidents. These exercises resulted in Dark Matter improving HackerOne’s logging capabilities, bumping up the daily log intake to roughly 50GB.

Tabletop exercises confirmed that the team can now answer any question that has come up in past incidents within an hour — a significant improvement from before the incident, which, in rare cases, left some questions unanswered. Dark Matter has found that the exercises are a great tool to uncover unknown unknowns. As with many security teams, it is often hard to know what to work on next. The exercises have kept us honest and resulted in our ability to continuously show impact for HackerOne through enablement and risk reduction.

Experiments

We also conducted a number of experiments in 2018 to further advance our bug bounty program, leverage the community more, and increase submission volume, including the following.

Research incentives

On April 18th, the team launched an experiment that would incentivize the community to submit research to our bug bounty program, regardless of whether they identified a security vulnerability. The hypothesis was that this would identify strengths in our application, which we could then incentivize hackers to explore deeper. A number of interesting reports were submitted throughout the months that followed, but there was no evidence of particular areas that should be incentivized. The team believes the lack of reports was potentially driven by a lack of communication around the experiment, and the experiment concluded on December 12.

Rewards ladder

Between June 22nd and July 21st, the team ran an additional experiment that aimed to increase and sustain engagement by increasing the rewards for vulnerabilities discovered during the time frame. The team allocated $10,000 to be split amongst the top 10 hackers based on Reputation score earned for each finding. The experiment led to a 100% month over month increase of unique participants in July and increased engagement throughout August, September, and November. However, the experiment had no noticeable effect on unique valid security vulnerabilities.

Reconnaissance data

Our team also sought to prove a hypothesis of whether more insight in our application architecture would result in more valid security vulnerabilities. The team started an experiment on December 12 that would disclose an unminified version of our JavaScript pipeline, our GraphQL schema, and the application routes of hackerone.com, also published on GitHub. Preliminary results show that disclosing this information decreases the time from a production deploy to first report, which gives the team better security coverage and helps direct hackers’ efforts. It has also helped identify the first critical severity vulnerability in 2019. This experiment is currently ongoing.

Product Security Improvements

To improve the security of our hacker community and customers, in addition to a number of internal security reviews and improvements, Dark Matter also shipped important outward facing features built to seamlessly integrate with and enhance our current platform. One such advancement is Two-Factor Authentication, which was previously a built in function for customers and has now been rolled out to support hackers. Account Recovery was then built to work alongside Two-Factor Authentication. Dark Matter also rolled out numerous SAML improvements, including Single Sign-On, Session Management, which allows enables you to review and manage all of your HackerOne sessions on all of the devices you’ve signed in to within the last 90 days, Program Submission Requirements to enable customers to set specific requirements for hackers to submit reports to programs, Automated CVE (common vulnerabilities and exposures) Assignments, a directory of publicly disclosed cybersecurity vulnerabilities that you can freely search, use, and incorporate into products and services, changes for GDPR, and improved rate limiting.

Together We Hit Harder

But all these advancements could not have been made without the rest of the product and engineering team. We’ve had a blast enabling HackerOne to move faster in 2018, and although the first quarter of 2019 is already over, we look forward to receive security vulnerabilities from the community, ship features that reduce risk, and continue to improve our incident response capabilities. We’re planning to add more assets in scope, increase rewards, and start paying for effort by including the community in our development process. If you have any feedback for the team how we can improve our bug bounty program, please drop us a note at feedback@hackerone.com.

Happy hacking!

Pei, Miju, Ben, Reed, and Jobert