July 6 - August 6 - September 6 -- that's 2 months elapsed, not three.
Five people working for 2 months is 10 person-months. Apple paid them just under $52,000, none of which was guaranteed. They had to pay whatever taxes are appropriate for their jurisdictions.
The amount of effort put into finding multiple critical - high vulnerabilities of a $1TN+ company and the result is $51k + taxes to possibly share between 5 hackers for 4 qualifying bugs for that bounty sounds like Apple took them for a cheap ride through their campus.
Compared to 1 hacker, 1 month, JWT signature check failure = 100k from Apple [0]:
The 4 exploits they got paid for don't seem like the biggest ones though.
I would expect Apple to pay $500k - $1M for this session in the end, and it would be in the best interest of all parties if this happened. Apple would encourage responsible disclosure (and attract more white-hat bug hunters) this way. The amount of vulnerabilities found is a proof by itself that team work does pay off, if the team is strong. Also, this is a drop in the bucket for Apple. It would probably cost them much more to have them on the payroll for the same amount of time.
Where did you come up with that number? $500k is much more than a sitewide external app pentest of comparable scope would cost Apple, by an integer multiple. The bugs here are good, but they're not "bug bounty black swan" good; they're what you'd expect from a sitewide pentest.
I agree Apple got a great deal here (that's the point of bounties, and anyone who thinks they're a bad deal for strong researchers is... right). But I'm always going to point out that HN has weird misconceptions about the economics of this stuff.
That second bug they describe would have allowed them to mess with inventory in a warehouse. They could have easily "disappeared" millions of dollars of products. Some of these other bugs would have required apple to disclose PII leak disclosure which could do tens of millions of dollars of damage to their company valuation.
You'll find, if you talk to people that do this work professionally, that bugs where you can tell yourself a story about the millions of dollars you could make are not uncommon, and that the rack rate for generating those bugs doesn't scale with their hypothetical value. I've done multiple projects for FIX gateways at exchanges. Those are fun stories to tell yourself! But those projects weren't even especially lucrative.
Pen test that took 6 months with 10 people would cost at least $2mm using an extremely low $200/hr rate. People who are best in the industry will be significantly higher.
Yes. I'd say "word to the wise", but I think very few people reading this thread buy pentest time in such large blocks: past a month and you start getting into steep discounts.
(This was not several months of full time work, but rather several months of part time work; but I'm stipulating the former condition.)
Your comment got me thinking, Apple probably was already buying large blocks of pentest time, and the comments in the thread make it seem like these were obvious flaws. Is that right? If we assume Apple already had a contracted pentest firm, can you speculate why didn't they find these flaws?
I don't know what "obvious flaws" means. I know from like a dozen years of consulting experience, and from 10 years of vuln research prior to that, that putting a different set of eyes on a target tends to get you a different set of bugs. Finding vulnerabilities is as much an art as a science, which makes sense when you think about what hunting for software vulnerabilities actually entails. If you could do it deterministically, you'd be saying something big about computer science.
I think we're on firmer ground saying that there are ways of delivering software that foreclose on "obvious bugs". But when we talk about fundamentally changing the way we deliver software --- in secure-by-default development environments, on secure-by-default deployment platforms, with security as a primary functional goal prioritized over time-to-market --- we're actually into real money now, not just another $250k on pentesters.
Yes, because it is worth in pentesting services 180k USD, no more no less. I mean, you can pay around 360k in London or SV rates and 180k in European for _similar_ skills people.
Calc based on 3 months, 5 people, 600USD/md rate.
EDIT as I can't reply to tpaceck below: no, those 2000usd/day rates do not exists in projects in size of 300MD like here. In general they do not exist for big projects.
Yes, I agree, you have rates around 1200 in high cost countries, yet as I wrote earlier, you can have similar/the same skill level at 600 usd/md if you're willing to work with guys not from HCC.
If "md" means "billable day", a $600 billable day is extremely low for this kind of work; that's closer to what people pay for network pentesting. $1500-$2000 is closer to the market (before discount, assuming senior but not principal level delivery).
When I worked as a 'consultant' (glorified contractor) .Net developer, the company charged > 90 Euro / 105 USD per hour for my time. So that would make my going rate be > 800 USD / day. This is in a country where 50K / year is a decent developer salary.
I do not believe you can find pen testers worth their salt who would cost _less_ than a non-distinctive developer. At least not one who will do more than run some automated report over all your endpoints.
A classic false comparison: the four experienced security researchers working for multiple months covers 55 issues, not "that one issue".
If we're cherry picking a single one, the associated involvement and timeframe drops dramatically, to something much closer to one or two people, tops, over the course of just a few days, tops.
That's something a pentesting team can absolutely achieve for far less than $500,000 over the course of a few days, too.
I’m unsure what your point is? I see dozens of different issues listed in the post, on different endpoints, all of which presumably took time to find. When they said they had a team of multiple people work for months on this, I am unsure why you think they haven’t spent their time as efficiently as “a pentesting team”. Actually, I’ll be stronger: looking through the list of things they discovered, it seems like they were absolutely churning out vulnerabilities for the entire period. A real team would have certainly cost much more than what they’ve currently been paid.
Issue count != time spent. I found about a dozen issues in a day once. And once, it took me three days to find one.
Always found at least a medium severity issue though.
Big engagements were typically a week, max. Usually one day of kickoff / getting “in the zone” for a project, three or so days of intensive testing, then the final day is usually writing reports (ugh, reports) all day.
There's really 2 options here. One, Apple doesn't employ a pen-testing team currently, which would be nuts, or, two, the pen-testing team couldn't find these bugs, or they'd already be found.
Apple has product security teams, in infra security team that covers a lot of this web attack surface, a large red team, researchers, and employs 3rd party firms to do sitewide tests.
Apple is also huge, and no huge company avoids vulnerabilities; staff as ambitiously as you want, but any disjoint group of competent testers attacking a new target is going to find a disjoint set of bugs.
Or option 3: apple is HUGE, in all respects: physical space, people with access, code base, etc. etc. and they already have plenty of teams in place, but a bug bounty program is a cheap supplemental. In which case paying out more for your bug bounty program than you pay your real teams would be really weird.
In that case, do you think that Apple is incompetent for not stumping up $250k or less for an external pentester to find these bugs? Plus maybe $100k more for an internal PM/point of contact for the pentester? Or do you think Apple handled it fine, the expected cost to the business of their security holes was less than $350k and they could just wait for them to come through the bug bounty program or for internal engineers to find them?
I think everything is complicated, and that is certainly isn't as simple as "Apple should pay paid $250k to a pentesting firm to find these bugs", because you could keep paying $250k over and over again and keep finding different bugs of comparable severity.
It's not a question of whether any spot assessment is worth $250k (though: Apple can get a sitewide pentest from experts for substantially less than that). It's a question of whether paying that continuously is worth it, or whether that many can be spent more productively on something else.
For what it's worth, "reputational damage" has always been a kind of rhetorical escape hatch from arguments that have become too mired in facts.
Apple paid with public exposure. Anything Apple is a story of interest, which has a value especially in security circles where half the business is a pure PR exercise.
I’ve spent time in my career with a “big gorilla” employer whose business is very visible within its community. Companies will “pay” a lot to say “We solved FooCorp’s problems with <x>” or “FooCorp bought our <y>”
Lazy buyers assume that their peers have their shit together.
While it's a great marketing and reputation building tool, it's still pretty poor to pay people in exposure; they could have taken each and every one of these exploits to the black market instead and they probably would have earned a lot more money.
Alternatively, the Apriso exploit alone apparently would have allowed them to create fake manufacturing-level employees with fake payroll going to arbitrary bank-account targets; so an unethical attacker probably could have collected an unbounded amount of money just from that (since it likely wouldn’t have been caught until after the first event; and payroll would happen all at once, paying out to as many different accounts as the attacker wished.)
PR is good but it wont keep the lights on. If you want them to return to work for you, pay them with exchange currency. Apple's motive should be to encourage skilled hackers to come forward with exploits - i.e. make it worth their time. Not drive them into the arms of a competitor.
They may have corrected it. Whilst $52k is cheap for 15 months of labour, that's as of October 4th. So it's not unreasonable for that number to go up significantly over time. It'll be interesting to see what their final total is.
I don't know what Apple would value 15 months of highly skilled security consultants at, but I can't imagine it'd be below $200k, so Apple is likely still getting a good deal even if they pay out a lot more.
Something tells me the real money comes from future consulting contracts and that this PR will more than pay for itself. Just like how everyone on HN agrees writing a book isn't a great use of time besides what it allows you to put on your resume.
Just because Apple got an amazing bargain doesn't mean the payout for them won't be great as well.
One problem is this puts a downward pressure on others who demand fair compensation for their labor. Not everyone wants to play a long game of "maybe i'll get paid in the future from the 'experience'"
This is the professional equivalent of having interns do a bunch of real work and throwing them a pizza party.
Unfortunately, it doesn’t matter if other people don’t want to play the long game. This team does, they’re executing it well, and it will boost their careers as a result. Everything was done voluntarily by consenting professionals with the rules of the game outlined up front. Can’t really fault them for that.
People can consent or do plenty of things that are allowable. That doesn't mean I can't fault the actions or dig deeper into whether or not it has other drawbacks (or even pros). Just because something is allowable doesn't mean it doesn't have other impacts.
But to be clear that doesn't mean I think they (or someone else) should not be allowed to make this choice. The possibility should definitely exist. I just don't think it's a good choice in terms of it being a norm.
This is very fair criticism for standard jobs like a regular software developer.
For a role like this, where outsized skill of someone who is and needs to be elite should be rewarded with enormously outsized pay, I think this a good model.
But, I do find it wild that a group as decorated as this already can't even get compensation that is commensurate with their skill and experience without having to rely on intangible future benefits.
This is exactly why they’re writing a blog post about it.
This type of social proof, when executed well, is a boon to one’s career opportunities and credibility for getting future consulting jobs.
If they’re not hired by Apple, they’re going to move to the top of the list for info section recruiters everywhere. Being able to point to this blog post makes them an easy sell relative to some other person with a generic resume.
As full time job might not mean 40 hours per week during the pandemic. This is briefly mentioned in the article.
“ This was originally meant to be a side project that we'd work on every once in a while, but with all of the extra free time with the pandemic we each ended up putting a few hundred hours into it.”
It is not about $/hour, it is about the knowledge they have learnt and that will help Apple protect against some bugs which would result in losses and other damages to consumers..
Everybody wins here. It's a bargain for Apple, because their ledgers deal with numbers that require the -illions suffixes, but it's ALSO $10k per person, which even after taxes is still a lot of money on top of their regular salary for anyone with bills to pay.
Good pentester costs typically $2000 per day. Given the amount of work they did, the return feels like a slap in the face. Certainly it won't encourage highly skilled people to hunt for security holes.
Apple hasn't paid them for the largest exploits yet. That $52k will likely blow-up to far more when Apple pays them so their work will end up being much more lucrative. Apple also pays in batches so they'll likely get a few more batches and some of those will be yuuuuuuuge!
These are not equivalent propositions. There is an incredible amount of value in working outside of a big corporation and its management hierarchy. It is a Dog and the Wolf situation. The food is always better under the collar.
Qualifying people for highly paid info security positions is shockingly broken right now. No one who knows what they are doing cares about credentials you can get from a training program or school, but they also complain constantly about how hard it is to find and hire qualified people. The result is: there is a lot of salary out there for people who can figure out how to get it.
Developing exploits that are acknowledged by major targets--even if done freelance or as a hobby--is one of the few ways to gain lines on your resume that everyone in the security field will pay attention to.
It's the whole "you need to volunteer for a year before we'll hire you" hiring method typically seen in low paid positions in the arts, but this time for high paid infosec positions...
The art world might not be a bad comparison. In both security and art, established people with money are looking for new people who have the ability to make an impact.
But the established folks don't know in advance what exactly that will be... if they did, they'd already be paying someone to do it.
As a new person, there's no better way to demonstrate your ability to make an impact than to just do it.
I work at a company that has an infosec division and I don't know how we got so lucky with the people there. They're seriously legit low level kernel type programmers who seem to be able to reverse engineer anything given enough time and are able to seriously reason about what's going on in security. The types of people who speak at and headline at the largest security conferences, etc. Again, no idea how we got so lucky to have a great crew.
I'm not an infosec person myself. But my experience is that upwards of 80% of the ones I interact with who aren't like the people I mentioned above are just hangers on because they like the group or being associated with "infosec" because it sounds cool or something. Maybe it's because you don't need to be an engineer to regurgitate OWASP vulnerabilities and tell people to use password managers, but perhaps that's enough to, after you look around the room of infosec people, feel like you're an "infosec person." To be clear, that stuff is important, but not anywhere close to sufficient. So a lot of applications for our roles come from these people, who just sit on twitter all day and retweet the Taylor Swift security person, but they're totally not technical and have done nothing of note other than write compliance plans.
My hypothesis is that it's all this noise that makes hiring good infosec people difficult. If I'm hiring a kernel programmer or SRE I seem to get much more signal in my applications, but hire someone for security or infosec and there's too much noise from people like above.
Information security is just a super wide field. To pick a couple famous examples: what Google Project Zero does, and what the "Swift on Security" person does, have almost nothing to do with each other.
They both matter, though. Basic blocking and tackling at the IT level is important, especially to large old institutions. Apple is obviously an apex technology company, but they're also a 45 year old public corporation... I'm not surprised they've got some vulnerabilities lurking in their subdomains.
Patrolling DNS and 3rd party corporate applications is not usually what people think is sexy security work, though. Problems avoided are harder to sell than problems discovered or bad guys defeated.
Oh totally, as I mentioned above I am not an infosec person and I hope I didn't imply otherwise (I did mention this specifically above). The above is just my impression from the outside but as someone who talks to and works with a lot of security/RE/infosec people.
That was just a really snarky way of saying that RE people and people who pay attention to OWASP are not comparables. Sorry, I should have just been direct about it.
It is impossible to quantify what is a good use of their time without knowing them. Also not everyone does things in the pursuit of money. I sell eggs and could easily ask 5$ a dozen with the demand I have. Instead I only ask 4$ and have lots of clients I only charge 2$ and some I just give eggs to when I have extra. These are people with no money or means. I don’t expect to ever get anything from these people but every once in a while ‘oh my car breaks down and guess who has the knowledge or tool I need the guy I have been giving eggs’. I know the world will eat you up and take all you have but I personally “invest” my time and effort into a few of the things I enjoy even if the reward is low. These researchers now have an excellent start to a resume which is always a good thing.
Well after covid started and the stores ran out of a lot of food I decided to get some chickens again. I have had a maximum of 6 in the past but decided to increase the flock since 6 birds is pretty much the same effort as 30 birds. I now have 33 in total and at this point in their life get one egg a day. They average something like 300+ eggs a year. I have sold enough to buy an automatic egg washer and now mainly worry about selling enough to cover feed costs. I do it because chickens are very therapeutic and I find them relaxing to be around. I have young kids so they are also learning the value of food and can eat all the eggs they want. So I wouldn’t really call it much of a business it is more of a hobby that I reap little reward other then my eggs and to help out a few others near me. I think if I ramped up to a few hundred birds I could make a bit of money but at the small size it keeps me from getting overwhelmed with too much work and I can just share my harvest with those around me. I have learned that making money is nice but I also get a great deal reward from helping others in need.
Bug bounties are not generally considered a good source of income. It's a way to hone your skills, gain experience, develop a bit of industry cachet and get paid a little in the process.
For one they did not only get the money but also the exposure that comes with anything Apple. A lot of people will probably want to hire these researchers.
Five people working for 2 months is 10 person-months. Apple paid them just under $52,000, none of which was guaranteed. They had to pay whatever taxes are appropriate for their jurisdictions.
I'd say Apple got an amazing bargain.