As an software engineering leader this comment almost perfectly describes what Ive seen " Well part of the reason could be that along with institutional sclerosis and scope creep, we are also putting folks through a selection process that reflexively moulds them into rule following perfectionists rather than those who are willing to take risks to create something new. In fact more of the major scientists make a point of saying how they wouldn't even get hired these days, much less be in the running for Nobels." I believe that the 10x engineer is very comfortable taking risk and occasionally failing and the process seems to beat that out of the most successful candidates
Indeed. Once you're raised to succeed by solving ever narrower mazes, it's hard to break out! That's the even more insidious problem of hard selection that persist afterwards in society and make us feel like something is systemically wrong.
I started as a programmer in 1980 and much like the Nobel winners I don't believe I'd get hired by corporate today. My last Job was enterprise chief architect at a top 10 bank so I had a bit of success but guys like me fail on too many of the "check the box" selection items today, Its different now but perhaps that's why the most impactful technical talent goes to the start up markets vs corporate today. You did a nice job describing a trend I've observed but have not been able to clearly articulate. It will be interesting to watch as technology moves from a back office to front line function in corporations in the coming years
Thank you! And I'd note that the startups are also starting to face that same trend of finding selection to be harder than anticipated, as the industry booms beyond all what folks imagined!
This is going to be one of my favourites, alongside Strategy Decay and A Study of Eccentricity.
Because it's something I've recently been thinking about + advocating for. Which brings me to my first question: Is there any obvious reason that "The Economic Costs of Entrance Exams" would not be a good research topic?
The costs in question being primarily oppurtunity costs (delayed employment for potentially productive citizens, overly-homogenized knowledge among candidates after years of preparation) as well as financial (cost-benefit of money spent on education). I've been told that it wouldn't be the easiest thing to analyse. But then again, I'm not above making non-rigorous arguments when I want to.
IMO the better you are at something, the more of an oppurtunity cost the practice time for tests is. Honestly disappoints me how the benefits of post-scarcity (people can crash more often and not die) end up being nullified by a combination of lifestyle creep + status.
Re. 10x engineers, I'm sure you've heard of the possibility that they're not all that great for the company as a whole (here's a link for the comment readers: https://www.hillelwayne.com/10x/ ).
Particularly interesting to me is the idea that maybe some fields lend themselves more to "top 10% and consistent/colabborative and y'know, actually hireable" vs. "top 1% but incredible". The popular concensus (esp. in software) seems to be the latter, but I'm starting to suspect that's (at least somewhat) wrong.
I think entrance exams, or selection methods in general, are widely under-theorised, at least in a rigorous fashion that doesn't already grant the premise. So you should do it! In fact, an addendum to what you wrote would be the new methods of finding talent, what I called the Discovery problem in the creator article, where folks come up with new strategies since they don't want to do an entrance exam - eg starting substacks, commonstock channels to show performance, NFTs or DAOs etc etc. Its another selection mechanism masquerading as something else.
And yes, the bad habits spreading wider from 10x engg is an interesting problem, though that's true of anyone in the co who has bad habits, so not sure it generalises. The bigger issue is that the very hunt for 10xes makes us worse at actually finding them.
Amazing post as usual! I’ve been on the receiving end of this problem as I do a great job at work but suck at job interviews. Lambda school is figuring out a potential solution, as they worked out a deal with employers to hire their students where Lambda pays their salary for awhile and then if they do well the company hires them. Like a six-month job interview basically.
Just do the research and question, question, question. I they have to answer your questions, they can't ask you as many. Also make sure you get everything ready for your morning routine the night before. A clear mind is an adaptive one, as far as interviews go. Also remember that your social media is your CV in the current paradigm, so if you want to use personal social media (if you are insane!), make sure it is incognito.
One really obvious thing- if they tell you they are looking for someone older/more sociable/etc- they are actually looking for push back 'but I've had to shoulder responsibility, so I am more mature', 'or I have a really active social life, but I didn't think that being a roadie for my friend's band would fit with your corporate brand'.
Bit late to the party, agree with general assessment but not with the recommendation.
1. Many companies rely on teams, you can't see how good someone is until they are part of the team, but firing members of a good team creates tension and results in politics
2. Most e.g. software companies aren't even trying. As an example, I implemented a hiring policy of "we filter 10-20 candidates based on CV (out of 50-400, usually) and pay them market rate to work on a standard 1 day project". Worked amazingly well and everybody loved it. To my knowledge basically no big company does this... Because? No idea, or rather a few, but hard to detail. Is it a perfect solution? No. But it allows selecting based on more realistic criteria and it stops people on overfittinf on fake problems to "pass interviews".
I think we agree, actually. Your point 2 is the key, rather than trying to weed the 20 down to 2 through ever more precise measurement tools you implemented a better solution! And it's done in trading floors sometimes, and funds, but rarely elsewhere.
Re 1, also agree. I had written about the paradox where we hire individuals and work in teams, to that point..
I wonder if you can model the tradeoff between hiring fast vs hiring the best. I imagine that the selection process relies on measurement outcome because of 1) hidden variables (i.e. actual outcome) and 2) opportunity cost incurred on employers. Hence, employers stop at 37% of the candidate pool.
I'd probably write up on this and delve deeper! Really inspiring post :)
Is that a grab from the world of optimal online dating strategy? Because, if it is the people using it don't understand the Maths. Generally, I tended to use the first ten to twenty applications for calibration purposes. It does create a fatal flaw if HR departments stack their top picks at the top- in the old days managers used to 'cut the deck', but this still presented problems because HR might have stacked the best candidates (or at least a few of them) into a tight bunch- which meant you were shafted if you happened to use this group for your calibration.
Thanks! The secretary problem is real, and probably links to this, but iterative hiring isn't really a paradigm in most places yet. They prefer to make the retention number go up and to the right instead.
I suspect the focus on retention may also have to do with how the downsides of poor retention are felt more immediately and explicitly, compared to the downsides of an unproductive/unfit team member. Poor retention is very visible with immediate costs (to budget, moral of the team, optics/PR). This makes it feel more high stakes of a problem to focus on. Essentially, failure to think more long term.
I agree. Retention is important for employers because there is sunk cost on onboarding. It also takes a while for newcomers to hit positive productivity. There might be a solution that combines iterative hiring with lowering sunk cost: industry rotational internships. A candidate works in Company X for 3 months, and if they aren't a good fit, then that candidate can work in Company Y that is in the same industry as X.
That sounds like a great solution. The exams are a proxy of the applicants' proficiency, an imperfect one. The obvious solution is to let them work in the specified job to see how they do, the proxy can be forgot about. But not the applicants', nor the organisation, are willing to test people in the job for a long while - that's pricey and time consuming. Rotational internships sound like an elegant solution.
You know that when Trevelyan introduced the civil service exam, attention to process was exactly what was needed? But as the pandemic has recently illustrated- more nimble civil services, which are able to adapt quickly to new circumstances and switch large numbers of people out of their old roles and into new ones are probably more to be desired in the modern era. This feeds into a more general observation that universities tend to train analytical thinking at the expense of the operational mindset- the perfect being the enemy of the good- a more optimal approach is to fight fires as they arise, and mitigate the future necessity with good AAR analysis.
Thanks detailing Berkson's Paradox, it wasn't something I was aware of, but will certainly bear it in mind in future. I also tend to think that the ecosystem of a business can become self-limiting- the pursuit of innovation and change can be either too broad (by focusing upon too many objectives, many of which are trivial and create perverse systems of aligned interest) or too narrow, which overlooks ideas that may not have an obvious application, but could be gold in the right hands. I've often wondered whether there might be some utility in a third party ideas exchange, where various companies allow their engineers and cognitive creatives to spitball upon openly discussed technical challenges, incorporating some form of corporate credit system, with cash prizes for solution providing contributors. It would probably require corporate vetting officers though, which might sabotage the idea from the outset.
'The whole problem with the world is that fools and fanatics are always so certain of themselves, and wiser people so full of doubts.' -Bertrand Russell. I've often wondered whether strong partisan biases, or Myside bias, correlates with an inability to make sound judgements or decisions. On any issue relating to empirical evidence it seems quite rare for people to change their minds or add new concepts into their intellectual lexicon of ideas.
One notices it particularly with areas like climate alarmism and climate scepticism. Some seem convinced that there is no such thing as man-made climate change, whilst others seem convinced that at the very least human civilisation is set to end. Neither type seems to have actually read any of the more recent IPCC climate reports and tend to scoff at those who argue that the green premium needs to rapidly fall, and fall to be impressed with the solar desalination technology developed at Cranfield University, in the process of being applied with the Saudi's NOEM project- https://www.youtube.com/watch?v=KdFIHecZDfc
Indeed and there was a time when the exam helped ensure meritocracy worked, after it's application in India anyway. Glad you liked it! Berksons paradox is a hoot !!
As an software engineering leader this comment almost perfectly describes what Ive seen " Well part of the reason could be that along with institutional sclerosis and scope creep, we are also putting folks through a selection process that reflexively moulds them into rule following perfectionists rather than those who are willing to take risks to create something new. In fact more of the major scientists make a point of saying how they wouldn't even get hired these days, much less be in the running for Nobels." I believe that the 10x engineer is very comfortable taking risk and occasionally failing and the process seems to beat that out of the most successful candidates
Indeed. Once you're raised to succeed by solving ever narrower mazes, it's hard to break out! That's the even more insidious problem of hard selection that persist afterwards in society and make us feel like something is systemically wrong.
I started as a programmer in 1980 and much like the Nobel winners I don't believe I'd get hired by corporate today. My last Job was enterprise chief architect at a top 10 bank so I had a bit of success but guys like me fail on too many of the "check the box" selection items today, Its different now but perhaps that's why the most impactful technical talent goes to the start up markets vs corporate today. You did a nice job describing a trend I've observed but have not been able to clearly articulate. It will be interesting to watch as technology moves from a back office to front line function in corporations in the coming years
Thank you! And I'd note that the startups are also starting to face that same trend of finding selection to be harder than anticipated, as the industry booms beyond all what folks imagined!
This is going to be one of my favourites, alongside Strategy Decay and A Study of Eccentricity.
Because it's something I've recently been thinking about + advocating for. Which brings me to my first question: Is there any obvious reason that "The Economic Costs of Entrance Exams" would not be a good research topic?
The costs in question being primarily oppurtunity costs (delayed employment for potentially productive citizens, overly-homogenized knowledge among candidates after years of preparation) as well as financial (cost-benefit of money spent on education). I've been told that it wouldn't be the easiest thing to analyse. But then again, I'm not above making non-rigorous arguments when I want to.
IMO the better you are at something, the more of an oppurtunity cost the practice time for tests is. Honestly disappoints me how the benefits of post-scarcity (people can crash more often and not die) end up being nullified by a combination of lifestyle creep + status.
Re. 10x engineers, I'm sure you've heard of the possibility that they're not all that great for the company as a whole (here's a link for the comment readers: https://www.hillelwayne.com/10x/ ).
Particularly interesting to me is the idea that maybe some fields lend themselves more to "top 10% and consistent/colabborative and y'know, actually hireable" vs. "top 1% but incredible". The popular concensus (esp. in software) seems to be the latter, but I'm starting to suspect that's (at least somewhat) wrong.
I think entrance exams, or selection methods in general, are widely under-theorised, at least in a rigorous fashion that doesn't already grant the premise. So you should do it! In fact, an addendum to what you wrote would be the new methods of finding talent, what I called the Discovery problem in the creator article, where folks come up with new strategies since they don't want to do an entrance exam - eg starting substacks, commonstock channels to show performance, NFTs or DAOs etc etc. Its another selection mechanism masquerading as something else.
And yes, the bad habits spreading wider from 10x engg is an interesting problem, though that's true of anyone in the co who has bad habits, so not sure it generalises. The bigger issue is that the very hunt for 10xes makes us worse at actually finding them.
Amazing post as usual! I’ve been on the receiving end of this problem as I do a great job at work but suck at job interviews. Lambda school is figuring out a potential solution, as they worked out a deal with employers to hire their students where Lambda pays their salary for awhile and then if they do well the company hires them. Like a six-month job interview basically.
Thanks!! And yeah that seems to be a good solution too align incentives better, despite recent troubles.
Just do the research and question, question, question. I they have to answer your questions, they can't ask you as many. Also make sure you get everything ready for your morning routine the night before. A clear mind is an adaptive one, as far as interviews go. Also remember that your social media is your CV in the current paradigm, so if you want to use personal social media (if you are insane!), make sure it is incognito.
One really obvious thing- if they tell you they are looking for someone older/more sociable/etc- they are actually looking for push back 'but I've had to shoulder responsibility, so I am more mature', 'or I have a really active social life, but I didn't think that being a roadie for my friend's band would fit with your corporate brand'.
'If'
Bit late to the party, agree with general assessment but not with the recommendation.
1. Many companies rely on teams, you can't see how good someone is until they are part of the team, but firing members of a good team creates tension and results in politics
2. Most e.g. software companies aren't even trying. As an example, I implemented a hiring policy of "we filter 10-20 candidates based on CV (out of 50-400, usually) and pay them market rate to work on a standard 1 day project". Worked amazingly well and everybody loved it. To my knowledge basically no big company does this... Because? No idea, or rather a few, but hard to detail. Is it a perfect solution? No. But it allows selecting based on more realistic criteria and it stops people on overfittinf on fake problems to "pass interviews".
I think we agree, actually. Your point 2 is the key, rather than trying to weed the 20 down to 2 through ever more precise measurement tools you implemented a better solution! And it's done in trading floors sometimes, and funds, but rarely elsewhere.
Re 1, also agree. I had written about the paradox where we hire individuals and work in teams, to that point..
I wonder if you can model the tradeoff between hiring fast vs hiring the best. I imagine that the selection process relies on measurement outcome because of 1) hidden variables (i.e. actual outcome) and 2) opportunity cost incurred on employers. Hence, employers stop at 37% of the candidate pool.
I'd probably write up on this and delve deeper! Really inspiring post :)
Is that a grab from the world of optimal online dating strategy? Because, if it is the people using it don't understand the Maths. Generally, I tended to use the first ten to twenty applications for calibration purposes. It does create a fatal flaw if HR departments stack their top picks at the top- in the old days managers used to 'cut the deck', but this still presented problems because HR might have stacked the best candidates (or at least a few of them) into a tight bunch- which meant you were shafted if you happened to use this group for your calibration.
Thanks! The secretary problem is real, and probably links to this, but iterative hiring isn't really a paradigm in most places yet. They prefer to make the retention number go up and to the right instead.
I suspect the focus on retention may also have to do with how the downsides of poor retention are felt more immediately and explicitly, compared to the downsides of an unproductive/unfit team member. Poor retention is very visible with immediate costs (to budget, moral of the team, optics/PR). This makes it feel more high stakes of a problem to focus on. Essentially, failure to think more long term.
Yes indeed, things which are easier to measure get higher visibility.
I agree. Retention is important for employers because there is sunk cost on onboarding. It also takes a while for newcomers to hit positive productivity. There might be a solution that combines iterative hiring with lowering sunk cost: industry rotational internships. A candidate works in Company X for 3 months, and if they aren't a good fit, then that candidate can work in Company Y that is in the same industry as X.
That sounds like a great solution. The exams are a proxy of the applicants' proficiency, an imperfect one. The obvious solution is to let them work in the specified job to see how they do, the proxy can be forgot about. But not the applicants', nor the organisation, are willing to test people in the job for a long while - that's pricey and time consuming. Rotational internships sound like an elegant solution.
The Source of the sadness is A.I., the killer of humanity.
You know that when Trevelyan introduced the civil service exam, attention to process was exactly what was needed? But as the pandemic has recently illustrated- more nimble civil services, which are able to adapt quickly to new circumstances and switch large numbers of people out of their old roles and into new ones are probably more to be desired in the modern era. This feeds into a more general observation that universities tend to train analytical thinking at the expense of the operational mindset- the perfect being the enemy of the good- a more optimal approach is to fight fires as they arise, and mitigate the future necessity with good AAR analysis.
Thanks detailing Berkson's Paradox, it wasn't something I was aware of, but will certainly bear it in mind in future. I also tend to think that the ecosystem of a business can become self-limiting- the pursuit of innovation and change can be either too broad (by focusing upon too many objectives, many of which are trivial and create perverse systems of aligned interest) or too narrow, which overlooks ideas that may not have an obvious application, but could be gold in the right hands. I've often wondered whether there might be some utility in a third party ideas exchange, where various companies allow their engineers and cognitive creatives to spitball upon openly discussed technical challenges, incorporating some form of corporate credit system, with cash prizes for solution providing contributors. It would probably require corporate vetting officers though, which might sabotage the idea from the outset.
'The whole problem with the world is that fools and fanatics are always so certain of themselves, and wiser people so full of doubts.' -Bertrand Russell. I've often wondered whether strong partisan biases, or Myside bias, correlates with an inability to make sound judgements or decisions. On any issue relating to empirical evidence it seems quite rare for people to change their minds or add new concepts into their intellectual lexicon of ideas.
One notices it particularly with areas like climate alarmism and climate scepticism. Some seem convinced that there is no such thing as man-made climate change, whilst others seem convinced that at the very least human civilisation is set to end. Neither type seems to have actually read any of the more recent IPCC climate reports and tend to scoff at those who argue that the green premium needs to rapidly fall, and fall to be impressed with the solar desalination technology developed at Cranfield University, in the process of being applied with the Saudi's NOEM project- https://www.youtube.com/watch?v=KdFIHecZDfc
Thanks for an enjoyable essay.
Indeed and there was a time when the exam helped ensure meritocracy worked, after it's application in India anyway. Glad you liked it! Berksons paradox is a hoot !!