Type I vs Type II Error

Am I the only one or is the type I vs type II error distinction kind of arbitrary. One of the EOC questions talks about a manager getting fired because the company falsely thought he was an underperformer. The answer then says this is a type II error, which is defined as failing to acknowledge that he is an outperformer.

I could have just as well argued that this is a type I error, falsely assuming he is an underperformer.

So we should just memorize, that everything is expressed in terms of outperforming. Either:

we assume someone is an outperformer although he is not (type I)

or

we fail to acknowledge someoen is an outperformer (type II)

These type of errors come from statistics. You should be thinking hypothesis testing here. In one case you reject the null when you shouldn’t have (type 1 error, very important not to do this, hence the low acceptable range for such errors 1-5%), in the other you fail to reject the null, when you should have, type 2 error). There is a specific way to set-up statistical tests depending on what you want to (dis)prove, so you cannot just rotate the statements as you wish. When you reject a null, there is very strong evidence that the null is not true, while failing to reject the null does not necessarily prove that the null is true.

We had a very nice analogy in statistics regarding the difference of what you accomplish with statistical test depending on the result. Say you have a murder case and a witness saw the murderer who had blue eyes, and the witness says he had blue eyes. When you catch a suspect, if it turns out he has brown eyes, you can safely reject the null that the suspect is the murderer, you have proven he is not, as he doesn’t have blue eyes. But if you catch a suspect and he has blue eyes, you fail to reject the null, but you have proven nothing. You could have caught any of the thousand people with blue eyes. You cannot reject the possibility he is the murderer, as the eye color did not disprove it, but you still need other evidence to prove that he’s the right guy.

Type I error: keep a bad manager

Type II error: fire a good manager

This clearly is a type II error, because the manager got fired despite his solid performance.

1 Like

Thanks guys for the explanations. Nenorr great analogy, I am familiar with the concept of type I and type II errors from statistics. I get all of that,

All I am saying is that the classification is arbitrary.

type I error could just ast well be:

  1. assume someone is an underperformer although he is not (type I)

  2. we fail to acknowledge someone is an underperformer (type II)

you can argue it’s arbitrary but this is what is taught and so stick with it.

CFA is not exactly an exam that empowers its candidates.

H0 = Managers adds no value

Ha = Manager adds value

To prove Ha, one must to reject H0.

The art of rejecting H0 when is false is the Power of Test.

If you fail to reject H0 when is false, this is (1- Power of test) or Type II Error.

Type 1 error is simply rejecting true H0. Thus, Type 1 error - not fire useless manager in this game which simply might be prescribed to just accidentally decision.

More serious is Type 2 error or fire a good manager.

Rejecting H0 is necessary, but not sufficient.

That’s better.

The probability of making a Type II error is (1 − Power of Test).

Fine. Thanks. I need an improvement in this area as well.

Thanks Edbert.

Thanks Flashback. I think I was a bit vague in my question. My problem is not the statistical background, I get the concept of statistical testing etc. but again, what I am saying is that the classification in this context is arbitrary. I just wanted to clarify that point, and have someone (as Edbert did) say, yes it is arbitrary but that is the way it is, so stick with it.

No further explanation needed from here on out. Thanks everyone.

Type I and II errors are not arbitrary. The selection of the null hypothesis may be somewhat arbitrary on the part of the curriculum, but it is consistently applied.

holy moly, I really opened Pandora’s box here.

Guys, I understand type I and II errors, all I am saying that they could have just as well decided to reverse the null and use underperformer instead of outperformer as the relevant metric.

You can define the null as:

H0 = Manager adds no value

Ha = Manager adds value

or:

H0 = Manager adds value

Ha = Manager adds no value

Now we can discuss all day long, which one has is the bigger error, and which one should be avoided (let’s please not do that)

What I am saying is that they have arbitrarily decided to go with one of them, and we need to memorize it.

If one more person feels the need to explain what the difference between type I vs type II error is… I am going to make her or him watch my 30 min stand up special from high school, and trust me there are a lot of type I and type II errors in terms of jokes that should not have made the cut.

1 Like

Currently I’m preparing a FRM and Hypothesis testing is the most easiest part there regardless this is there 2 steps further challenging than CFAI statistics.

Tartaglia, really doesn’t matter how you test hypothesis as long as you know what you’re doing. I have an impression that you are just trying to memorize and simplify the rules with no understanding the concept behind.

I am absolutely comfortable with understanding the concept behind it, what I am saying is the hypothesis is set by the curriculum in a certain way, and if you don’t know that definition, you can understand it as much as you want, you won’t be able to answer it correctly:

Example:

Q. If the fund adopted a more relaxed appraisal criteria, the most likely impact would be an increase in:

Type II error only. both types of errors. Type I error only.

Solution C is correct. If the fund relaxes the appraisal criteria, they are more likely to make a Type I error, retaining a poor manager. A and B are incorrect because Type II errors would decrease.

If you don’t know that Type I errors is defined as retaining a poor manager you cannot answer this.

If I know what I am doing (as you said), I might say Type 1 error is defined as firing a good manager …

Are what you’re sayin’ above the only facts mentioned in question? BTW, I can recall that H0 was - Manager adds no value as I mentioned before while alternative hypothesis was - Manager adds value. Maybe such construction is a custom in Investment Performance Appraisals area. Since, I don’t work there, cannot be sure. My last statement was a general opinion about testing hypothesized value.

Yes, that is all that was provided.

Flashback, your definition is actually opposite to what everyone in the forum here said, and what the questions in the curriculum said:

See what someone above posted:

Type I error: keep a bad manager

Type II error: fire a good manager

And this is exactly my point. You have memorized it the other way around, and your answer will be different than someone else’s answer who memorized it the other way around.

It does not mean your answer is wrong, but for the purpose of the exam, it would be the incorrect choice.

Maybe, yet I passed each CFA exam by first attempt including this one by such “random walk” approach. Thus give me a chance.

Keep 1 fire 2

I am not playing a blame game here nor am I questioning any ones abilities (congrats to passing)

All I am saying (since my first post) is that the classification is not something one can figure out, but rather something we need to memorize, they just decided that type I error means: keep a bad manager and NOT fire a good manager. And if you don’t know that, you can have your doctorate in statistics, but you are not going to be able to answer that question (unless you guess correctly).

And I just wanted someone (as Edbert already did) tell me:

“Yeah dawg, you right about that, but don’t matter, stop wasting ya time and get busy memorizing it.”

His words, not mine, at least that is what I remember.

Hey so I get your doubt (I had the same doubt initially), let me keep it simple:
Type 1 errors include explicit costs and is an error of commission (doing something that was wrong)
Type 2 errors include implicit (opportunity) costs and is an error of omission (not doing what was right).
So the decision to not hire or to fire (and later realising that he was an outperformer) would be a type 2 error (as you’re missing out on it, although that’s an opportunity cost not an explicit cost affecting your performance directly)
I strongly recommend you refer to the section 2.2, book 6 CFA curriculum Level 3, to deep dive further as its given quite well.