A brief recap from our earlier section on Hypothesis testing,

Formulating a hypothesis is one of the preliminary steps to set the objective of our experiment and to state what exactly we want to prove! Alternate hypothesis is what we are trying to prove and null hypothesis is a claim made by others and it is what we are trying to reject/refute.

Statistical significance

Our objective in Hypothesis testing is to see if the null hypothesis can be rejected. (Why? – Because, what we want to prove falls under ‘Alternate Hypothesis’).

We conduct the test using sample data and in order to reject the null hypothesis, we require statistical significance for our test results. The reason we require statistical significance is, the test results that we have got may be purely by chance (Sample with outliers or a non-representative sample got picked up randomly due to which the test results are skewed and there is a chance for incorrect inference being made).

P-value can be thought of as a measure for statistical significance of a particular test. Lower the P-value, higher is the statistical significance. As discussed in our earlier section, p-value denotes the area under the normal curve that is far off from the ‘claimed population mean’. For a two tailed test, we either compare the actual P-value with Alpha/2 or compare 2*p with Alpha. For 1 tailed test, Compare the P-value directly with Alpha value

Illustration

Scenario – Label of a particular food brand claims that the protein content is 15% in each packet.

Step 1 – Think of formulating a hypothesis

• If we agree to the claim of 15%, then all good, no need to test.
• If we suspect the claim and willing to challenge it, we formulate the hypothesis as below.
• Null Hypothesis – Protein content = 15%
• Alternate Hypothesis – Protein content NOT EQUAL to 15% (We want to prove this)

Step 2 – Take a sample and test it for protein content

• We sample 100 different packets and the average protein content for a packet comes out to be 14.7% and the standard deviation is 1.2%.

Step 3 – calculate z value for the above sample statistic • z = (14.7%-15%)/(1.2%/sqrt(100)) = -2.5
• Calculate p-value for the corresponding z value; p value corresponding to -2.5 z value comes out to be 0.0062 on one side and 0.0124 for a two tailed test.

Step 4 – Check for statistical significance

• Compare p-value to Alpha (1 – confidence level %); Compare 1.24% with 5% or 0.62% with 2.5%
• If the p value is less than Alpha, our test is significant; In this case, our test is significant
• We succeeded in what we want to prove
• We can REJECT the NULL HYPOTHESIS. Types of errors in Hypothesis testing

We REJECT the null hypothesis if the calculated p value is less than Alpha. For a test where we set 95% confidence levels, we will accept the test if the p-value is less than 5%. It means the risk we are taking while rejecting the null hypothesis and concluding the evidence for alternate hypothesis is 5%.

Type I error

What if we rejected the null hypothesis, but, it is actually true? We rejected it based on the sample data, but, the random sample we selected itself consists mostly of outliers or the sample does not well enough represent the population. We may infer something from sample data and thereby reject the null hypothesis, but, it may not be true for the entire population.

Suppose we reject the null hypothesis, the chances that we made an error are 5% in the case of 95% confidence level. Similarly, the chance of making an error in rejecting null hypothesis is 1% in case of 99% confidence level. This type of error is called Type I error.

• Higher the confidence level %, lower is the alpha and less is the probability of making a type I error.
• If in reality, the null hypothesis is not true, then there does not arise a chance of committing type I error.

In Short, Type I error is rejecting a true hypothesis by mistake.

Dilemma

So, in order to minimise such error %, should you always maximise the confidence level %?  If we do so, the probability that we reject the null hypothesis are less and thereby the chances of proving alternate hypothesis are less (Remember – Our overall objective of testing is to reject null hypothesis; “what we want to prove is alternate hypothesis”).

If I set a low confidence level % and thereby reject the null hypothesis, I may lose the credibility if it comes out to be a type I error. If I set the confidence level % too high, the chances that I will try to prove my point are very less.

Cost/Loss associated with the possible type I error needs to be factored in while setting the confidence level %

Type II Error

What if we failed to reject the null hypothesis when it is actually false? Let us say, the calculated P-value is 8% and the confidence level we set for the test is 5%. Here, as there is no statistical significance at this confidence level, we DO NOT reject the null hypothesis. However, in reality the null hypothesis is false. Here we are committing a type II error.

Type II error is failure to reject the null hypothesis when it is actually false.