Hypothesis testing is a statistical method used to evaluate the validity of a hypothesis by examining differences in sample statistics and population parameters. 

In business, it is very important to determine whether a strategy is effective or not, and it is important to build a hypothesis test based on data.

An important concept in this hypothesis testing is the null hypothesis and the counter hypothesis.

# Null and alternative hypotheses

- Null hypothesis: Given a hypothesis, how likely is it that the hypothesis is true?

 → Nothing new, no difference, no effect.

- Alternative hypothesis: The opposite of the null hypothesis (the possibility that the null hypothesis is false).

 → There is something new, there is a difference, there is an effect.

source: https://www.scribbr.com/statistics/null-and-alternative-hypotheses/

This was the hardest part for me to understand at first. 

Why bother with the null hypothesis when you can just use the counterfactual that there is an effect?

To the best of my knowledge, here's a quick summary of why we introduce the null hypothesis.

# Reasons to introduce the null hypothesis 

1) The null hypothesis is the first hypothesis that is set before the experiment, and it is the standard against which the results of the experiment are validated.

 - If you only test the null hypothesis, even if the results of the experiment support the null hypothesis, it's hard to tell if it's true or not. 

 - For example, if the null hypothesis is "this ad campaign is effective at increasing sales", we don't know for sure whether the ad campaign actually works or not. 

 - Therefore, we need to have both a null hypothesis and a counter hypothesis, and then run an experiment to reject the null hypothesis and accept the counter hypothesis, so that we can determine if the results are significant.

2) The null hypothesis helps you clearly set the purpose and method of the experiment. 

 - Rejecting or not rejecting the null hypothesis based on the results of an experiment can lead to different interpretations depending on the purpose of the experiment. 

3) It's easier to prove the null hypothesis than it is to prove something new with the alternative hypothesis.

 - We must also consider the possibility that there is subjectivity involved in our research that we don't know about.

# Hypothesis testing process 

The hypothesis testing process includes the following steps: 

1. Formulate a null hypothesis and an alternative hypothesis.

 - Null hypothesis: The ad campaign has no effect on sales growth.

 - Null hypothesis: Ad campaigns have an impact on sales growth.

2. Perform sampling to collect sales data before and after the ad campaign you want to test.

3. Calculate a test statistic based on the sample data.

4. Calculate the distribution of the test statistic according to the null and alternative hypotheses.

5. Determine the significance level of the null hypothesis, and if the test statistic is less than the significance level, reject the null hypothesis and accept the alternative hypothesis. Conversely, if the test statistic is greater than the significance level, the null hypothesis is accepted.

The above procedure can be used to test the effectiveness of an ad campaign in increasing sales. The test statistic can be calculated by comparing the sales data before and after the ad campaign, and the significance level can be set to a statistical standard such as 0.05.

# Glossary  

1) What is rejecting and accepting a hypothesis (+ significance level)?

First, we need to understand the concept of significance level (alpha). The significance level is a value that represents the standard of probability that an experimental result occurs by chance, usually set at 0.05 or 0.01. 

ex: significance level (alpha) = 0.05 → 95% confidence (1-0.05)

In general, the smaller the significance level, the more rigorous the validation.

Case 1. Probability of significance (p-value) < significance level (alpha)

 - Reject the null hypothesis and accept the alternative hypothesis. 

  Note: Since the probability of the experimental result occurring by chance is very small, we can conclude that the result is indeed significant. 

Case 2. Probability of significance (p-value) >= significance level (alpha)

 - Reject the null hypothesis. 

 Since the probability of the experimental result occurring by chance is high, it is difficult to conclude that the result is significant.

2) What is significance probability?

The probability of significance is one of the important values used in hypothesis testing in statistics. Given a certain test statistic, the probability that a value equal to or more extreme than that statistic will be observed if the null hypothesis is true.

The method for calculating significance depends on the test statistic and the type of distribution, but it is usually calculated using the probability density function of the test statistic or the cumulative distribution function.

For example, when testing the difference in means between two groups in an experiment, the test statistic might be the t-statistic. Given the mean, standard deviation, and sample size of the two groups, you can calculate the t-statistic and use it to find the probability of significance.

Another example is the chi-square test. You can calculate the chi-square test statistic and use the chi-square distribution to calculate the probability of significance.

3) What are type 1 and type 2 errors?

Type 1 error and type 2 error are errors that can occur in statistical hypothesis testing.

A type 1 error is when you reject the null hypothesis even though it is true. This error is more likely to occur if you set your significance level (alpha) too low. 

The significance level is the maximum probability that you can reject the null hypothesis given that it is true.

The Type 2 error is when you reject the null hypothesis even though the null hypothesis is false. This error is most likely to occur when the sample size required to verify that the null hypothesis is false is insufficient, or when the power is low. 

Power is the ability to detect when the null hypothesis is false.

source: https://www.scribbr.com/statistics/type-i-and-type-ii-errors/ 

There are many different test methods and test statistics, but at the end of the day, it's all about choosing the right test method and test statistic for the given data and the purpose of the analysis, and using it to calculate the probability of significance to test the hypothesis.

It gets more and more complicated, doesn't it? I don't think we need to take it lightly and memorize it.

Our goal is to quickly understand only what we need in our business, so we take good notes on what each test is used for and when, so that we only need to understand which test to use and what conclusions to draw, and Python will do the math for us : )

We just need to be able to write the appropriate values to Python.