Analysis of Variance (ANOVA)
I. What is ANOVA
Analysis of Variance (ANOVA) is a statistical method used to test differences between two or more means. It is used to determine whether statistically significant differences exist between the means of three or more groups. ANOVA separates observed variance data into different components to use for additional tests.
In ANOVA, the variance between the data samples is compared to the variation within each particular sample. If the between-group variance is high and the within-group variance is low, this provides evidence that the means of the groups are significantly different.
There are two main types of ANOVA tests: one-way and two-way. One-way ANOVA is used for three or more groups of data to gain information about the relationship between the dependent and independent variables. Two-way ANOVA is used when there are two independent variables.
ANOVA is based on comparing the variance between the data samples to the variation within each particular sample. If the between-group variance is high and the within-group variance is low, this provides evidence that the means of the groups are significantly different.
ANOVA terminology includes the following key terms: factor, levels, response variable, within-group variance, between group variance, grand mean, treatment sum of squares (SSTR), error sum of squares (SSE), total sum of squares (SST), degrees of freedom (df), mean square (MS), F-ratio, null hypothesis, alternative hypothesis, and p-value.
II. One-Way ANOVA
One-Way ANOVA is a statistical method used to determine whether there are any significant differences between the means of three or more independent groups. It is commonly used in scientific research to compare the means of different treatments or interventions. It is also used in business to compare the performance of different products or services.
It is used to test the null hypothesis that there are no significant differences in the means of the groups. The alternative hypothesis, on the other hand, suggests that at least one group mean is significantly different from the others. The hypotheses can be stated as follows:
Null hypothesis (H0): μ1 = μ2 = μ3 = … = μk (where k is the number of groups)
Alternative hypothesis (Ha): At least one group mean is significantly different from the others.
Before conducting One-Way ANOVA, it is important to ensure that certain assumptions are met. These assumptions include:
Independence: The observations within each group are independent of each other.
Normality: The dependent variable follows a normal distribution within each group.
Homogeneity of variances: The variances of the dependent variable are equal across all groups.
Violations of these assumptions can affect the validity of the results obtained from One-Way ANOVA. Therefore, it is crucial to assess these assumptions before proceeding with the analysis.
For example, suppose we want to test whether there is a difference in the average weight loss between three different diets: a low-carb diet, a low-fat diet, and a Mediterranean diet. We can use One-Way ANOVA to determine whether there is a statistically significant difference between the means of the three groups.
Another example is to understand if there is a difference in income based on education level. ANOVA can also be used to analyze the interaction between different levels of variables, such as advertising levels and pricing levels.
The following steps are involved in performing a One-Way ANOVA:
State the null hypothesis and alternative hypothesis: The null hypothesis is that all groups have the same mean, while the alternative hypothesis is that at least one group has a different mean.
Calculate the F-ratio: The F-ratio is calculated by dividing the between-group variance by the within-group variance. It measures the extent to which the group means differ from each other compared to the variability within each group.
The F-statistic is calculated using the following formula:
F = (Between-group variability (or MSB)) / (Within-group variability (or MSW))
If the F-statistic is large enough to reject the null hypothesis, it indicates that there are significant differences in the means of the groups.
Determine the p-value: The p-value is the probability of obtaining a test statistic as extreme as the one calculated from the sample data, assuming the null hypothesis is true.
Compare the p-value to the significance level: If the p-value is less than the significance level (usually 0.05), then we reject the null hypothesis and conclude that there is a statistically significant difference between the means of the groups.
Example:
Here is an example problem to illustrate how to solve a One-Way ANOVA problem.
Suppose we want to test whether there is a difference in the average weight loss between three different diets: a low-carb diet, a low-fat diet, and a Mediterranean diet. We randomly select 10 people for each diet and record their weight loss over 6 months. The following table shows the results:
Diet | Sample Mean | Sample Variance | Sample Size |
Low-Carb | 10 | 2.5 | 10 |
Low-Fat | 8 | 1.5 | 10 |
Mediterranean | 12 | 3.0 | 10 |
We can use One-Way ANOVA to determine whether there is a statistically significant difference between the means of the three groups.
Let us look into the one-way ANOVA formula for calculating F-statistics or F-ratio:
F = MSB/MSW
In this formula,
F: The F-ratio is a test statistic used to determine whether the means of two or more groups are significantly different from each other.
MSB: The mean square between groups is the variance between the sample means.
MSW: The mean square within groups is the variance within each particular sample.
You usually employ statistical software like R, Excel, Stata, SPSS, etc. to conduct a one-way ANOVA because manual calculations can be tedious. Regardless of the software chosen or manual method, the resulting output will include the following table:
where:
X̅𝑖 = sample mean from the 𝑖𝑡ℎ group
𝑠𝑖2 = sample variance from the 𝑖𝑡ℎ group
𝑛𝑖 = sample size from the 𝑖𝑡ℎ group
𝑘 = number of groups
𝑁 = 𝑛1 + 𝑛2 + ⋯ + 𝑛𝑘 = sum of the individual sample sizes for groups
Grand mean from all groups = X̅GM = ∑X̅𝑖/𝑁
Sum of squares between groups = SSB = ∑𝑛𝑖(X̅𝑖 − X̅GM)2
Sum of squares within groups = SSW = ∑(𝑛𝑖−1)𝑠𝑖2
Mean squares between groups (or the between-groups variance 𝑠2𝐵) = 𝑀𝑆𝐵 = 𝑆𝑆𝐵/𝑘−1
Mean squares within-group (or error within-groups variance 𝑠2𝑊) = 𝑀𝑆𝑊 = 𝑆𝑆𝑊/𝑁−𝑘
𝐹 = 𝑀𝑆𝐵/𝑀𝑆𝑊 (F is the test statistic)
First, we calculate the Total Sum of Squares (SST):
where, Xij is the jth observation in group i, and X̅ is the overall sample mean.
Using the values from the table, we get:
SST = (10−10.0)2+(10−10.0)2+(8−10.0)2+(8−10.0)2+(12−10.0)2+(12−10.0)2+(10−10.0)2+(10−10.0)2+(10−10.0)2+(10−10.0)2
SST = 40
Next, we calculate the Treatment Sum of Squares (SSR):
where, Ti is the sum of observations for group i, T̅ is the overall sum of observations divided by the total sample size, and ni is the sample size for group i.
Using the values from the table, we get:
SSR = 10(10−30/3)2 + 10(8−30/3)2+ 10(12−30/3)2
SSR = 8
Next, we calculate the degrees of freedom:
Degrees of freedom between groups (dfbetween) = k − 1 = 3 − 1 = 2
Degrees of freedom within groups (dfwithin) = n − k = 30 − 3 = 27
Degrees of freedom total (dftotal) = n − 1 = 30 − 1 = 29
Next, we calculate the mean square between groups:
MSB: Between-group variability (MSB = SSR/dfbetween)
MSB = 8 / 2
MSB = 4
Next, we calculate the mean square within groups:
MSW: Within-group variability (MSE = SSE/dfwithin)
MSW = 32 / 27
MSW = 1.19
Finally, we calculate the F-ratio:
F = MSB / MSW
F = 4 / 1.19
F = 3.36
To interpret the results, we compare the F-ratio to the critical value of F with the degree of freedom dfbetween and dfwithin at a significance level of 0.05. If the calculated F-ratio is greater than the critical value of F, we reject the null hypothesis and conclude that there is a statistically significant difference between the means. If the calculated F-ratio is lesser than the critical value of F, we fail to reject the null hypothesis and conclude that there is no significant difference between the means.
In this example, the critical value of F for the degrees of freedom of 2 and 27 at 0.05 level of significance is 3.35. As the calculated F-ratio (3.36) is greater than the critical value of F (3.35), we reject the null hypothesis and conclude that there is a statistically significant difference between the average weight loss between three different diets: a low-carb diet, a low-fat diet, and a Mediterranean diet amongst the subjects tested.
III. Two-Way ANOVA
II. One-Way ANOVA
One-Way ANOVA is a statistical method used to determine whether there are any significant differences between the means of three or more independent groups. It is commonly used in scientific research to compare the means of different treatments or interventions. It is also used in business to compare the performance of different products or services.
It is used to test the null hypothesis that there are no significant differences in the means of the groups. The alternative hypothesis, on the other hand, suggests that at least one group mean is significantly different from the others. The hypotheses can be stated as follows:
Null hypothesis (H0): μ1 = μ2 = μ3 = … = μk (where k is the number of groups)
Alternative hypothesis (Ha): At least one group mean is significantly different from the others.
Before conducting One-Way ANOVA, it is important to ensure that certain assumptions are met. These assumptions include:
Independence: The observations within each group are independent of each other.
Normality: The dependent variable follows a normal distribution within each group.
Homogeneity of variances: The variances of the dependent variable are equal across all groups.
Violations of these assumptions can affect the validity of the results obtained from One-Way ANOVA. Therefore, it is crucial to assess these assumptions before proceeding with the analysis.
For example, suppose we want to test whether there is a difference in the average weight loss between three different diets: a low-carb diet, a low-fat diet, and a Mediterranean diet. We can use One-Way ANOVA to determine whether there is a statistically significant difference between the means of the three groups.
Another example is to understand if there is a difference in income based on education level. ANOVA can also be used to analyze the interaction between different levels of variables, such as advertising levels and pricing levels.
The following steps are involved in performing a One-Way ANOVA:
State the null hypothesis and alternative hypothesis: The null hypothesis is that all groups have the same mean, while the alternative hypothesis is that at least one group has a different mean.
Calculate the F-ratio: The F-ratio is calculated by dividing the between-group variance by the within-group variance. It measures the extent to which the group means differ from each other compared to the variability within each group.
The F-statistic is calculated using the following formula:
F = (Between-group variability) / (Within-group variability)
If the F-statistic is large enough to reject the null hypothesis, it indicates that there are significant differences in the means of the groups.
Determine the p-value: The p-value is the probability of obtaining a test statistic as extreme as the one calculated from the sample data, assuming the null hypothesis is true.
Compare the p-value to the significance level: If the p-value is less than the significance level (usually 0.05), then we reject the null hypothesis and conclude that there is a statistically significant difference between the means of the groups.
Example:
Here is an example problem to illustrate how to solve a One-Way ANOVA problem.
Suppose we want to test whether there is a difference in the average weight loss between three different diets: a low-carb diet, a low-fat diet, and a Mediterranean diet. We randomly select 10 people for each diet and record their weight loss over a period of 6 months. The following table shows the results:
|
Diet |
Sample Mean |
Sample Variance |
Sample Size |
|
Low-Carb |
10 |
2.5 |
10 |
|
Low-Fat |
8 |
1.5 |
10 |
|
Mediterranean |
12 |
3.0 |
10 |
We can use One-Way ANOVA to determine whether there is a statistically significant difference between the means of the three groups.
Let us look into the one-way ANOVA formula for calculating F-statistics or F-ratio:
F = MSB/MSW
In this formula,
F: The F-ratio is a test statistic used to determine whether the means of two or more groups are significantly different from each other.
MSB: The mean square between groups is the variance between the sample means.
MSW: The mean square within groups is the variance within each particular sample.
You usually employ statistical software like R, Excel, Stata, SPSS, etc. to conduct a one-way ANOVA because manual calculations can be tedious. Regardless of the software chosen or manual method, the resulting output will include the following table:
where:
SSR: regression sum of squares
SSE: error sum of squares
SST: total sum of squares (SST = SSR + SSE)
dfr: regression degrees of freedom (dfr = k-1)
dfe: error degrees of freedom (dfe = n-k)
dft: total degrees of freedom (dft = n-1)
k: total number of groups
n: total observations
MSR: regression mean square (MSR = SSR/dfr)
MSE: error mean square (MSE = SSE/dfe)
F: The F test statistic (F = MSR/MSE)
p: The p-value that corresponds to Fdfr, dfe
First, we calculate the Total Sum of Squares (SST):
where, k is the number of groups, ni is the sample size for group i, Xij is the jth observation in group i, and X̅ is the overall sample mean.
Using the values from the table, we get:
SST = (10−10.0)2+(10−10.0)2+(8−10.0)2+(8−10.0)2+(12−10.0)2+(12−10.0)2+(10−10.0)2+(10−10.0)2+(10−10.0)2+(10−10.0)2
SST = 40
Next, we calculate the Treatment Sum of Squares (SSR):
where, Ti is the sum of observations for group i, T̅ is the overall sum of observations divided by the total sample size, and ni is the sample size for group i.
Using the values from the table, we get:
SSR = 10(10−30/3)2 + 10(8−30/3)2+ 10(12−30/3)2
SSR = 8
Next, we calculate the Error Sum of Squares (SSE):
SSE = SST − SSR
Using the values from above, we get:
SSE = 40 – 8 = 32
Next, we calculate the degrees of freedom:
Degrees of freedom between groups (dfr) = k − 1 = 3 − 1 = 2
Degrees of freedom within groups (dfe) = n − k = 30 − 3 = 27
Degrees of freedom total (dft) = n − 1 = 30 − 1 = 29
Next, we calculate the mean square between groups:
MSR: regression mean square (MSR = SSR/dfr)
MSR = 8 / 2
MSR = 4
Next, we calculate the mean square within groups:
MSE: error mean square (MSE = SSE/dfe)
MSE = 1.19
Finally, we calculate the F-ratio:
F = MSB (or MSR) / MSW (or MSE)
F = 4 / 1.19
F = 3.36
To interpret the results, we compare the F-ratio to the critical value of F with the degree of freedom dfbetween (dfr) and dfwithin (dfe) at a significance level of 0.05. If the calculated F-ratio is greater than the critical value of F, we reject the null hypothesis and conclude that there is a statistically significant difference between the means. If the calculated F-ratio is lesser than the critical value of F, we fail to reject the null hypothesis and conclude that there is no significant difference between the means.
In this example the critical value of F for the degrees of freedom of 2 and 27 at 0.05 level of significance is 3.35. As the calculated F-ratio (3.36) is greater than the critical value of F (3.35), we reject the null hypothesis and conclude that there is a statistically significant difference between the the average weight loss between three different diets: a low-carb diet, a low-fat diet, and a Mediterranean diet amongst the subjects tested.
III. Two-Way ANOVA
Two-way ANOVA is an extension of one-way ANOVA that allows us to examine the effects of two independent variables, also known as factors, on a dependent variable. It enables us to determine whether there are significant differences in the means
of the dependent variable based on the levels of each factor individually and their interaction. It is used to estimate how the mean of a quantitative variable changes according to the levels of two categorical variables.
It is used to determine whether there is a statistically significant difference between the means of three or more independent groups that have been split on two variables (sometimes called “factors”). Two-Way ANOVA reveals the effect of
each independent variable on the dependent variable, as well as their relationship or interaction.
The two factors in a two-way ANOVA can be either categorical or continuous variables. The categorical variables are often referred to as “factors” or “groups,” while the continuous variables are known as “covariates.” The dependent variable, on the other hand, is typically a continuous variable.
Before conducting a two-way ANOVA, it is important to ensure that the assumptions of the analysis are met. These assumptions include:
Independence: The observations within each group and across groups are independent of each other.
Normality: The dependent variable follows a normal distribution within each combination of factor levels.
Homogeneity of variances: The variances of the dependent variable are equal across all combinations of factor levels.
Let us take an example of a dataset with two independent variables, Factor A and Factor B, and one dependent variable, the response variable.
Suppose we are conducting an experiment to study the effects of two factors, A and B, on the time it takes for individuals to complete a task. We have three levels for each factor, resulting in a 3×3 factorial design. The data might look like this:
|
A/B |
B1 |
B2 |
B3 |
|
A1 |
10 |
12 |
15 |
|
A2 |
14 |
16 |
18 |
|
A3 |
20 |
22 |
25 |
Here, the values in each cell represent the time taken to complete the task for a specific combination of Factor A and Factor B levels.
Now, let’s go through the steps of conducting a two-way ANOVA:
Step 1: Set up the hypotheses
Null Hypothesis (H0): There is no significant difference in means due to Factor A, Factor B, or their interaction.
Alternative Hypothesis (Ha): There is a significant difference in means due to at least one of the factors or their
interaction.
Step 2: Calculate means
Calculate the means for each combination of Factor A and Factor B, and the overall mean:
|
A/B |
B1 |
B2 |
B3 |
Mean |
|
A1 |
10 |
12 |
15 |
12.33 |
|
A2 |
14 |
16 |
18 |
16.0 |
|
A3 |
20 |
22 |
25 |
22.33 |
|
Mean |
14.67 |
16.67 |
19.33 |
16.89 (Overall Mean) |
Step 3: Calculate the sum of squares (SS)
Calculate the sum of squares for Factor A (SS A), Factor B (SS B), and the interaction (SS AB):
SS A = ∑nij (Yˉi⋅ − Yˉ⋅⋅)2
SS B = ∑nij (Yˉ⋅j − Yˉ⋅⋅)2
SS AB = ∑nij (Yˉij − Yˉi⋅ − Yˉ⋅j + Yˉ⋅⋅)2
SS A = (3 × (10−12.33)2) + (3× (14−16)2) +…
SS B = (3 × (10−14.67)2) + (3 × (12−16.67)2) +…
SS AB = (1× (10−12.33−14+16.89)2) +….
These calculations will give you the sum of squares for Factor A (SS A), Factor B (SS B), and the interaction (SS AB).
Step 4: Calculate the degrees of freedom (df)
Degrees of freedom for Factor A (df A), Factor B (df B), and the interaction (df AB) can be calculated based on the number of levels:
Here, a is the number of levels for Factor A (3 in this case), and b is the number of levels for Factor B (also 3).
df A = a−1 = 3−1 = 2
df B = b−1 = 3−1 = 2
df AB = (a−1) × (b−1) = 2 × 2 = 4
Step 5: Calculate the mean squares (MS)
Calculate the mean squares for Factor A (MS A), Factor B (MS B), and the interaction (MS AB):
MS A=SS A/df A
MS B=SS B/df B
MS AB=SS AB/df AB
Using the derived values from step 3:
MS A = SS A/df A = (Sum of squares for Factor A)/2
MS B = SS B/df B = (Sum of squares for Factor B)/2
MS AB = SS AB/df AB = (Sum of squares for Interaction)/4
Step 6: Calculate the F-ratio
Calculate the F-ratio for Factor A (F-ratio A), Factor B (F-ratio B), and the interaction (F-ratio AB):
F−ratio A = MS A/MSError
F−ratio B = MS B/ MSError
F−ratio AB = MS AB/ MSError
Where MSError is the mean square for error, which is calculated as:
MSError = Sum of squares for error / Degrees of freedom for error
This involves the calculation of the sum of squares for error (SSE), which is the total variation not accounted for by the factors being studied, and the degrees of freedom for error (dfError).
Additional calculations:
SSE = Total Sum of Squares − (SS A+SS B+SS AB)
dfError = Total degrees of freedom − (df A + df B + df AB)
Total degrees of freedom = n−1, where n is the total number of observations.
These additional calculations are necessary to complete the computation of MSError.
Once MSError is determined, you can use it in the F-ratio calculations.
Step 7: Determine significance
Compare the obtained F-ratios with critical values from the F-distribution table or use statistical software to find p-values. If the p-value is less than the significance level (commonly 0.05), reject the null hypothesis.
Step 8: Post-hoc tests (if needed)
If the ANOVA indicates significance, conduct post-hoc tests (e.g., Tukey’s HSD) to identify which specific groups differ.
Interpretation
If the p-value is less than 0.05, you would reject the null hypothesis, indicating that at least one of the factors or their interaction has a significant effect on the response variable. Examine effect sizes to understand the practical significance of the findings. If there is a significant interaction, be cautious about interpreting main effects alone. Interaction effects suggest that the impact of one factor depends on the level of another.