1.2.2 Analyzing the different bootstrap results Assignment.
1.2.2 Analyzing the different bootstrap results
• 1) [2 Pts] Based on what you have learned it class, which of the three bootstrap methods (out-of-bag, 0.632, or 0.632+) do you expect to yield a generalization accuracy estimate from the training set that is closest to the true generalization performance of the model? Explain your reasoning in 1-3 sentences. (Tip: Think about optimistic and pessimistic bias).
< 0.632+ method would give a more accurate estimation, since the out-of-bag method gives a pessimistic results; the 0.632 method's optimistics influence may outweigh the out-of-bag's pessimistic influence, leading to a slightly optimistics results. Howeverm 0.632+ methods gives a corrected version compared to 0.632 method's result. >
• 2) [2 Pts] Based on your observations from the experiment in 1.2.1), which bootstrap approach (out-of-bag, 0.632, or 0.632+) yields an accuracy estimate from the training dataset that is closest to the test set accuracy from exercise 1.1.4? Is this reasonable? Explain your answer in 1-3 sentences. Also, to answer this question, assume that the test set accuracy from 1.1.4) is a perfect estimate of the true generalization accuracy of the model.
< From the output above, 0.632 method yields a more accurate estimation. Because compared to the estimation of 0.632+ method, 0.632 method's estimation would be slightly overfitting and lead to a optimistic outcome. However, overfitting would contribute to a "seemingly" more accurate result. >• 3) [2 Pts] Based on your observations from the experiment in 1.2.1), are the overall results consistent with what you expected in your answer above (question 1))? Explain your reasoning in 3-5 sentences. Also, to answer this question, assume that the test set accuracy from 1.1.4) is a perfect estimate of the true generalization accuracy of the model.
Tip: Discuss which methods are optimistically and pessimistically biased and whether this was expected.
< The expected results of the estimation accuracy would be 0.632+ > 0.632 > oob, the outcome is 0.632 > 0.632+ > oob. Although the outcome is not exactly as what I expected, but the accuracy of the estimation of out-of-bag method is much lower than other two. Also, the difference between the estimation accuracy of 0.632 and 0.632+ is not very big, but this may due to the overfitting of the 0.632 method. Therefore, I would say the overall results are consistent with what I expected. >
Based on the lessons from class, I expect that a 0.632+ bootstrap method to yield a generalization accuracy estimate from the training set that is closest to the model's true generalization performance. This is because the out-of-bag approach offers a pessimistic outcome. The 0.632 approaches provide an optimistic influence that has the possibility of outweighing the pessimistic influence from the out-of-bag. This leads to a partially optimistic result. When we consider the 0.632+ approach, we can get an accurate version when we compare it with the outcome of the 0.632.
Question 2
Based on the foregoing output, 0.632 gives a more accur...