I wrote a machine learning paper using a novel approach on a specific dataset, which produced some positive results. I trained several models, evaluated them, and conducted thorough interpretation and discussion based on my findings. One of the reviewers asked for a sensitivity analysis on some preprocessing parameters/algorithms. Interestingly, one change led to slightly better outcomes than my original method.
My question is: what are the expectations in this situation? Do I need to rewrite the entire paper, or should I just report this observation in the sensitivity analysis? While it’s great that the changes improved the results, I’m frustrated at the thought of having to rewrite much of the interpretation (like feature importance, graphs, discussion, etc.) based on the new findings. What are your thoughts and experiences?
What does “slightly” mean in this context? I would assume the improvement is minimal and may not apply to other datasets. Just include the performance in the sensitivity analysis; there’s no need to redo all your work for something that could be due to noise.
You might consider conducting a statistical test to determine if the performance difference between the current best model and the new best model is significant. If it isn’t significant, you can still mention it in the paper and indicate that the statistical test shows no statistically significant difference. Then, you can choose either algorithm for feature importance analysis, which would allow you to avoid major changes to the paper.
I would personally prefer to redo it because I believe that improvement is significant. However, if you choose not to, you could use that as motivation to conduct a thorough sensitivity analysis in your future work.
What else do you need to do besides rerunning the existing code? For me, the feature importance and graphs will work without changes if you’re only modifying the preprocessing. Then, you just need to find and replace a few numbers. The difference between 79% and 82% isn’t significant, but saying “over 80” is a fairly notable improvement.