I can only imagine the insights/gains from authoring a paper proposing a standard, having never written one myself. Perhaps learning about a model’s internal performance is one. If you’ve written anything comparable, did you think your background was useful? Would you suggest?
Writing a benchmark paper is a great way to dive deep into a model’s performance and share valuable insights with the research community. It can boost your reputation, open up new opportunities, and provide a standard reference for others. If you have the right background, it’s worth a shot.
Domain knowledge helps, but collaboration is key. Focus on problem, solution, evaluation, and community.
When you say ‘benchmark paper,’ are you referring to a paper that proposes a new benchmark or one that analyzes the performance of existing models on established benchmarks?
I think it is really helpful whether you are starting as a PhD student or if you intend to conduct research in that field. Because setting up the benchmark would imply that you have a number of strategies ready to go whenever you decide to conduct additional relevant study. In my experience, putting these strategies into practice and experimenting with them has always helped me come up with fresh concepts and recognize holes in the literature. Finally, albeit it’s not very significant, it might function as a citation generator.