Not able to recreate the benchmark scores using embedding gemma. (for MTEB SCIDOCS).
When I went through your research paper , I was able to find the ndcg@10 score for SCIDOCS retrieval task from table 11. I am trying to recreate this score using sentence transformers library as mentioned in your official documentation page. I am still not able to recreate this. Tried checking the implementation with bge m3's score for which I was able to exactly reproduce . By this I confirmed there is no flaw in the implementation code. Please let me know is I have to make use of the quantized model to recreate the score or am I missing out something. Also I am adding prefix to the queries and documents as mentioned in the paper. thanks in advance.
Hi @sairam-rs , Thank you for bringing this to our attention and for the detailed steps you have taken. To help us investigate this discrepancy, could you please provide a few more details like - What version of the sentence-transformers library are you using and Could you share a minimal, reproducible code snippet that shows how you are loading the model and running the evaluation on SCIDOCS and the referred paper?
thanks @Renu11 for replying . My sentence-transformers version is > 5.0.0. However , I resolved the issue by changing the transformers version from the latest general release to the specific tag for embedding-gemma preview version from transformers github. Now the SCIDOCS results are very close to the paper. Thanks once again.