-
Notifications
You must be signed in to change notification settings - Fork 3.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
#47- Create API endpoints that return leaderboards #250
Conversation
- New leaderboards router - Two endpoints /api/v1/leaderboards/assistant & /api/v1/leaderboards/prompter - Function in prompt_repository to calculate user scores
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very nice, very clean thank you! I think this is a good first step.
Could you rename the route prefix to be /experimental/leaderboards
? just so we have nobody fully depend on that yet
Thanks @yk . That makes sense |
Thanks a lot for the first stats-based leaderboard. This is definitely a metric we should include but alone it encourages the user to flood the db with texts. I will try to write down current ideas that are floating araound for the user scoring. Regarding online-calculation: Could you pehaps create a litte benchmark test that measures how long the message-count leaderboard query takes for of a fake-DB with 500k messages (each 500 charaters long) and 1k users thtat randomly are the authors of the fake messages? That would be helpful to get a first idea of the performance. It would also be a good test-script that we could use for other scenarios. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we can start with this.
#47
Leaderboard endpoints proof of concept
There's a couple of ways to structure the endpoints. Using this structure we could end up with:
/api/v1/leaderboards/create/prompter/{timespan}
... create/combined, create/assistant
/api/v1/leaderboards/evaluate/prompter/{timespan}
... evaluate/user, evaluate/combined ...