Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improved consistency check #203

Merged
merged 15 commits into from
Feb 3, 2025
Prev Previous commit
Next Next commit
Add classification to inconsistency check
wasnertobias committed Jan 30, 2025

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
commit d9c8fb5f56ba0e15c1ff7f3037acfbd4f9be88cb
22 changes: 10 additions & 12 deletions app/pipeline/prompts/inconsistency_check_prompts.py
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
basic_prompt = """\
<Instruction>
As detail-oriented expert, find inconsistencies between the provided problem statement and the template repository of \
As detail-oriented expert, find inconsistencies between the provided problem statement and a template file of \
a programming exercise.
The student will use the the template repository as a framework to write code that solves the problem statement, it might be missing implementations which the students have to do. You are also provided with the solution repository, which should contain the full and correct implementation of the exercise. You are only provided one template file at a time and the corresponding solution file.
The student will use multiple template files to write code that solves the problem statement. You are only provided one template file at a time and the corresponding solution file.
Checks:
- Given the problem statement, identify any missing or incorrect information in the template repository.
- Given the template repository, identify any missing or incorrect information in the problem statement.
- Ensure that the theme of the problem statement is consistent with the template repository.
- Given the problem statement, identify any missing or incorrect information in the template file. (TEMPLATE_MISSING_INFO)
- Given the template file, identify any missing or incorrect information in the problem statement. (PROBLEM_SATEMENT_MISSING_INFO)
- Ensure that the theme of the problem statement is consistent with the template files. (THEME_INCONSISTENCY)
- Ensure that the problem statement is clear and concise and it covers everything that the student needs to know in \
order to solve the exercise.
order to solve the exercise. (PROBLEM_STATEMENT_CLARITY)
It is not an inconsistency, if the problem statement clearly states that the student is responsible for writing a \
specific part of the code.
@@ -19,13 +19,13 @@
{problem_statement}
</Problem Statement>
<TemplateRepository>
<Template File>
{template_repository}
</TemplateRepository>
</Template File>
<SolutionRepository>
<Solution File>
{solution_repository}
</SolutionRepository>
</Solution File>
<Response>
Be smart about it, give a structured and actionable response that an instructor can use to significantly improve the \
@@ -34,7 +34,5 @@
will be confused and frustrated. This is a high stakes exam, so we need to be very thorough.
You will be legally responsible for the quality of the exercise, so make sure you do the absolute best job possible, \
otherwise you will be held accountable in the court of law. Do not quote whole files! 🔫
Do not mention that the template repository is missing requirements mentioned in the problem statement, that is not an inconsistency. The students are supposed to implement those parts, without any guidance such as comments or partial code.
Do only list the inconsistencies (unnumbered), no need to mention something before or after the list of inconsistencies.
</Response>
"""
Loading