Consolidated ChatGPT API improvements: Improve Compatibility, add requests specific token limits, and textual stop sequences #734
+617
−165
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ocess_inference_result`
…o more closely align with OpenAI
…tion from the ChatGPT API
…und for textual stop sequence matches
This removes direct references the internals of BufferedOutput to allow for better abstraction
joshuacoles
force-pushed
the
stop-sequences
branch
from
1e54165 to
3731605
Compare
This was referenced Mar 7, 2025
Closed
|
@AlexCheema post our discussions yesterday I've chosen to consolidate all my changes up but not including structural generation (and hence function calling) into this PR to make it easier to review. I am going to add some tests and then this will be ready to review. I am going to rebase the latter two features on-top of this with a git history with less false starts |
|
I've added the tests and ensured bench.py runs successfully. This is now ready to review! |
joshuacoles
changed the title
|
I have used the official OpenAI SDK to perform the ChatGPT API tests as this is the reference client implementation. I have not however added this to any requirements / setup file as I could not find the other testing dependencies (eg pytest and pytest-asyncio) listed anywhere. |
…e zero-length token output
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Building on #720 and #716 this introduces textual stop sequences to the ChatGPT API, aligning with the OpenAI API Reference, allowing a request to specify that generation should cease after a given textual sequence has been generated.
This is implemented by buffering tokens on the final node of the inference loop until enough characters have been generated to guarantee that the stop sequences listed are not present. If a stop sequence is found in the middle of a token or spanning tokens, the tokens returned from the final node to the ChatGPT API will not be a faithful replication of the tokens that the inference process emitted, but instead the text prior to the stop sequence retokenised.
This data is passed around in the
GenerationOptionsobject as I introduced in #720. This PR should be considered after its two dependents, I will rebase this and apply any changes to it as needed.