Protect your OpenAI application with Arcjet

[giscus[bot]]

Protect your OpenAI application with Arcjet

Protect your OpenAI chat interface from abuse with Arcjet rate limiting and bot protection.

https://docs.arcjet.com/integrations/openai/

[dakdevs]

Can we dynamically set the rate limit based on the user?

Also, is there any way to update the bucket based on the actual OpenAI usage data?

[davidmytton]

Can we dynamically set the rate limit based on the user?

Yes! The rule config is all just code so you can dynamically set the limit. Our recommended pattern is to define the aj client outside of the handler and then use withRule to configure the rule dynamically inside the handler. This allows you to change it per request. There is an example here where the limit is set depending on whether the user is logged in or not. We have another example where the limit is read in from a database and set in the session.

Also, is there any way to update the bucket based on the actual OpenAI usage data?

You can use something like https://github.com/hmarr/openai-chat-tokens to estimate the number of tokens required per call. This is the library we use to set the value for requested which is what is deducted from the token bucket.

To get the real data from OpenAI you'd need to query their API or record it somewhere else. If you store it in your own database then you can use the example above to set the bucket config based on values from the database.