Make user-submitted data available for new tasks

[yk]

We want to be able to run fully on user-submitted data. Thus, when a user is asked to provide a prompt, and does so, we would like to be able to then re-use that prompt for other tasks, for example the "rank prompts" tasks, but also the "act as assistant" task, where a user is asked to answer as if they were the assistant to a given prompt. That prompt should come from the database of user-submitted prompts.

Extensions to this could be that the user-submitted prompts are sampled according to how well they are ranked against other submitted prompts in the ranking task, but it's not necessary for start.

The same requirement exists for when users submit user-answers or assistant-answers, we would also like to be able to re-use those as data for further tasks (so we can build up conversations over time, one message per task).

[mjagkow]

Hi, I would like to contribute to this task, but before this I have a couple of questions to ask:

• What do you consider to be a thread? From what I can see right now in the code a thread is only formed by a pair of ("task-started-post", "user-reply-post"). It feels like all messages of one conversation should share the same thread, is it correct?
• Do you consider only linear conversations? With parent post id one could construct a number of conversation variants branching from one or more intermediate posts.

Also are there any other docs describing the algorithm, data scheme and interaction protocol you would like to implement besides the readmes and high level protocol architecture article on Notion? This would help a lot in understanding :)

[yk]

hey thanks so much for being here :)

What do you consider to be a thread? From what I can see right now in the code a thread is only formed by a pair of ("task-started-post", "user-reply-post"). It feels like all messages of one conversation should share the same thread, is it correct?

the idea (which might not yet be fully reflected in the code) is that the interactions of the system with the users result in tree-structures. At the root is a prompt, which a user has entered. We then send out this prompt as a task to multiple users, each of which is tasked to "play" the assistant and give a response. Given those responses, we can do two things: First, we can have other people rank the responses, which will be useful for training the reward model. Second, we can take each response and form a thread from (initial prompt, assistant response), and give each of these threads as a task to multiple users. Thus, each thread results in multiple replies from users (who are now "playing" the user). Again, we can rank those and also combine each (initial prompt, assistant reply, user reply) as one thread to continue the tasks.

So, in my view, a thread is a chain of messages between user and assistant. We build threads by going down one path of the tree we collect by people fulfilling their tasks. I hope that makes it a bit more clear and answers the second question: The result is always a linear conversation, but during data collection, we collect a tree.

Also are there any other docs describing the algorithm, data scheme and interaction protocol you would like to implement besides the readmes and high level protocol architecture article on Notion? This would help a lot in understanding :)

Not yet, no, but you're right it's important we improve this!

[mjagkow]

Hi there, thank you for an explanation, it helped a lot :)

I guess PostgreSQL is capable of recursive queries to walk over the posts of one thread using parent_id, but for now I'd suggest a much simpler way to sample a conversation:

1. (DB) pick random thread_id (which is the root post id)
2. (DB) select all posts with this thread_id
3. (Backend) pick a random message
4. (Backend) trace back the conversation line back from this post to the root using parent_id

Later it might also be useful to have a more customizable way to pick the last message of a conversation (the one we choose at 3.) instead of a random one.

I also have some suggestions for the data model:

• Currently ACK request generates a new post entry which is used to link subsequent interaction to the "task description message". I think work_package (aka task) is a better place for this link, it will keep posts clean and easy to work with. So at the time of interaction the task may be identified by (frontend_key, user_id, task_message_id). On the other hand post.frontend_post_id seems unnecessary in this case, I can't see other use for it yet.
• When the user replies to the task message (user_reply task) and a new post is inserted we need to link it to its thread and parent post. At this moment the information from the frontend is sufficient only to identify the task. Thus we need to link the task to its post of origin, parent_post_id should do the trick.
• Posts may need a flag to designate the end of this line of conversation. In this case we'll also need a way to determine whether this post is a valid end of a conversation, maybe a new task type.
• If the data is generated not only by users but by the RL agents as well then the posts may need to have a flag to distinguish between them. If not could you please clarify how is_assistant field is calculated when the conversation is formed in the backend? Should it just be an alternating pattern starting from the initial prompt?

As a bonus I've also updated the sequence diagram of the interaction between user and backend. Now it should better reflect what happens at the time of a request:

sequenceDiagram
  par Task Creation
    User DMs / Channel->>Frontend: Asks for a new task
		Frontend ->> Backend: Requests new task
		Backend  ->> DB: Inserts new task and links it<br/>to the thread and the parent post
    DB ->> Backend: Generates task_id
		Backend  ->> Frontend: Returns the task with type and payload
		alt Frontend displays the task
		Frontend ->> User DMs / Channel: Presents the task in a message and asks the user to interact
		Frontend ->> Backend: Sends ACK with presented "task description message" ID
		Backend ->> DB: Updates task with "task description message" ID<br/>and set "ack"=True
		else Frontend failed to present the task
		Frontend ->> Backend: Sends NACK for a given task
		Backend ->> DB: Updates the task with "ack"=False
		end
	end
	par Task Fulfillment
		User DMs / Channel ->> Frontend: Interacts with the post (reply, rating, etc.)
		Frontend ->> Backend: Posts interaction along with "task description message" ID
		Backend ->> DB: Updates the task with "done"=True
		Backend ->> DB: Inserts new post with the interaction data<br/>and links it to the parent post using the data from the task
		Backend ->> Frontend: Sends "Task Done"
		Frontend ->> User DMs / Channel: Replies "thank you"
	end

Loading

There may be an insignificant theoretical race condition when the user responds before ACK has been delivered, but I think backend should back this request off and frontend should handle it.
This diagram is also available at https://www.notion.so/Open-Assistant-protocol-25a906a38efd488cac9414a6c8022e24

[yk]

Currently ACK request generates a new post entry which is used to link subsequent interaction to the "task description message". I think work_package (aka task) is a better place for this link, it will keep posts clean and easy to work with. So at the time of interaction the task may be identified by (frontend_key, user_id, task_message_id). On the other hand post.frontend_post_id seems unnecessary in this case, I can't see other use for it yet.

Yes, you're right. The initial goal was to make the frontends as stateless as possible, but this seems less and less viable, so we'll have to live with a degree of state, i.e. for the frontend to at least be able to link some reply to the initial message of the task (could be done by traversing reply structures). The issue is that there might not always be the exact mapping of 1 task description <-> 1 message, but I guess when that breaks, we can think of a solution then.

When the user replies to the task message (user_reply task) and a new post is inserted we need to link it to its thread and parent post. At this moment the information from the frontend is sufficient only to identify the task. Thus we need to link the task to its post of origin, parent_post_id should do the trick.

Agree

Posts may need a flag to designate the end of this line of conversation. In this case we'll also need a way to determine whether this post is a valid end of a conversation, maybe a new task type.

I suggest we attach some "depth" level to posts, i.e. how far down the conversation they are, so always their parent's depth plus 1, then we can either have a hard cap on depth or we just sample in inverse proportion to depth or so.

If the data is generated not only by users but by the RL agents as well then the posts may need to have a flag to distinguish between them. If not could you please clarify how is_assistant field is calculated when the conversation is formed in the backend? Should it just be an alternating pattern starting from the initial prompt?

The current idea is that threads are turn-taking, but does not distinguish whether the system created one message or a human. I.e. is_assistant just indicates that the message is written from the perspective of the assistant, but could still be fulfilled by a human worker. If it does not already exist, we could add to each interaction explicitly which user made that interaction, with null values indicating no user, i.e. the system itself.

Thanks a lot for updating the diagram, very good thinking!

[andreaskoepf]

• When the user replies to the task message (user_reply task) and a new post is inserted we need to link it to its thread and parent post. At this moment the information from the frontend is sufficient only to identify the task. Thus we need to link the task to its post of origin, parent_post_id should do the trick.

Currently I think the main problem is the task generation. When the backend receives the user's reply it adds it nicely as child-post in PromptRepository.store_text_reply(). What is missing IMO is the original message in the database. When I designed the thread-tree-structure of the post-table my idea was that the source posts of a conversation would also reside in this tree so that the task would refer to them.

[andreaskoepf]

I guess PostgreSQL is capable of recursive queries to walk over the posts of one thread using parent_id, but for now I'd suggest a much simpler way to sample a conversation:

The posts table has a thread_id column which is always the id of the root post of the thread. This should allow to fetch at thread-level and for most cases hopefully will not require recursive-queries.

[andreaskoepf]

@mjagkow I can quickly make some of your suggested changes. What is your state working on this? Are you on the OA discord?