Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[autoparallel] update CommSpec to CommActions #1768

Conversation

YuliangLiu0306
Copy link
Contributor

What does this PR do

In previous PR #1735 , the sequential order has been added into CommSpec, which integrate into a CommAction object. In this PR, all the handlers using CommSpec wiil be updated to use CommAction.

Summary of code changes

  1. Add comm_type to CommAction object, and if the CommType is BEFORE, an identifier such as arg_index or key_for_kwarg will be added.
  2. We distinguish the parameters and non-parameters tensor. For Parameters grad, a hook function will be registered to param, however, a runtime communication node will be added for a non-parameter tensor.

@FrankLeeeee FrankLeeeee merged commit b0f7c8b into hpcaitech:main Oct 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants