Skip to content

Commit

Permalink
Add documentation for gRPC message and memory limits delphix#390
Browse files Browse the repository at this point in the history
  • Loading branch information
jeffngo committed Sep 10, 2021
1 parent 82dd1a3 commit 798560d
Show file tree
Hide file tree
Showing 6 changed files with 38 additions and 4 deletions.
1 change: 1 addition & 0 deletions docs/docs/Best_Practices/.pages
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,4 @@ arrange:
- Unicode_Data.md
- Working_with_Powershell.md
- Scratch_Paths.md
- Message_Limits.md
27 changes: 27 additions & 0 deletions docs/docs/Best_Practices/Message_Limits.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Message Limits

There are limits on how much data can be sent back and forth between the plugin and engine at a time. There are five scenarios where this comes into play:
1. Inputs sent from plugin to engine, as arguments to a [Plugin Operation](/References/Plugin_Operations.md). For example, the schema-defined `Repository` object that is provided as input to plugin operations.
2. Outputs sent back from the plugin to the engine, as the return values from plugin operations.
3. Exception messages and call stacks thrown by plugin code. For example, the `message` field within [User Visible Errors](/Best_Practices/User_Visible_Errors.md).
4. Inputs sent from plugin to engine, as arguments to a [Platform library](/References/Platform_Libraries.md) function. For example, the `message` field that is passed to `logger.debug`.
5. Outputs sent back from the plugin to the engine, as the return values from Platform Library functions. For example, the `stdout` resulting from a call to `libs.run_bash`.

For case 1 and 2, the total size of data must be less than 4 mebibytes (4 MiB).

For case 3, the total size of data must be less than 128 kibibytes (128 KiB).

For case 4 and 5, the total size of data must be less than 32 mebibytes (32 MiB).

The actual size of this information at runtime is dependent on how the Python interpreter chooses to represent the information, so it's not always possible to know ahead of time what the exact size will be.

Here are some examples of where problems may occur:
1. Using `libs.run_bash` to print the entire contents of a large file or stdout.
2. Using a single `logger` command with many pages of output.
3. Throwing an exception with a large message or stack trace.

## How to tell if the message size was exceeded
When a plugin operation or platform library callback fails with a `RpcError`.

## What to do if the maximum metadata or message size is exceeded
Please reach out to us via the [Virtualization SDK GitHub repository](https://github.com/delphix/virtualization-sdk/) for guidance.
2 changes: 2 additions & 0 deletions docs/docs/Best_Practices/User_Visible_Errors.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ message | String | Description of the failure to show the end user.
action | String | **Optional**. List of actions that the end user could take to fix the problem. If not provided, it defaults to `Contact the plugin author to correct the error.`
output | String | **Optional**. Output or stack trace from the failure to give the end user more information so that they can self diagnose. If not provided, it defaults to the stack trace of the failure.

!!! warning
There is a limit to how much data can be stored within the fields of a `UserError`. See [Message Limits](/Best_Practices/Message_Limits.md) for details.

## Example

Expand Down
2 changes: 2 additions & 0 deletions docs/docs/References/Logging.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ logger.setLevel(logging.DEBUG)

To avoid this complexity, add the `PlatformHandler` to the root logger. The root logger can be retrieved with `logging.getLogger()`.

!!! warning
There is a limit to how much data can be stored within a log message. See [Message Limits](/Best_Practices/Message_Limits.md) for details.

## Usage
Once the `PlatformHandler` has been added to the logger, logging is done with Python's [Logger](https://docs.python.org/2/library/logging.html#logger-objects) object. Below is a simple example including the basic setup code used above:
Expand Down
6 changes: 3 additions & 3 deletions docs/docs/References/Platform_Libraries.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Argument | Type | Description
-------- | ---- | -----------
remote_connection | [RemoteConnection](Classes.md#remoteconnection) | Connection associated with the remote host to run the command on.
command | String | Command to run on the host.
variables | dict[String, String] | **Optional**. Environement variables to set when running the command.
variables | dict[String, String] | **Optional**. Environment variables to set when running the command.
use_login_shell | boolean | **Optional**. Whether to use a login shell.
check | boolean | **Optional**. Whether or not to raise an exception if the `exit_code` in the `RunBashResponse` is non-zero.

Expand Down Expand Up @@ -85,7 +85,7 @@ Argument | Type | Description
-------- | ---- | -----------
remote_connection | [RemoteConnection](Classes.md#remoteconnection) | Connection associated with the remote host to run the command on.
command | String | Expect(Tcl) command to run.
variables | dict[String, String] | **Optional**. Environement variables to set when running the command.
variables | dict[String, String] | **Optional**. Environment variables to set when running the command.

### Returns
An object of `RunExpectResponse`
Expand Down Expand Up @@ -127,7 +127,7 @@ Argument | Type | Description
-------- | ---- | -----------
remote_connection | [RemoteConnection](Classes.md#remoteconnection) | Connection associated with the remote host to run the command on.
command | String | Command to run to the remote host.
variables | dict[String, String] | **Optional**. Environement variables to set when running the command.
variables | dict[String, String] | **Optional**. Environment variables to set when running the command.
check | boolean | **Optional**. Whether or not to raise an exception if the `exit_code` in the `RunPowershellResponse` is non-zero.

### Returns
Expand Down
4 changes: 3 additions & 1 deletion docs/docs/References/Plugin_Operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,9 @@
!!! warning
For each operation, the argument names must match exactly. For example, the Repository Discovery
operation must have a single argument named `source_connection`.


!!! warning
There are limits to how much data can be stored within the input and output of a plugin operation. See [Message Limits](/Best_Practices/Message_Limits.md) for details.

Plugin Operation | **Required** | Decorator | Delphix Engine Operations
---------------- | -------- | --------- | -------------------------
Expand Down

0 comments on commit 798560d

Please sign in to comment.