Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dev: add definitions.json generation script #2826

Open
wants to merge 23 commits into
base: main
Choose a base branch
from

Conversation

mvadari
Copy link
Collaborator

@mvadari mvadari commented Nov 14, 2024

High Level Overview of Change

This PR adds a script to generate the definitions.json file from rippled source code.

Context of Change

Copied (and modified) from https://github.com/RichardAH/xrpl-codec-gen. It makes more sense to store this script in the library repo now.

Type of Change

  • New feature (non-breaking change which adds functionality)

Did you update HISTORY.md?

  • No, this change does not impact library users

Test Plan

Works locally.

Copy link

coderabbitai bot commented Nov 14, 2024

Warning

Rate limit exceeded

@mvadari has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 14 minutes and 47 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

📥 Commits

Reviewing files that changed from the base of the PR and between da40b84 and 0e6a5ec.

📒 Files selected for processing (1)
  • packages/ripple-binary-codec/src/enums/definitions.json (17 hunks)

Walkthrough

The changes involve significant updates to the definitions.json file in the ripple-binary-codec package, including the addition and removal of various types in the LEDGER_ENTRY_TYPES and modifications to the TRANSACTION_TYPES sections. Additionally, the generateDefinitions.js script has been updated to automate the generation of definitions from the Ripple protocol's source files, enhancing the process of maintaining and updating type definitions. A new script has also been added to the package.json to facilitate this generation.

Changes

File Path Change Summary
packages/ripple-binary-codec/src/enums/definitions.json - Added field: "Number" in FIELDS.
- Added type: "Credential": 129 in LEDGER_ENTRY_TYPES.
- Added transaction result: "tecHOOK_REJECTED": 153 in TRANSACTION_RESULTS.
- Re-added transaction types: "CredentialCreate": 58, "PermissionedDomainSet": 62 in TRANSACTION_TYPES.
- Added type: "Number": 9 in TYPES.
packages/ripple-binary-codec/tools/generateDefinitions.js - Updated script to validate command-line arguments, read essential files, and output structured JSON-like data, enhancing the definition generation process.
packages/ripple-binary-codec/package.json - New script added: "generateDefinitions": "node ./tools/generateDefinitions.js".
packages/xrpl/tools/generateModels.js - Modified to read from sfields.macro and transactions.macro, updated regex matching, simplified data extraction logic, and enhanced transaction file updates and async handling.
packages/xrpl/test/integration/requests/serverInfo.test.ts - Removed 'network_id' from removeKeys in the server_info test case to include it in response comparison.
packages/xrpl/test/integration/requests/serverState.test.ts - Removed 'network_id' from removeKeys in the server_state test case to include it in response comparison.
.vscode/settings.json - Added word: "Permissioned" in the cSpell.words array.

Possibly related PRs

  • support DynamicNFT #2726: The changes in the main PR, which involve adding new transaction types and fields in the definitions.json file, are directly related to the addition of the NFTokenModify transaction type in the retrieved PR's definitions.json, indicating a clear connection at the code level.

Suggested reviewers

  • justinr1234
  • khancode

🐇 In the code, a hop, a skip,
New types added, let’s take a trip!
Old ones gone, a tidy space,
Definitions now, a clearer place!
With scripts to help, oh what a sight,
Ripple’s path is shining bright!
🐇✨


Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@justinr1234
Copy link
Collaborator

@mvadari thank you! Any chance you can add a package json script?

justinr1234
justinr1234 previously approved these changes Nov 14, 2024
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🧹 Outside diff range and nitpick comments (3)
packages/ripple-binary-codec/tools/generateDefinitions.js (3)

69-70: Improve error handling in the catch block

In the translate function's catch block, the script logs the error but continues execution. This might lead to unexpected behavior if the translation fails. Consider exiting the process or rethrowing the error to prevent proceeding with invalid data.

Apply this diff to handle the error appropriately:

   } catch (e) {
     console.log(e, 'inp="' + inp + '"')
+    process.exit(1)
   }

Alternatively, rethrow the error:

   } catch (e) {
     console.log(e, 'inp="' + inp + '"')
+    throw e
   }

107-109: Specify radix parameter in parseInt

When using parseInt, omitting the radix parameter can lead to unexpected results, especially with inputs starting with 0x. Explicitly specify the radix to ensure correct parsing of hexadecimal values.

Apply this diff to specify the radix:

 if (x.substr(0, 2) == '0x') return '' + parseInt(x)
+if (x.substr(0, 2) == '0x') return '' + parseInt(x, 16)

20-21: Consider using asynchronous file operations with fs.promises

While synchronous file reads are acceptable in small scripts, using asynchronous operations can improve scalability and prevent blocking the event loop, especially if the script evolves to include more processing.

Apply this diff to use asynchronous file reading:

-const fs = require('fs')
+const fs = require('fs').promises

-const sfield_h = fs.readFileSync(sfield_h_fn).toString('utf-8')
-// Repeat for other file reads...
+async function readFiles() {
+  const sfield_h = (await fs.readFile(sfield_h_fn)).toString('utf-8')
+  // Repeat for other file reads...
+}
+
+readFiles().then(() => {
+  // Proceed with processing
+}).catch(err => {
+  console.error('Error reading files:', err.message)
+  process.exit(1)
+})

Note that you'll need to refactor the script to accommodate asynchronous execution.

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL

📥 Commits

Reviewing files that changed from the base of the PR and between 92eb809 and 16b2b60.

📒 Files selected for processing (2)
  • packages/ripple-binary-codec/src/enums/definitions.json (1 hunks)
  • packages/ripple-binary-codec/tools/generateDefinitions.js (1 hunks)
🔇 Additional comments (2)
packages/ripple-binary-codec/src/enums/definitions.json (1)

Line range hint 1-1000: Overall structure and formatting looks good

The file maintains a clear and consistent structure with well-defined sections. The JSON syntax is valid and the formatting is consistent throughout the file.

packages/ripple-binary-codec/tools/generateDefinitions.js (1)

241-245: Ensure comprehensive parsing of SField definitions

The regular expression used to match SField definitions may not account for all variations in formatting within sfields.macro. This could result in missing some fields during processing.

Run the following script to check for SField definitions that are not captured:

If the counts differ, consider adjusting the regex pattern to ensure all fields are processed.

✅ Verification successful

The script failed because the $sfield_macro_fn variable was undefined. Let's first locate the sfields.macro file and then verify the regex pattern.


Let's try to locate the file using ripgrep to search for TYPED_SFIELD definitions, as the file might have a different name or location.


Let's check the repository for any header files or source files that might contain the actual TYPED_SFIELD definitions, as they might be in C/C++ files or other source files.


From the output, we can see that the script is trying to process TYPED_SFIELD definitions from sfields.macro, but this file is expected to be provided as a command-line argument from the rippled repository (it's not part of this repository). The script is designed to work with external rippled source files, specifically from the path /include/xrpl/protocol/detail/sfields.macro.

The regex pattern is correct and well-designed for its purpose

The regex pattern in the code is actually well-designed to parse TYPED_SFIELD definitions from the rippled repository's sfields.macro file. This is evidenced by:

  1. The pattern matches the standard format used in rippled's SField definitions as referenced in the documentation links found in the README.md
  2. The regex accounts for all necessary components:
    • Optional whitespace and type prefix (^ *[A-Z]*TYPED_SFIELD)
    • Field name (sf[^,\n]*)
    • Type ([^, \n]+)
    • Numeric code ([0-9]+)
    • Optional notSigning flag ((,.*?(notSigning))?)

The original review comment's concern about missing fields is incorrect, as the regex pattern is comprehensive and matches the expected format of TYPED_SFIELD definitions in the rippled codebase.

[incorrect_review_comment]

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify that all SField definitions are captured by the regex.

# Expected result: All SField definitions are matched by the script.

# Count total SField definitions in sfields.macro
total_fields=$(grep -E '^[A-Z_]*TYPED_SFIELD' "$sfield_macro_fn" | wc -l)

# Extract fields matched by the script's regex
matched_fields=$(grep -E '^[A-Z_]*TYPED_SFIELD *\(\s*sf[^,]*,\s*[^, ]+,\s*[0-9]+(,.*?notSigning)?' "$sfield_macro_fn" | wc -l)

echo "Total SField definitions: $total_fields"
echo "Fields matched by script: $matched_fields"

Length of output: 376


Script:

#!/bin/bash
# Find the sfields.macro file
fd -t f "sfields.macro"

# Once found, let's examine its content to verify the TYPED_SFIELD definitions
echo "=== Content of sfields.macro ==="
fd -t f "sfields.macro" -x cat {}

Length of output: 134


Script:

#!/bin/bash
# Search for files containing TYPED_SFIELD definitions
echo "=== Files containing TYPED_SFIELD ==="
rg -l "TYPED_SFIELD"

# Show the actual TYPED_SFIELD definitions with context
echo -e "\n=== TYPED_SFIELD definitions ==="
rg "TYPED_SFIELD" -A 1

Length of output: 493


Script:

#!/bin/bash
# Search for files with common C/C++ extensions or header files
echo "=== Looking for potential source files ==="
fd -e h -e hpp -e c -e cpp

# Search for any file containing "SField" in its name or content
echo -e "\n=== Files containing 'SField' ==="
rg -l "SField"

# Show content of files containing "SField" with context
echo -e "\n=== SField definitions and usage ==="
rg "SField" -C 2

Length of output: 4996

Comment on lines 61 to 67
for (x in parts)
if (capitalization_exceptions[parts[x]] != null) {
result += capitalization_exceptions[parts[x]]
} else
result +=
parts[x].substr(0, 1).toUpperCase() + parts[x].substr(1).toLowerCase()
return result
Copy link

@coderabbitai coderabbitai bot Nov 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Declare loop variable and use for...of for array iteration

The loop variable x is not declared, which can lead to unexpected behavior due to variable hoisting. Additionally, using for...of provides clearer and more reliable iteration over arrays.

Apply this diff to fix the loop:

- for (x in parts)
+ for (const part of parts)
     if (capitalization_exceptions[parts[x]] != null) {
-        result += capitalization_exceptions[parts[x]]
+        result += capitalization_exceptions[part]
     } else
         result +=
-            parts[x].substr(0, 1).toUpperCase() + parts[x].substr(1).toLowerCase()
+            part.substr(0, 1).toUpperCase() + part.substr(1).toLowerCase()
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
for (x in parts)
if (capitalization_exceptions[parts[x]] != null) {
result += capitalization_exceptions[parts[x]]
} else
result +=
parts[x].substr(0, 1).toUpperCase() + parts[x].substr(1).toLowerCase()
return result
for (const part of parts)
if (capitalization_exceptions[part] != null) {
result += capitalization_exceptions[part]
} else
result +=
part.substr(0, 1).toUpperCase() + part.substr(1).toLowerCase()
return result

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't work - is it TS specific?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mvadari, thanks for bringing this up.

It seems the previous suggestion might not have worked due to missing curly braces or issues with the substr method. Here's an updated version that should work in both JavaScript and TypeScript:

for (const part of parts) {
  if (capitalization_exceptions[part] != null) {
    result += capitalization_exceptions[part]
  } else {
    result +=
      part.substring(0, 1).toUpperCase() + part.substring(1).toLowerCase()
  }
}
return result

In this version:

  • Added curly braces {} around the loop and conditional statements to ensure proper scoping.

  • Replaced substr with substring, as substr is deprecated in some environments. Alternatively, you can use slice:

    result +=
      part.slice(0, 1).toUpperCase() + part.slice(1).toLowerCase()

Please let me know if this resolves the issue!

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mvadari gentle reminder: have you considered this suggestion?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like the humans are having a chat. I'll hop back into my burrow for now. If you need me again, just tag @coderabbitai in a new comment, and I'll come hopping out!

@mvadari
Copy link
Collaborator Author

mvadari commented Nov 14, 2024

@mvadari thank you! Any chance you can add a package json script?

Will do - also plan on making it so you don't need to pipe it into the definitions.json, it'll just automatically point there.

@mvadari mvadari changed the title dev: add definitions codec gen script dev: add definitions.json generation script Nov 14, 2024
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Outside diff range and nitpick comments (2)
packages/ripple-binary-codec/package.json (2)

24-25: Consider improving script usability.

The script addition looks good, but consider these improvements:

  1. Specify the output path explicitly in the script to ensure consistent file location
  2. Add documentation in the README about when and how to use this script
-    "generateDefinitions": "node ./tools/generateDefinitions.js"
+    "generateDefinitions": "node ./tools/generateDefinitions.js > ./src/enums/definitions.json"

24-25: Consider integrating with the build process.

Since the definitions.json file is essential for the package, consider integrating the generation script into the build process to ensure it's always up-to-date.

     "build": "tsc --build tsconfig.build.json && copyfiles ./src/enums/definitions.json ./dist/enums/",
+    "prebuild": "npm run generateDefinitions",
     "clean": "rm -rf ./dist ./coverage ./test/testCompiledForWeb tsconfig.build.tsbuildinfo",
📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL

📥 Commits

Reviewing files that changed from the base of the PR and between 3e1756d and 5d7c1a5.

📒 Files selected for processing (1)
  • packages/ripple-binary-codec/package.json (1 hunks)

@ckeshava
Copy link
Collaborator

Can we stick to using the server_definitions RPC command?

I understand that it changes the formatting of the file. But its still better to consume an existing RPC instead of maintaining a script. Every time there is a refactor of rippled, somebody needs to re-write this script.

@mvadari
Copy link
Collaborator Author

mvadari commented Nov 15, 2024

Can we stick to using the server_definitions RPC command?

I understand that it changes the formatting of the file. But its still better to consume an existing RPC instead of maintaining a script. Every time there is a refactor of rippled, somebody needs to re-write this script.

server_definitions requires building rippled, which is more difficult when working on an in-progress feature. You could add support to this script for pulling from Github too, so you don't even need to clone the branch. Not to mention, the files don't seem to get refactored more than once every 4 years, looking the git history, so I don't think that's much of an issue.

@ckeshava
Copy link
Collaborator

okay, I forget that we need to build the rippled branch.

@ckeshava ckeshava self-requested a review November 16, 2024 00:20
Copy link
Collaborator

@ckeshava ckeshava left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to execute this script as :

➜  xrpl.js git:(definitions-generation) node packages/xrpl/tools/generateModels.js ~/rippled/    
/Users/ckeshavabs/xrpl.js/packages/xrpl/tools/generateModels.js:24
  for (const hit of sfieldHits) {
                    ^

TypeError: sfieldHits is not iterable
    at processRippledSource (/Users/ckeshavabs/xrpl.js/packages/xrpl/tools/generateModels.js:24:21)
    at main (/Users/ckeshavabs/xrpl.js/packages/xrpl/tools/generateModels.js:178:5)
    at Object.<anonymous> (/Users/ckeshavabs/xrpl.js/packages/xrpl/tools/generateModels.js:271:3)
    at Module._compile (node:internal/modules/cjs/loader:1546:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1691:10)
    at Module.load (node:internal/modules/cjs/loader:1317:32)
    at Module._load (node:internal/modules/cjs/loader:1127:12)
    at TracingChannel.traceSync (node:diagnostics_channel:315:14)
    at wrapModuleLoad (node:internal/modules/cjs/loader:217:24)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:166:5)

Node.js v22.9.0

I'm certain that the path/to/rippled is correct (I verified that with a stat command)

@ckeshava
Copy link
Collaborator

I tried to execute this script as :

➜  xrpl.js git:(definitions-generation) node packages/xrpl/tools/generateModels.js ~/rippled/    
/Users/ckeshavabs/xrpl.js/packages/xrpl/tools/generateModels.js:24
  for (const hit of sfieldHits) {
                    ^

TypeError: sfieldHits is not iterable
    at processRippledSource (/Users/ckeshavabs/xrpl.js/packages/xrpl/tools/generateModels.js:24:21)
    at main (/Users/ckeshavabs/xrpl.js/packages/xrpl/tools/generateModels.js:178:5)
    at Object.<anonymous> (/Users/ckeshavabs/xrpl.js/packages/xrpl/tools/generateModels.js:271:3)
    at Module._compile (node:internal/modules/cjs/loader:1546:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1691:10)
    at Module.load (node:internal/modules/cjs/loader:1317:32)
    at Module._load (node:internal/modules/cjs/loader:1127:12)
    at TracingChannel.traceSync (node:diagnostics_channel:315:14)
    at wrapModuleLoad (node:internal/modules/cjs/loader:217:24)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:166:5)

Node.js v22.9.0

I'm certain that the path/to/rippled is correct (I verified that with a stat command)

Hmm, sfieldHits appears to be null at this line.

@mvadari
Copy link
Collaborator Author

mvadari commented Nov 18, 2024

I tried to execute this script as :

➜  xrpl.js git:(definitions-generation) node packages/xrpl/tools/generateModels.js ~/rippled/    
/Users/ckeshavabs/xrpl.js/packages/xrpl/tools/generateModels.js:24
  for (const hit of sfieldHits) {
                    ^

TypeError: sfieldHits is not iterable
    at processRippledSource (/Users/ckeshavabs/xrpl.js/packages/xrpl/tools/generateModels.js:24:21)
    at main (/Users/ckeshavabs/xrpl.js/packages/xrpl/tools/generateModels.js:178:5)
    at Object.<anonymous> (/Users/ckeshavabs/xrpl.js/packages/xrpl/tools/generateModels.js:271:3)
    at Module._compile (node:internal/modules/cjs/loader:1546:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1691:10)
    at Module.load (node:internal/modules/cjs/loader:1317:32)
    at Module._load (node:internal/modules/cjs/loader:1127:12)
    at TracingChannel.traceSync (node:diagnostics_channel:315:14)
    at wrapModuleLoad (node:internal/modules/cjs/loader:217:24)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:166:5)

Node.js v22.9.0

I'm certain that the path/to/rippled is correct (I verified that with a stat command)

Is the version of rippled at that location post-refactor?

@ckeshava
Copy link
Collaborator

I tried to execute this script as :

➜  xrpl.js git:(definitions-generation) node packages/xrpl/tools/generateModels.js ~/rippled/    
/Users/ckeshavabs/xrpl.js/packages/xrpl/tools/generateModels.js:24
  for (const hit of sfieldHits) {
                    ^

TypeError: sfieldHits is not iterable
    at processRippledSource (/Users/ckeshavabs/xrpl.js/packages/xrpl/tools/generateModels.js:24:21)
    at main (/Users/ckeshavabs/xrpl.js/packages/xrpl/tools/generateModels.js:178:5)
    at Object.<anonymous> (/Users/ckeshavabs/xrpl.js/packages/xrpl/tools/generateModels.js:271:3)
    at Module._compile (node:internal/modules/cjs/loader:1546:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1691:10)
    at Module.load (node:internal/modules/cjs/loader:1317:32)
    at Module._load (node:internal/modules/cjs/loader:1127:12)
    at TracingChannel.traceSync (node:diagnostics_channel:315:14)
    at wrapModuleLoad (node:internal/modules/cjs/loader:217:24)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:166:5)

Node.js v22.9.0

I'm certain that the path/to/rippled is correct (I verified that with a stat command)

Is the version of rippled at that location post-refactor?

I'm at the tip of 0ec17b6026298dc150099b66a2cecad0fe561d1b on the develop branch of rippled. I also tested against the tip of this PR: XRPLF/rippled#5161

I think both of them are post-refactor of rippled

@mvadari
Copy link
Collaborator Author

mvadari commented Nov 18, 2024

Is the version of rippled at that location post-refactor?

I'm at the tip of 0ec17b6026298dc150099b66a2cecad0fe561d1b on the develop branch of rippled. I also tested against the tip of this PR: XRPLF/rippled#5161

I think both of them are post-refactor of rippled

Both work fine for me:

mvadari@mvadari-H2R24-MBP xrpl-js % node packages/ripple-binary-codec/tools/generateDefinitions.js path/to/perm_domains 
File written successfully to /Users/mvadari/Documents/xrpl-js/packages/ripple-binary-codec/src/enums/definitions.json
mvadari@mvadari-H2R24-MBP xrpl-js % node packages/ripple-binary-codec/tools/generateDefinitions.js path/to/develop     
File written successfully to /Users/mvadari/Documents/xrpl-js/packages/ripple-binary-codec/src/enums/definitions.json

@mvadari
Copy link
Collaborator Author

mvadari commented Nov 18, 2024

Ahh, I haven't updated the generateModels script in this PR. Let me do that real quick.

@mvadari
Copy link
Collaborator Author

mvadari commented Nov 18, 2024

@ckeshava the model generation script should work now too.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🧹 Outside diff range and nitpick comments (2)
packages/xrpl/tools/generateModels.js (2)

20-21: Add comments to clarify complex regular expressions

The regular expression used to match sfields is quite complex and may be difficult for maintainers to understand. Adding comments explaining the purpose and structure of the regex would improve readability and maintainability.


31-32: Consider documenting complex regular expressions

The regular expression used for matching transaction formats is complex and may not be immediately clear to future developers. Adding comments or documentation explaining the regex pattern would enhance maintainability and ease future modifications.

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL

📥 Commits

Reviewing files that changed from the base of the PR and between 5d7c1a5 and 6267de4.

📒 Files selected for processing (2)
  • packages/ripple-binary-codec/tools/generateDefinitions.js (1 hunks)
  • packages/xrpl/tools/generateModels.js (2 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • packages/ripple-binary-codec/tools/generateDefinitions.js
🔇 Additional comments (1)
packages/xrpl/tools/generateModels.js (1)

28-29: Consistent use of 'folder' parameter

Great job using the folder parameter here to construct the file path. This ensures consistency and enhances the function's flexibility.

@mvadari mvadari mentioned this pull request Nov 18, 2024
9 tasks
@ckeshava
Copy link
Collaborator

The formatting of the existing definitions.json file is different from the output of this script. I don't have a preference for either style.

However, it would be helpful if both server_definitions RPC command and this script are aligned in the output format.

@tequdev
Copy link
Contributor

tequdev commented Jan 16, 2025

This is just an idea, but it would be useful if we could get definitions.json as a artifact from rippled's github actions instead of generating it in this library. (This would mean that each of the other client libraries would no longer need to generate definitions.json in this way.)

@ckeshava
Copy link
Collaborator

This is just an idea, but it would be useful if we could get definitions.json as a artifact from rippled's github actions instead of generating it in this library. (This would mean that each of the other client libraries would no longer need to generate definitions.json in this way.)

Yes, this would solve for the requirements of all the stakeholders. I agree.

@ckeshava
Copy link
Collaborator

The formatting of the existing definitions.json file is different from the output of this script. I don't have a preference for either style.

However, it would be helpful if both server_definitions RPC command and this script are aligned in the output format.

@mvadari how does this sound to you?

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (2)
packages/xrpl/test/integration/requests/serverState.test.ts (1)

119-119: LGTM! Consider documenting the git property.

The addition of 'git' to removeKeys is appropriate for handling the new git information in server responses.

Consider adding a comment in the test file explaining what git information is expected in the server response and why it's excluded from the comparison.

packages/xrpl/test/integration/requests/serverInfo.test.ts (1)

128-128: LGTM! Consider refactoring common removeKeys.

The addition of 'git' to removeKeys maintains consistency with serverState.test.ts.

Since both test files now share many common removeKeys, consider extracting them into a shared constant. This would improve maintainability and ensure consistency across tests. Example:

// In a shared test utilities file
export const COMMON_SERVER_RESPONSE_REMOVE_KEYS = [
  'time',
  'uptime',
  'complete_ledgers',
  // ... other common keys ...
  'git',
] as const;

// In test files
import { COMMON_SERVER_RESPONSE_REMOVE_KEYS } from './testUtils';
// ...
const removeKeys = [...COMMON_SERVER_RESPONSE_REMOVE_KEYS, /* file-specific keys */];
📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between aca417c and c6fd141.

📒 Files selected for processing (2)
  • packages/xrpl/test/integration/requests/serverInfo.test.ts (1 hunks)
  • packages/xrpl/test/integration/requests/serverState.test.ts (1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (7)
  • GitHub Check: snippets (22.x)
  • GitHub Check: integration (22.x)
  • GitHub Check: snippets (20.x)
  • GitHub Check: integration (20.x)
  • GitHub Check: snippets (18.x)
  • GitHub Check: integration (18.x)
  • GitHub Check: browser (18.x)

@mvadari
Copy link
Collaborator Author

mvadari commented Feb 6, 2025

This is just an idea, but it would be useful if we could get definitions.json as a artifact from rippled's github actions instead of generating it in this library. (This would mean that each of the other client libraries would no longer need to generate definitions.json in this way.)

@tequdev In the xrpl-py version I added support for Github branch links, which feels a bit easier to use. I tried adding it here too but it would have required some pretty heavy code refactors to support the async methods needed to fetch data from a URL - happy to pass that WIP commit to anyone who wants to implement that part.

@mvadari
Copy link
Collaborator Author

mvadari commented Feb 6, 2025

The formatting of the existing definitions.json file is different from the output of this script. I don't have a preference for either style.
However, it would be helpful if both server_definitions RPC command and this script are aligned in the output format.

@mvadari how does this sound to you?

Updated the code to do this.

@mvadari mvadari requested a review from khancode February 7, 2025 18:56
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🔭 Outside diff range comments (1)
packages/ripple-binary-codec/src/enums/definitions.json (1)

2553-2571: 🛠️ Refactor suggestion

⚠️ Potential issue

Duplicate AcceptedCredentials Entry Detected:
There are two identical "AcceptedCredentials" entries here—one defined from lines 2553–2562 and an immediately following duplicate from lines 2564–2571. This redundancy could lead to conflicts or unexpected behavior when the definitions are merged. It is recommended to remove the duplicate entry to ensure a single source of truth.

🧹 Nitpick comments (3)
packages/ripple-binary-codec/src/enums/definitions.json (3)

1844-1852: CredentialType Modification:
The CredentialType field now explicitly marks "isVLEncoded": true and sets "nth": 31. Please verify that these changes conform to the intended serialization rules for credential types and align with the latest rippled definitions.


2753-2761: CredentialIDs Field Update Verification:
The "CredentialIDs" field now has "isVLEncoded": true (as shown on the lines with tilde markers). Please confirm that this update is intentional and that it remains consistent with how similar vector types (e.g. for Vector256) are handled throughout the codebase.


1-3193: Overall Consistency and Maintainability Check:
This definitions file is auto-generated by the generateDefinitions.js script. As such, its contents must remain in strict sync with the source definitions from rippled. Please ensure that any modifications (especially manual ones) are minimized and that robust tests or schema validations are in place to verify the integrity of this file with each update.

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 0bc8a10 and 9abd99a.

📒 Files selected for processing (5)
  • .vscode/settings.json (1 hunks)
  • packages/ripple-binary-codec/package.json (1 hunks)
  • packages/ripple-binary-codec/src/enums/definitions.json (17 hunks)
  • packages/xrpl/test/integration/requests/serverInfo.test.ts (1 hunks)
  • packages/xrpl/test/integration/requests/serverState.test.ts (0 hunks)
💤 Files with no reviewable changes (1)
  • packages/xrpl/test/integration/requests/serverState.test.ts
✅ Files skipped from review due to trivial changes (1)
  • .vscode/settings.json
🚧 Files skipped from review as they are similar to previous changes (2)
  • packages/ripple-binary-codec/package.json
  • packages/xrpl/test/integration/requests/serverInfo.test.ts
⏰ Context from checks skipped due to timeout of 90000ms (7)
  • GitHub Check: integration (22.x)
  • GitHub Check: integration (20.x)
  • GitHub Check: integration (18.x)
  • GitHub Check: snippets (22.x)
  • GitHub Check: snippets (20.x)
  • GitHub Check: snippets (18.x)
  • GitHub Check: browser (18.x)
🔇 Additional comments (4)
packages/ripple-binary-codec/src/enums/definitions.json (4)

2888-2890: 'Credential' Ledger Entry Type Update:
The ledger entry type "Credential" is now assigned the value 129. Ensure that this update accurately reflects the latest protocol definitions from rippled and that there are no conflicting or duplicate entries within the LEDGER_ENTRY_TYPES section.


3114-3116: Credential Transaction Types Update:
Within the TRANSACTION_TYPES section, the operations for credentials have been updated to:
"CredentialAccept": 59
"CredentialCreate": 58 (reintroduced)
"CredentialDelete": 60
Please verify that these values and their ordering are correct and that the reintroduction of "CredentialCreate" is deliberate.


3144-3146: PermissionedDomainSet Reintroduction:
The "PermissionedDomainSet" transaction type is reintroduced with the value 62 immediately following "PermissionedDomainDelete": 63. Confirm that this ordering and value assignment are consistent with the updated protocol expectations.


3174-3176: New 'Number' Type Addition:
A new type "Number" has been added to the TYPES section with the value 9. This addition aligns with the updates made in the FIELDS section. Please ensure that all parts of the code generation process and downstream type references correctly incorporate this new type.

@@ -21,7 +21,8 @@
"prepublishOnly": "npm test",
"test": "npm run build && jest --verbose false --silent=false ./test/*.test.ts",
"test:browser": "npm run build && karma start ./karma.config.js",
"lint": "eslint . --ext .ts --ext .test.js"
"lint": "eslint . --ext .ts --ext .test.js",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"lint": "eslint . --ext .ts --ext .test.js",
"lint": "eslint . --ext .ts",

Unrelated to this PR: There are no *.test.js files in this sub-directory tree.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO this should stay in case there are any in the future.

@@ -60,7 +60,6 @@ describe('server_state', function () {
load_factor_fee_queue: 256,
load_factor_fee_reference: 256,
load_factor_server: 256,
network_id: 63456,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I remember deleting these lines from the main branch, I'm surprised you need to do this change again.

@@ -14,32 +14,26 @@ function readFile(filename) {
let jsTransactionFile
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reminder ^

)
const sfieldHits = sfieldCpp.match(
/^ *CONSTRUCT_[^\_]+_SFIELD *\( *[^,\n]*,[ \n]*"([^\"\n ]+)"[ \n]*,[ \n]*([^, \n]+)[ \n]*,[ \n]*([0-9]+)(,.*?(notSigning))?/gm,
const sfieldHits = sfieldMacroFile.matchAll(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
const sfieldHits = sfieldMacroFile.matchAll(
// Typical structure of SField decalraction:
// TYPED_SFIELD(sfTxnSignature, VL, 4, SField::sMD_Default, SField::notSigning)
const sfieldHits = sfieldMacroFile.matchAll(

It is helpful to visualize the intent of this regex

const sfieldHits = sfieldCpp.match(
/^ *CONSTRUCT_[^\_]+_SFIELD *\( *[^,\n]*,[ \n]*"([^\"\n ]+)"[ \n]*,[ \n]*([^, \n]+)[ \n]*,[ \n]*([0-9]+)(,.*?(notSigning))?/gm,
const sfieldHits = sfieldMacroFile.matchAll(
/^ *[A-Z]*TYPED_SFIELD *\( *sf([^,\n]*),[ \n]*([^, \n]+)[ \n]*,[ \n]*([0-9]+)(,.*?(notSigning))?/gm,
)
const sfields = {}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
const sfields = {}
const sfieldNameTypeMap = {}

sfields does not indicate the purpose of this variable

const txFormatsHits = txFormatsCpp.match(
/^ *add\(jss::([^\"\n, ]+),[ \n]*tt[A-Z_]+,[ \n]*{[ \n]*(({sf[A-Za-z0-9]+, soe(OPTIONAL|REQUIRED|DEFAULT)},[ \n]+)*)},[ \n]*[pseudocC]+ommonFields\);/gm,
const txFormatsHits = transactionsMacroFile.matchAll(
/^ *TRANSACTION\(tt[A-Z_]+ *,* [0-9]+ *, *([A-Za-z]+)[ \n]*,[ \n]*\({[ \n]*(({sf[A-Za-z0-9]+, soe(OPTIONAL|REQUIRED|DEFAULT)(, soeMPT(None|Supported|NotSupported))?},[ \n]+)*)}\)\)$/gm,
)
const txFormats = {}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
const txFormats = {}
const txNameFormatMap = {}


// Translate from rippled string format to what the binary codecs expect
function translate(inp) {
try {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the need for this try-catch block? Which parts of this snippet could throw an error? I'd prefer to eliminate it (or) at least reduce the scope of the try-catch block.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It results in a better error message - so if something is wrong, it's easier to debug.

return inp.replace('UINT', 'Hash')
else return inp.replace('UINT', 'UInt')

const nonstandardRenames = {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't find any instances where AMM, DIR_NODE and PAYCHAN are used. The recent commits on rippled:develop do not trigger these keys in the dictionary.

Were they used in historical versions of the rippled codebase?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addLine(' "LEDGER_ENTRY_TYPES": {')

const unhex = (x) => {
x = ('' + x).trim()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Converts the (possible) number to a string

@ckeshava
Copy link
Collaborator

@mvadari I have incorporated fetching rippled's cpp source files from github in this branch. Use this commit

Additionally, I'd like to point out erroneous behavior of this script. It does not generate AcceptedCredentials SField inside definitions.json . I observed this at the latest tip of rippled' develop branch at this commit. You can view the erroneous diff in my custom branch commit. This is not a product of my Github-fetch changes, it is reproducible with the current PR code as well.

@khancode
Copy link
Collaborator

Is it also useful to add a script that gets the definitions using server_definitions RPC? For example, if we wanted definitions on devnet?

@mvadari
Copy link
Collaborator Author

mvadari commented Feb 26, 2025

Is it also useful to add a script that gets the definitions using server_definitions RPC? For example, if we wanted definitions on devnet?

Yes, probably. But I don't have capacity to add that right now, so I would prefer that it happen in a separate PR.

@mvadari
Copy link
Collaborator Author

mvadari commented Feb 26, 2025

Additionally, I'd like to point out erroneous behavior of this script. It does not generate AcceptedCredentials SField inside definitions.json . I observed this at the latest tip of rippled' develop branch at this commit. You can view the erroneous diff in my custom branch commit. This is not a product of my Github-fetch changes, it is reproducible with the current PR code as well.

It's removing a duplicate.

Copy link

New and removed dependencies detected. Learn more about Socket for GitHub ↗︎

Package New capabilities Transitives Size Publisher
npm/[email protected] environment, filesystem, unsafe 0 11.2 MB prettier-bot

View full report↗︎

@mvadari
Copy link
Collaborator Author

mvadari commented Feb 26, 2025

@mvadari I have incorporated fetching rippled's cpp source files from github in this branch. Use this commit

This commit doesn't work for me.

xrpl-js % node packages/ripple-binary-codec/tools/generateDefinitions.js ../rippled-all/develop
(node:86407) Warning: To load an ES module, set "type": "module" in the package.json or use the .mjs extension.
(Use `node --trace-warnings ...` to show where the warning was created)
/Users/mvadari/Documents/xrpl-js/packages/ripple-binary-codec/tools/generateDefinitions.js:1
import { createRequire } from 'module'
^^^^^^

SyntaxError: Cannot use import statement outside a module

Those imports appear to be unneeded, removing them results in this:

mvadari@mvadari-H2R24-MBP xrpl-js % node packages/ripple-binary-codec/tools/generateDefinitions.js ../rippled-all/develop
/Users/mvadari/Documents/xrpl-js/packages/ripple-binary-codec/tools/generateDefinitions.js:45
const sfieldHeaderFile = await readFile(
                         ^^^^^

SyntaxError: await is only valid in async functions and the top level bodies of modules

Copy link
Collaborator

@khancode khancode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have much context on rippled's definitions but AFAICT, it LGTM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants