[Improve](metadata)Start the script to set metadata_failure_recovery #24308

CalvinKirs · 2023-09-13T09:09:54Z

Backend

Many people set meta_recovery in fe.conf, but often forget to change it back after doing it, so fe keeps using it like this, and as a result, problems with the metadata later occur.

We can add this parameter to start_fe.sh --metadata_failure_recovery to do this. In this way, if the user starts fe at the same time, it will automatically fail next time.

Changes

allowed use start_fe.sh -- metadata_failure_recovery or start_fe.sh --daemon -- metadata_failure_recovery
Setting up metadata recovery via configuration files is no longer allowed

Test

2023-09-13 17:30:05,808 INFO (main|1) [BDBEnvironment.setup():101] start group reset
2023-09-13 17:30:06,913 INFO (MASTER fe_38192775_8511_425f_9d31_39aaf9f69841(-1)|1) [BDBEnvironment.setup():105] group has been reset.

github-actions · 2023-09-13T09:10:55Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In bin/start_fe.sh line 229:
    nohup ${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.DorisFE ${HELPER:+${HELPER}}  ${METADATA_FAILURE_RECOVERY} "$@" >>"${LOG_DIR}/fe.out" 2>&1 </dev/null &
                                                                                                                                                                                              ^--------------------------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.

Did you mean: 
    nohup ${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.DorisFE ${HELPER:+${HELPER}}  "${METADATA_FAILURE_RECOVERY}" "$@" >>"${LOG_DIR}/fe.out" 2>&1 </dev/null &


In bin/start_fe.sh line 232:
    ${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.DorisFE ${HELPER:+${HELPER}} ${OPT_VERSION:+${OPT_VERSION}}  ${METADATA_FAILURE_RECOVERY} "$@" </dev/null
                                                                                                                                                                                                                       ^--------------------------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.

Did you mean: 
    ${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.DorisFE ${HELPER:+${HELPER}} ${OPT_VERSION:+${OPT_VERSION}}  "${METADATA_FAILURE_RECOVERY}" "$@" </dev/null

For more information:
  https://www.shellcheck.net/wiki/SC2248 -- Prefer double quoting even when v...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors


'shfmt ' returned error 1 finding the following formatting issues:

----------
--- bin/start_fe.sh.orig
+++ bin/start_fe.sh
@@ -55,7 +55,7 @@
         ;;
     --metadata_failure_recovery)
         METADATA_FAILURE_RECOVERY="-r"
-        shift 
+        shift
         ;;
     --helper)
         HELPER="$2"
@@ -226,10 +226,10 @@
         echo "Internal Error. USE IMAGE_TOOL like : ./start_fe.sh --image image_path"
     fi
 elif [[ "${RUN_DAEMON}" -eq 1 ]]; then
-    nohup ${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.DorisFE ${HELPER:+${HELPER}}  ${METADATA_FAILURE_RECOVERY} "$@" >>"${LOG_DIR}/fe.out" 2>&1 </dev/null &
+    nohup ${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.DorisFE ${HELPER:+${HELPER}} ${METADATA_FAILURE_RECOVERY} "$@" >>"${LOG_DIR}/fe.out" 2>&1 </dev/null &
 else
     export DORIS_LOG_TO_STDERR=1
-    ${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.DorisFE ${HELPER:+${HELPER}} ${OPT_VERSION:+${OPT_VERSION}}  ${METADATA_FAILURE_RECOVERY} "$@" </dev/null
+    ${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.DorisFE ${HELPER:+${HELPER}} ${OPT_VERSION:+${OPT_VERSION}} ${METADATA_FAILURE_RECOVERY} "$@" </dev/null
 fi
 
 echo $! >"${pidfile}"
----------

You can reformat the above files to meet shfmt's requirements by typing:

  shfmt  -w filename

CalvinKirs · 2023-09-13T09:34:46Z

run buildall

github-actions · 2023-09-13T09:35:25Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors

'shellcheck ' found no issues.

shfmt errors


'shfmt ' returned error 1 finding the following formatting issues:

----------
--- bin/start_fe.sh.orig
+++ bin/start_fe.sh
@@ -55,7 +55,7 @@
         ;;
     --metadata_failure_recovery)
         METADATA_FAILURE_RECOVERY="-r"
-        shift 
+        shift
         ;;
     --helper)
         HELPER="$2"
@@ -226,10 +226,10 @@
         echo "Internal Error. USE IMAGE_TOOL like : ./start_fe.sh --image image_path"
     fi
 elif [[ "${RUN_DAEMON}" -eq 1 ]]; then
-    nohup ${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.DorisFE ${HELPER:+${HELPER}}  "${METADATA_FAILURE_RECOVERY}" "$@" >>"${LOG_DIR}/fe.out" 2>&1 </dev/null &
+    nohup ${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.DorisFE ${HELPER:+${HELPER}} "${METADATA_FAILURE_RECOVERY}" "$@" >>"${LOG_DIR}/fe.out" 2>&1 </dev/null &
 else
     export DORIS_LOG_TO_STDERR=1
-    ${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.DorisFE ${HELPER:+${HELPER}} ${OPT_VERSION:+${OPT_VERSION}}  "${METADATA_FAILURE_RECOVERY}" "$@" </dev/null
+    ${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.DorisFE ${HELPER:+${HELPER}} ${OPT_VERSION:+${OPT_VERSION}} "${METADATA_FAILURE_RECOVERY}" "$@" </dev/null
 fi
 
 echo $! >"${pidfile}"
----------

You can reformat the above files to meet shfmt's requirements by typing:

  shfmt  -w filename

yiguolei

LGTM

github-actions · 2023-09-13T09:42:54Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-09-13T09:42:57Z

PR approved by anyone and no changes requested.

CalvinKirs · 2023-09-13T09:57:26Z

run buildall

zy-kkk

LGTM

doris-robot · 2023-09-13T10:07:09Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46 seconds
stream load tsv: 595 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162208254 Bytes

sollhui

LGTM

yiguolei

LGTM

yiguolei · 2023-09-14T01:14:23Z

run clickbench

yiguolei · 2023-09-14T01:15:09Z

run ckbench

doris-robot · 2023-09-14T01:54:43Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.58 seconds
stream load tsv: 605 seconds loaded 74807831229 Bytes, about 117 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162244970 Bytes

…24308)

CalvinKirs force-pushed the metadata-recover branch from c3b603f to 74b92ff Compare September 13, 2023 09:34

yiguolei previously approved these changes Sep 13, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 13, 2023

github-actions bot added the reviewed label Sep 13, 2023

CalvinKirs dismissed yiguolei’s stale review via 808b394 September 13, 2023 09:54

[Improve](metadata)Start the script to set metadata_failure_recovery

c672779

CalvinKirs force-pushed the metadata-recover branch from 808b394 to c672779 Compare September 13, 2023 09:55

zy-kkk approved these changes Sep 13, 2023

View reviewed changes

sollhui approved these changes Sep 13, 2023

View reviewed changes

yiguolei added the dev/2.0.2 label Sep 14, 2023

yiguolei approved these changes Sep 14, 2023

View reviewed changes

yiguolei added the kind/behavior-changed label Sep 14, 2023

yiguolei merged commit 64337a8 into apache:master Sep 14, 2023

CalvinKirs deleted the metadata-recover branch September 14, 2023 02:49

xiaokang added merge_conflict dev/2.0.2-merged and removed dev/2.0.2 labels Sep 14, 2023

xiaokang pushed a commit that referenced this pull request Sep 14, 2023

[Improve](metadata)Start the script to set metadata_failure_recovery (#…

ae9d176

…24308)

xiaokang mentioned this pull request Sep 28, 2023

Release Note 2.0.2 #25011

Open

xiaokang mentioned this pull request Dec 3, 2023

Release Note 2.0.3 #27909

Open

xiaokang mentioned this pull request Dec 28, 2023

[fix](compatibility) add back metadata_failure_recovery config for compatibility #29206

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Improve](metadata)Start the script to set metadata_failure_recovery #24308

[Improve](metadata)Start the script to set metadata_failure_recovery #24308

CalvinKirs commented Sep 13, 2023 •

edited

Loading

github-actions bot commented Sep 13, 2023

CalvinKirs commented Sep 13, 2023

github-actions bot commented Sep 13, 2023

yiguolei left a comment

github-actions bot commented Sep 13, 2023

github-actions bot commented Sep 13, 2023

CalvinKirs commented Sep 13, 2023

zy-kkk left a comment

doris-robot commented Sep 13, 2023

sollhui left a comment

yiguolei left a comment

yiguolei commented Sep 14, 2023

yiguolei commented Sep 14, 2023

doris-robot commented Sep 14, 2023

[Improve](metadata)Start the script to set metadata_failure_recovery #24308

[Improve](metadata)Start the script to set metadata_failure_recovery #24308

Conversation

CalvinKirs commented Sep 13, 2023 • edited Loading

Backend

Changes

Test

github-actions bot commented Sep 13, 2023

sh-checker report

CalvinKirs commented Sep 13, 2023

github-actions bot commented Sep 13, 2023

sh-checker report

yiguolei left a comment

Choose a reason for hiding this comment

github-actions bot commented Sep 13, 2023

github-actions bot commented Sep 13, 2023

CalvinKirs commented Sep 13, 2023

zy-kkk left a comment

Choose a reason for hiding this comment

doris-robot commented Sep 13, 2023

sollhui left a comment

Choose a reason for hiding this comment

yiguolei left a comment

Choose a reason for hiding this comment

yiguolei commented Sep 14, 2023

yiguolei commented Sep 14, 2023

doris-robot commented Sep 14, 2023

CalvinKirs commented Sep 13, 2023 •

edited

Loading

`sh-checker report`

`sh-checker report`