Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

能公开一些公开数据集上的性能测试数据吗? #1

Closed
astor-oss opened this issue Aug 11, 2017 · 2 comments
Closed

能公开一些公开数据集上的性能测试数据吗? #1

astor-oss opened this issue Aug 11, 2017 · 2 comments
Labels
question

Comments

@astor-oss
Copy link

No description provided.

@cyongli
Copy link

cyongli commented Aug 11, 2017

我们正在整理这部分的测试报告，不日就会上传了。

@imay imay added the question label Aug 16, 2017
@Knuth1989
Copy link

请问你们用的哪个公开数据集？

@zepazhao zepazhao mentioned this issue Sep 13, 2017
Closed
@lonly197 lonly197 mentioned this issue Nov 16, 2017
Closed
@worker24h worker24h mentioned this issue Jun 4, 2019
Merged
imay pushed a commit that referenced this issue Jun 4, 2019
@worker24h @imay
fixup leak memory (#1244)
ae75e44 
When I declared that the compilation mode was BUILD_TYPE=LSAN, there was a memory leak after running doris.

be.out:
Direct leak of 32816 byte(s) in 1 object(s) allocated from:
    #0 0x1089666 in __interceptor_malloc ../../../../libsanitizer/lsan/lsan_interceptors.cc:53
    #1 0x7ff459547280 in __alloc_dir (/lib64/libc.so.6+0xc0280)

SUMMARY: LeakSanitizer: 32816 byte(s) leaked in 1 allocation(s).
@wangbo wangbo mentioned this issue Sep 30, 2019
Merged
@wangbo wangbo mentioned this issue Oct 11, 2019
Merged
@wangbo wangbo mentioned this issue Oct 22, 2019
Closed
@zhangy5 zhangy5 mentioned this issue Oct 22, 2019
Closed
@zhangy5 zhangy5 mentioned this issue Nov 6, 2019
Closed
@HangyuanLiu HangyuanLiu mentioned this issue Nov 13, 2019
Closed
@kangpinghuang kangpinghuang mentioned this issue Nov 14, 2019
Closed
@jmk1011 jmk1011 mentioned this issue Dec 23, 2019
Closed
@worker24h worker24h mentioned this issue Feb 17, 2020
Closed
@imay imay closed this as completed Apr 3, 2020
@marising marising mentioned this issue Jun 1, 2020
Closed
@morningman morningman mentioned this issue Jun 23, 2020
Open
morningman referenced this issue in morningman/doris Jul 11, 2020
@hf200012
Merge pull request #1 from morningman/doris-ui
659db56 
[CodeRefactor] Clean up some code
@ZhangYu0123 ZhangYu0123 mentioned this issue Sep 3, 2020
Closed
@caiconghui caiconghui mentioned this issue Mar 3, 2021
Closed
@xinghuayu007 xinghuayu007 mentioned this issue Apr 16, 2021
Open
@xqinghu xqinghu mentioned this issue May 20, 2021
Closed
@hlh5 hlh5 mentioned this issue Dec 2, 2021
Open
3 tasks
@xinyiZzz xinyiZzz mentioned this issue Dec 16, 2021
Closed
3 tasks
@caiconghui caiconghui mentioned this issue Dec 19, 2021
Open
@junjie4585 junjie4585 mentioned this issue Feb 21, 2022
Closed
3 tasks
@larryju larryju mentioned this issue Feb 24, 2022
Open
3 tasks
morrySnow referenced this issue in morrySnow/incubator-doris Mar 16, 2022
@yiguolei
Merge pull request #1 from dataroaring/tpcds_from_trino
af64f72 
add tpcds from trino
mrhhsg added a commit that referenced this issue Jan 12, 2025
@mrhhsg
[fix](pipelien) should not finalize probe when wake up early in SetPr…
ac08f9f 
…obeSinkOperatorX (#46706)

### What problem does this PR solve?

```
*** Query id: 80819fcc223e4a45-b46155de6e0c4eee ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1736352810 (unix time) try "date -d @1736352810" if you are using GNU date ***
*** Current BE git commitID: 08683cb ***
*** SIGSEGV address not mapped to object (@0x38) received by PID 8736 (TID 11549 OR 0x7f8dd0922640) from PID 56; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris_branch-3.0/doris/be/src/common/signal_handler.h:421
 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 3# 0x00007F92019CA520 in /lib/x86_64-linux-gnu/libc.so.6
 4# auto doris::pipeline::SetProbeSinkOperatorX::_refresh_hash_table(doris::pipeline::SetProbeSinkLocalState&)::{lambda(auto:1&&)#1}::operator(), HashTableNoState>, DefaultHash, HashTableGrower<10ul>, Allocator > >&>(doris::vectorized::MethodSerialized, HashTableNoState>, DefaultHash, HashTableGrower<10ul>, Allocator > >&) const at /root/doris_branch-3.0/doris/be/src/pipeline/exec/set_probe_sink_operator.cpp:213
 5# doris::pipeline::SetProbeSinkOperatorX::_finalize_probe(doris::pipeline::SetProbeSinkLocalState&) at /root/doris_branch-3.0/doris/be/src/pipeline/exec/set_probe_sink_operator.cpp:184
 6# doris::pipeline::SetProbeSinkOperatorX::sink(doris::RuntimeState*, doris::vectorized::Block*, bool) at /root/doris_branch-3.0/doris/be/src/pipeline/exec/set_probe_sink_operator.cpp:98
 7# doris::pipeline::PipelineTask::execute(bool*) at /root/doris_branch-3.0/doris/be/src/pipeline/pipeline_task.cpp:387
 8# doris::pipeline::TaskScheduler::_do_work(unsigned long) at /root/doris_branch-3.0/doris/be/src/pipeline/task_scheduler.cpp:138
 9# doris::ThreadPool::dispatch_thread() in /mnt/ssd01/doris-branch40preview/NEREIDS_ASAN/be/lib/doris_be
10# doris::Thread::supervise_thread(void*) at /root/doris_branch-3.0/doris/be/src/util/thread.cpp:499
11# start_thread at ./nptl/pthread_create.c:442
12# 0x00007F9201AAE850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83
```
yiguolei pushed a commit that referenced this issue Jan 13, 2025
@mrhhsg
[fix](pipelien) should not finalize probe when wake up early in SetPr…
6bc2b1e 
…obeSinkOperatorX (#46706) (#46831)

```
*** Query id: 80819fcc223e4a45-b46155de6e0c4eee ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1736352810 (unix time) try "date -d @1736352810" if you are using GNU date ***
*** Current BE git commitID: 08683cb ***
*** SIGSEGV address not mapped to object (@0x38) received by PID 8736 (TID 11549 OR 0x7f8dd0922640) from PID 56; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris_branch-3.0/doris/be/src/common/signal_handler.h:421
 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 3# 0x00007F92019CA520 in /lib/x86_64-linux-gnu/libc.so.6
 4# auto doris::pipeline::SetProbeSinkOperatorX::_refresh_hash_table(doris::pipeline::SetProbeSinkLocalState&)::{lambda(auto:1&&)#1}::operator(), HashTableNoState>, DefaultHash, HashTableGrower<10ul>, Allocator > >&>(doris::vectorized::MethodSerialized, HashTableNoState>, DefaultHash, HashTableGrower<10ul>, Allocator > >&) const at /root/doris_branch-3.0/doris/be/src/pipeline/exec/set_probe_sink_operator.cpp:213
 5# doris::pipeline::SetProbeSinkOperatorX::_finalize_probe(doris::pipeline::SetProbeSinkLocalState&) at /root/doris_branch-3.0/doris/be/src/pipeline/exec/set_probe_sink_operator.cpp:184
 6# doris::pipeline::SetProbeSinkOperatorX::sink(doris::RuntimeState*, doris::vectorized::Block*, bool) at /root/doris_branch-3.0/doris/be/src/pipeline/exec/set_probe_sink_operator.cpp:98
 7# doris::pipeline::PipelineTask::execute(bool*) at /root/doris_branch-3.0/doris/be/src/pipeline/pipeline_task.cpp:387
 8# doris::pipeline::TaskScheduler::_do_work(unsigned long) at /root/doris_branch-3.0/doris/be/src/pipeline/task_scheduler.cpp:138
 9# doris::ThreadPool::dispatch_thread() in /mnt/ssd01/doris-branch40preview/NEREIDS_ASAN/be/lib/doris_be
10# doris::Thread::supervise_thread(void*) at /root/doris_branch-3.0/doris/be/src/util/thread.cpp:499
11# start_thread at ./nptl/pthread_create.c:442
12# 0x00007F9201AAE850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83
```
mrhhsg added a commit to mrhhsg/doris that referenced this issue Jan 13, 2025
@mrhhsg
[fix](pipelien) should not finalize probe when wake up early in SetPr…
c1626a8 
…obeSinkOperatorX (apache#46706)

```
*** Query id: 80819fcc223e4a45-b46155de6e0c4eee ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1736352810 (unix time) try "date -d @1736352810" if you are using GNU date ***
*** Current BE git commitID: 08683cb ***
*** SIGSEGV address not mapped to object (@0x38) received by PID 8736 (TID 11549 OR 0x7f8dd0922640) from PID 56; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris_branch-3.0/doris/be/src/common/signal_handler.h:421
 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 3# 0x00007F92019CA520 in /lib/x86_64-linux-gnu/libc.so.6
 4# auto doris::pipeline::SetProbeSinkOperatorX::_refresh_hash_table(doris::pipeline::SetProbeSinkLocalState&)::{lambda(auto:1&&)apache#1}::operator(), HashTableNoState>, DefaultHash, HashTableGrower<10ul>, Allocator > >&>(doris::vectorized::MethodSerialized, HashTableNoState>, DefaultHash, HashTableGrower<10ul>, Allocator > >&) const at /root/doris_branch-3.0/doris/be/src/pipeline/exec/set_probe_sink_operator.cpp:213
 5# doris::pipeline::SetProbeSinkOperatorX::_finalize_probe(doris::pipeline::SetProbeSinkLocalState&) at /root/doris_branch-3.0/doris/be/src/pipeline/exec/set_probe_sink_operator.cpp:184
 6# doris::pipeline::SetProbeSinkOperatorX::sink(doris::RuntimeState*, doris::vectorized::Block*, bool) at /root/doris_branch-3.0/doris/be/src/pipeline/exec/set_probe_sink_operator.cpp:98
 7# doris::pipeline::PipelineTask::execute(bool*) at /root/doris_branch-3.0/doris/be/src/pipeline/pipeline_task.cpp:387
 8# doris::pipeline::TaskScheduler::_do_work(unsigned long) at /root/doris_branch-3.0/doris/be/src/pipeline/task_scheduler.cpp:138
 9# doris::ThreadPool::dispatch_thread() in /mnt/ssd01/doris-branch40preview/NEREIDS_ASAN/be/lib/doris_be
10# doris::Thread::supervise_thread(void*) at /root/doris_branch-3.0/doris/be/src/util/thread.cpp:499
11# start_thread at ./nptl/pthread_create.c:442
12# 0x00007F9201AAE850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83
```
yiguolei pushed a commit that referenced this issue Jan 13, 2025
@mrhhsg
[fix](pipelien) should not finalize probe when wake up early in SetPr…
fa60c60 
…obeSinkOperatorX (#46706) (#46832)

```
*** Query id: 80819fcc223e4a45-b46155de6e0c4eee ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1736352810 (unix time) try "date -d @1736352810" if you are using GNU date ***
*** Current BE git commitID: 08683cb ***
*** SIGSEGV address not mapped to object (@0x38) received by PID 8736 (TID 11549 OR 0x7f8dd0922640) from PID 56; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris_branch-3.0/doris/be/src/common/signal_handler.h:421
 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 3# 0x00007F92019CA520 in /lib/x86_64-linux-gnu/libc.so.6
 4# auto doris::pipeline::SetProbeSinkOperatorX::_refresh_hash_table(doris::pipeline::SetProbeSinkLocalState&)::{lambda(auto:1&&)#1}::operator(), HashTableNoState>, DefaultHash, HashTableGrower<10ul>, Allocator > >&>(doris::vectorized::MethodSerialized, HashTableNoState>, DefaultHash, HashTableGrower<10ul>, Allocator > >&) const at /root/doris_branch-3.0/doris/be/src/pipeline/exec/set_probe_sink_operator.cpp:213
 5# doris::pipeline::SetProbeSinkOperatorX::_finalize_probe(doris::pipeline::SetProbeSinkLocalState&) at /root/doris_branch-3.0/doris/be/src/pipeline/exec/set_probe_sink_operator.cpp:184
 6# doris::pipeline::SetProbeSinkOperatorX::sink(doris::RuntimeState*, doris::vectorized::Block*, bool) at /root/doris_branch-3.0/doris/be/src/pipeline/exec/set_probe_sink_operator.cpp:98
 7# doris::pipeline::PipelineTask::execute(bool*) at /root/doris_branch-3.0/doris/be/src/pipeline/pipeline_task.cpp:387
 8# doris::pipeline::TaskScheduler::_do_work(unsigned long) at /root/doris_branch-3.0/doris/be/src/pipeline/task_scheduler.cpp:138
 9# doris::ThreadPool::dispatch_thread() in /mnt/ssd01/doris-branch40preview/NEREIDS_ASAN/be/lib/doris_be
10# doris::Thread::supervise_thread(void*) at /root/doris_branch-3.0/doris/be/src/util/thread.cpp:499
11# start_thread at ./nptl/pthread_create.c:442
12# 0x00007F9201AAE850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83
```
@yiguolei yiguolei mentioned this issue Jan 15, 2025
Merged
16 tasks
yiguolei added a commit that referenced this issue Jan 15, 2025
@yiguolei
[bugfix](memtable) arena is freed early and will cause use after free (…
81534dd 
…#46997)

### What problem does this PR solve?

Related PR: #40912

Problem Summary:

Do not reset _arena in MemTable::to_block(), because it is still used in
~MemTable() when releasing agg places

Fix the following use-after-free

Use:

==3628099==ERROR: AddressSanitizer: heap-use-after-free on address
0x52100381be60 at pc 0x5648f30893f8 bp 0x7f8842433310 sp 0x7f8842433308
READ of size 8 at 0x52100381be60 thread T4767 (wg_flush_broker)
#0 0x5648f30893f7 in
phmap::priv::raw_hash_set<phmap::priv::FlatHashSetPolicy<unsigned long>,
phmap::Hash<unsigned long>, phmap::EqualTo<unsigned long>,
std::allocator<unsigned long>>::destroy_slots()
doris/thirdparty/installed/include/parallel_hashmap/phmap.h:1992:14
#1 0x5648f30936f6 in
phmap::priv::raw_hash_set<phmap::priv::FlatHashSetPolicy<unsigned long>,
phmap::Hash<unsigned long>, phmap::EqualTo<unsigned long>,
std::allocator<unsigned long>>::~raw_hash_set()
doris/thirdparty/installed/include/parallel_hashmap/phmap.h:1236:23
#2 0x5648f3089276 in phmap::flat_hash_set<unsigned long,
phmap::Hash<unsigned long>, phmap::EqualTo<unsigned long>,
std::allocator<unsigned long>>::~flat_hash_set()
doris/thirdparty/installed/include/parallel_hashmap/phmap.h:4577:7
#3 0x5648f308922a in doris::BitmapValue::~BitmapValue()
doris/be/src/util/bitmap_value.h:824:7
#4 0x56490d319fa6 in
doris::vectorized::AggregateFunctionBitmapData<doris::vectorized::AggregateFunctionBitmapUnionOp>::~AggregateFunctionBitmapData()
doris/be/src/vec/aggregate_functions/aggregate_function_bitmap.h:127:8
#5 0x56490d49636a in
doris::vectorized::IAggregateFunctionDataHelper<doris::vectorized::AggregateFunctionBitmapData<doris::vectorized::AggregateFunctionBitmapUnionOp>,
doris::vectorized::AggregateFunctionBitmapOp<doris::vectorized::AggregateFunctionBitmapUnionOp>>::destroy(char*)
const doris/be/src/vec/aggregate_functions/aggregate_function.h:563:92
#6 0x5648f68376e9 in doris::MemTable::~MemTable()
doris/be/src/olap/memtable.cpp:159:27
Free:

0x52100381be60 is located 352 bytes inside of 4096-byte region
[0x52100381bd00,0x52100381cd00)
freed by thread T4767 (wg_flush_broker) here:
#0 0x5648f2f3ee46 in free (doris/output/be/lib/doris_be+0x57418e46)
(BuildId: 298b9c91a1ec8fe0)
#1 0x5648f3080dfc in DefaultMemoryAllocator::free(void*)
doris/be/src/vec/common/allocator.h:108:41
#2 0x5648f3080b3f in Allocator<false, false, false,
DefaultMemoryAllocator>::free(void*, unsigned long)
doris/be/src/vec/common/allocator.h:323:13
#3 0x5648f30b6dee in doris::vectorized::Arena::Chunk::~Chunk()
doris/be/src/vec/common/arena.h:77:31
#4 0x5648f30b6d1f in doris::vectorized::Arena::~Arena()
doris/be/src/vec/common/arena.h:151:16
#5 0x5648f30b695a in
std::default_delete<doris::vectorized::Arena>::operator()(doris::vectorized::Arena*)
const
env/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/unique_ptr.h:99:2
#6 0x5648f30b67c8 in std::__uniq_ptr_impl<doris::vectorized::Arena,
std::default_delete<doris::vectorized::Arena>>::reset(doris::vectorized::Arena*)
env/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/unique_ptr.h:211:4
#7 0x5648f30b5d8c in std::unique_ptr<doris::vectorized::Arena,
std::default_delete<doris::vectorized::Arena>>::reset(doris::vectorized::Arena*)
env/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/unique_ptr.h:509:7
#8 0x5648f684253b in
doris::MemTable::_to_block(std::unique_ptr<doris::vectorized::Block,
std::default_delete<doris::vectorized::Block>>*)
doris/be/src/olap/memtable.cpp:522:12
#9 0x5648f6842ac5 in
doris::MemTable::to_block(std::unique_ptr<doris::vectorized::Block,
std::default_delete<doris::vectorized::Block>>*)
doris/be/src/olap/memtable.cpp:528:5
#10 0x5648f6907a72 in
doris::FlushToken::_do_flush_memtable(doris::MemTable*, int, long*)
doris/be/src/olap/memtable_flush_executor.cpp:144:9
#11 0x5648f690932c in
doris::FlushToken::_flush_memtable(std::shared_ptr<doris::MemTable>,
int, long) doris/be/src/olap/memtable_flush_executor.cpp:183:16
#12 0x5648f6915d18 in doris::MemtableFlushTask::run()
doris/be/src/olap/memtable_flush_executor.cpp:60:20
github-actions bot pushed a commit that referenced this issue Jan 15, 2025
@yiguolei
[bugfix](memtable) arena is freed early and will cause use after free (…
8aa5d66 
…#46997)

### What problem does this PR solve?

Related PR: #40912

Problem Summary:

Do not reset _arena in MemTable::to_block(), because it is still used in
~MemTable() when releasing agg places

Fix the following use-after-free

Use:

==3628099==ERROR: AddressSanitizer: heap-use-after-free on address
0x52100381be60 at pc 0x5648f30893f8 bp 0x7f8842433310 sp 0x7f8842433308
READ of size 8 at 0x52100381be60 thread T4767 (wg_flush_broker)
#0 0x5648f30893f7 in
phmap::priv::raw_hash_set<phmap::priv::FlatHashSetPolicy<unsigned long>,
phmap::Hash<unsigned long>, phmap::EqualTo<unsigned long>,
std::allocator<unsigned long>>::destroy_slots()
doris/thirdparty/installed/include/parallel_hashmap/phmap.h:1992:14
#1 0x5648f30936f6 in
phmap::priv::raw_hash_set<phmap::priv::FlatHashSetPolicy<unsigned long>,
phmap::Hash<unsigned long>, phmap::EqualTo<unsigned long>,
std::allocator<unsigned long>>::~raw_hash_set()
doris/thirdparty/installed/include/parallel_hashmap/phmap.h:1236:23
#2 0x5648f3089276 in phmap::flat_hash_set<unsigned long,
phmap::Hash<unsigned long>, phmap::EqualTo<unsigned long>,
std::allocator<unsigned long>>::~flat_hash_set()
doris/thirdparty/installed/include/parallel_hashmap/phmap.h:4577:7
#3 0x5648f308922a in doris::BitmapValue::~BitmapValue()
doris/be/src/util/bitmap_value.h:824:7
#4 0x56490d319fa6 in
doris::vectorized::AggregateFunctionBitmapData<doris::vectorized::AggregateFunctionBitmapUnionOp>::~AggregateFunctionBitmapData()
doris/be/src/vec/aggregate_functions/aggregate_function_bitmap.h:127:8
#5 0x56490d49636a in
doris::vectorized::IAggregateFunctionDataHelper<doris::vectorized::AggregateFunctionBitmapData<doris::vectorized::AggregateFunctionBitmapUnionOp>,
doris::vectorized::AggregateFunctionBitmapOp<doris::vectorized::AggregateFunctionBitmapUnionOp>>::destroy(char*)
const doris/be/src/vec/aggregate_functions/aggregate_function.h:563:92
#6 0x5648f68376e9 in doris::MemTable::~MemTable()
doris/be/src/olap/memtable.cpp:159:27
Free:

0x52100381be60 is located 352 bytes inside of 4096-byte region
[0x52100381bd00,0x52100381cd00)
freed by thread T4767 (wg_flush_broker) here:
#0 0x5648f2f3ee46 in free (doris/output/be/lib/doris_be+0x57418e46)
(BuildId: 298b9c91a1ec8fe0)
#1 0x5648f3080dfc in DefaultMemoryAllocator::free(void*)
doris/be/src/vec/common/allocator.h:108:41
#2 0x5648f3080b3f in Allocator<false, false, false,
DefaultMemoryAllocator>::free(void*, unsigned long)
doris/be/src/vec/common/allocator.h:323:13
#3 0x5648f30b6dee in doris::vectorized::Arena::Chunk::~Chunk()
doris/be/src/vec/common/arena.h:77:31
#4 0x5648f30b6d1f in doris::vectorized::Arena::~Arena()
doris/be/src/vec/common/arena.h:151:16
#5 0x5648f30b695a in
std::default_delete<doris::vectorized::Arena>::operator()(doris::vectorized::Arena*)
const
env/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/unique_ptr.h:99:2
#6 0x5648f30b67c8 in std::__uniq_ptr_impl<doris::vectorized::Arena,
std::default_delete<doris::vectorized::Arena>>::reset(doris::vectorized::Arena*)
env/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/unique_ptr.h:211:4
#7 0x5648f30b5d8c in std::unique_ptr<doris::vectorized::Arena,
std::default_delete<doris::vectorized::Arena>>::reset(doris::vectorized::Arena*)
env/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/unique_ptr.h:509:7
#8 0x5648f684253b in
doris::MemTable::_to_block(std::unique_ptr<doris::vectorized::Block,
std::default_delete<doris::vectorized::Block>>*)
doris/be/src/olap/memtable.cpp:522:12
#9 0x5648f6842ac5 in
doris::MemTable::to_block(std::unique_ptr<doris::vectorized::Block,
std::default_delete<doris::vectorized::Block>>*)
doris/be/src/olap/memtable.cpp:528:5
#10 0x5648f6907a72 in
doris::FlushToken::_do_flush_memtable(doris::MemTable*, int, long*)
doris/be/src/olap/memtable_flush_executor.cpp:144:9
#11 0x5648f690932c in
doris::FlushToken::_flush_memtable(std::shared_ptr<doris::MemTable>,
int, long) doris/be/src/olap/memtable_flush_executor.cpp:183:16
#12 0x5648f6915d18 in doris::MemtableFlushTask::run()
doris/be/src/olap/memtable_flush_executor.cpp:60:20
@Gabriel39 Gabriel39 mentioned this issue Feb 11, 2025
Merged
16 tasks
Gabriel39 added a commit that referenced this issue Feb 13, 2025
@Gabriel39
[fix](scanner) fix shared rowset reader in different scanners (#47744)
b79c5b2 
*** Query id: b7482b4d3ab94bcf-acdb53e8a8fcff81 ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1739024048 (unix time) try "date -d @1739024048" if you
are using GNU date ***
*** Current BE git commitID: a733a56 ***
*** SIGSEGV address not mapped to object (@0x0) received by PID 5943
(TID 7905 OR 0x7fd0b0c80640) from PID 0; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int,
siginfo_t*, void*) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/common/signal_handler.h:421
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0]
in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
2# JVM_handle_linux_signal in
/usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 3# 0x00007FD4985D8520 in /lib/x86_64-linux-gnu/libc.so.6
4# doris::segment_v2::FileColumnIterator::next_batch(unsigned long*,
COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&,
bool*) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/column_reader.cpp:1271
5#
doris::segment_v2::SegmentIterator::_read_columns(std::vector<unsigned
int, std::allocator<unsigned int> > const&,
std::vector<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>,
std::allocator<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>
> >&, unsigned long) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1575
6# doris::segment_v2::SegmentIterator::_seek_and_peek(unsigned int) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1253
7#
doris::segment_v2::SegmentIterator::_lookup_ordinal_from_sk_index(doris::RowCursor
const&, bool, unsigned int, unsigned int*) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1148
8# doris::segment_v2::SegmentIterator::_get_row_ranges_by_keys() at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:432
9# doris::segment_v2::SegmentIterator::_lazy_init() at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:382
10#
doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*)
at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2012
11#
doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)
at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1920
12#
doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::vectorized::Block*)
at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.h:45
13#
doris::vectorized::VUnionIterator::next_batch(doris::vectorized::Block*)
at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/vgeneric_iterators.cpp:405
14# doris::BetaRowsetReader::next_block(doris::vectorized::Block*) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/beta_rowset_reader.cpp:376
15#
doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row()
at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/vcollect_iterator.cpp:509
16# doris::vectorized::VCollectIterator::Level0Iterator::init(bool) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/vcollect_iterator.cpp:462
17#
doris::vectorized::VCollectIterator::build_heap(std::vector<std::shared_ptr<doris::RowsetReader>,
std::allocator<std::shared_ptr<doris::RowsetReader> > >&) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/vcollect_iterator.cpp:126
18#
doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams
const&) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/block_reader.cpp:148
19#
doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams
const&) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/block_reader.cpp:222
20# doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:232
21#
doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>,
std::shared_ptr<doris::vectorized::ScanTask>) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:221
22# std::_Function_handler<void (),
doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>,
std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()()
const::{lambda()#1}>::_M_invoke(std::_Any_data const&) at
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
23# doris::ThreadPool::dispatch_thread() in
/mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be
24# doris::Thread::supervise_thread(void*) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/util/thread.cpp:499
25# start_thread at ./nptl/pthread_create.c:442
26# 0x00007FD4986BC850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this issue Feb 13, 2025
@Gabriel39
[fix](scanner) fix shared rowset reader in different scanners (apache…
a313424 
…#47744)

*** Query id: b7482b4d3ab94bcf-acdb53e8a8fcff81 ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1739024048 (unix time) try "date -d @1739024048" if you
are using GNU date ***
*** Current BE git commitID: a733a56 ***
*** SIGSEGV address not mapped to object (@0x0) received by PID 5943
(TID 7905 OR 0x7fd0b0c80640) from PID 0; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int,
siginfo_t*, void*) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/common/signal_handler.h:421
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0]
in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
2# JVM_handle_linux_signal in
/usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 3# 0x00007FD4985D8520 in /lib/x86_64-linux-gnu/libc.so.6
4# doris::segment_v2::FileColumnIterator::next_batch(unsigned long*,
COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&,
bool*) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/column_reader.cpp:1271
5#
doris::segment_v2::SegmentIterator::_read_columns(std::vector<unsigned
int, std::allocator<unsigned int> > const&,
std::vector<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>,
std::allocator<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>
> >&, unsigned long) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1575
6# doris::segment_v2::SegmentIterator::_seek_and_peek(unsigned int) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1253
7#
doris::segment_v2::SegmentIterator::_lookup_ordinal_from_sk_index(doris::RowCursor
const&, bool, unsigned int, unsigned int*) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1148
8# doris::segment_v2::SegmentIterator::_get_row_ranges_by_keys() at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:432
9# doris::segment_v2::SegmentIterator::_lazy_init() at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:382
10#
doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*)
at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2012
11#
doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)
at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1920
12#
doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::vectorized::Block*)
at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.h:45
13#
doris::vectorized::VUnionIterator::next_batch(doris::vectorized::Block*)
at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/vgeneric_iterators.cpp:405
14# doris::BetaRowsetReader::next_block(doris::vectorized::Block*) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/beta_rowset_reader.cpp:376
15#
doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row()
at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/vcollect_iterator.cpp:509
16# doris::vectorized::VCollectIterator::Level0Iterator::init(bool) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/vcollect_iterator.cpp:462
17#
doris::vectorized::VCollectIterator::build_heap(std::vector<std::shared_ptr<doris::RowsetReader>,
std::allocator<std::shared_ptr<doris::RowsetReader> > >&) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/vcollect_iterator.cpp:126
18#
doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams
const&) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/block_reader.cpp:148
19#
doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams
const&) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/block_reader.cpp:222
20# doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:232
21#
doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>,
std::shared_ptr<doris::vectorized::ScanTask>) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:221
22# std::_Function_handler<void (),
doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>,
std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()()
const::{lambda()apache#1}>::_M_invoke(std::_Any_data const&) at
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
23# doris::ThreadPool::dispatch_thread() in
/mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be
24# doris::Thread::supervise_thread(void*) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/util/thread.cpp:499
25# start_thread at ./nptl/pthread_create.c:442
26# 0x00007FD4986BC850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83
BiteTheDDDDt pushed a commit that referenced this issue Feb 14, 2025
@Mryange
[fix](beut) fix core in exchange sink test (#47926)
68662f9 
### What problem does this PR solve?

In BE_TEST, a query context is not initialized, resulting in a core
dump.
```
==1732796==ERROR: AddressSanitizer: SEGV on unknown address 0x0000000001e0 (pc 0x55c664a74e74 bp 0x7ffcc76e9d20 sp 0x7ffcc76e9cf0 T0)
==1732796==The signal is caused by a READ memory access.
==1732796==Hint: address points to the zero page.
    #0 0x55c664a74e74 in std::__shared_ptr<doris::ResourceContext, (__gnu_cxx::_Lock_policy)2>::__shared_ptr(std::__shared_ptr<doris::ResourceContext, (__gnu_cxx::_Lock_policy)2> const&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/shared_ptr_base.h:1522:7
    #1 0x55c664a74e0e in std::shared_ptr<doris::ResourceContext>::shared_ptr(std::shared_ptr<doris::ResourceContext> const&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/shared_ptr.h:204:7
    #2 0x55c66878ff18 in doris::QueryContext::resource_ctx() /mnt/disk12/yanxuecheng/doris/be/src/runtime/query_context.h:292:62
    #3 0x55c69a2f2c19 in doris::vectorized::BlockSerializer::serialize_block(doris::vectorized::Block const*,
```

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [x] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [x] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [x] Confirm the release note
- [x] Confirm test cases
- [x] Confirm document
- [x] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
xinyiZzz added a commit that referenced this issue Feb 19, 2025
@xinyiZzz
[fix](resource) Fix register ResourceContext to RuntimeQueryStatistic…
e7c966c 
…sMgr (#48004)

### What problem does this PR solve?

Wait for QueryTaskController to be initialized before register
ResourceContext

fix
```
/root/doris/be/src/runtime/workload_management/task_controller.h:59:44: runtime error: load of value 3200171710, which is not a valid value for type 'TQueryType::type'
    #0 0x555ff9336162 in doris::TaskController::query_type() /root/doris/be/src/runtime/workload_management/task_controller.h:59:44
    #1 0x555ff9336162 in doris::RuntimeQueryStatisticsMgr::report_runtime_query_statistics() /root/doris/be/src/runtime/runtime_query_statistics_mgr.cpp:346:50
    #2 0x555ff50d789d in doris::Daemon::report_runtime_query_statistics_thread() /root/doris/be/src/common/daemon.cpp:487:65
    #3 0x555ff9a3efae in std::function<void ()>::operator()() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
    #4 0x555ff9a3efae in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5
    #5 0x7f4935d00608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
    #6 0x7f4935fad132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95
```
@Gabriel39 Gabriel39 mentioned this issue Feb 19, 2025
Closed
16 tasks
@HappenLee HappenLee mentioned this issue Feb 20, 2025
Merged
16 tasks
@Gabriel39 Gabriel39 mentioned this issue Feb 21, 2025
Merged
16 tasks
Gabriel39 added a commit that referenced this issue Feb 21, 2025
@Gabriel39
[fix](scheduler) Fix invalid access after freed (#48168)
b7c973e 
WRITE of size 1 at 0x6160007e86f0 thread T1983 (Pipe_normal [wo)
#0 0x55fc8065b975 in std::__atomic_base<bool>::store(bool,
std::memory_order)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:457:2
#1 0x55fc8065b975 in std::__atomic_base<bool>::operator=(bool)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:349:2
#2 0x55fc8065b975 in std::atomic<bool>::operator=(bool)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/atomic:80:22
#3 0x55fc8065b975 in doris::pipeline::PipelineTask::set_running(bool)
/root/doris/be/src/pipeline/pipeline_task.h:192:47
#4 0x55fc8065b975 in
doris::pipeline::TaskScheduler::_do_work(int)::$_0::operator()() const
/root/doris/be/src/pipeline/task_scheduler.cpp:121:23
#5 0x55fc8065b975 in
doris::Defer<doris::pipeline::TaskScheduler::_do_work(int)::$_0>::~Defer()
/root/doris/be/src/util/defer_op.h:37:16
#6 0x55fc8065b975 in doris::pipeline::TaskScheduler::_do_work(int)
/root/doris/be/src/pipeline/task_scheduler.cpp:162:5
#7 0x55fc4c57cd19 in doris::ThreadPool::dispatch_thread()
/root/doris/be/src/util/threadpool.cpp:608:24
#8 0x55fc4c55395e in std::function<void ()>::operator()() const
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
#9 0x55fc4c55395e in doris::Thread::supervise_thread(void*)
/root/doris/be/src/util/thread.cpp:498:5
#10 0x7f9ee3d25608 in start_thread
/build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
#11 0x7f9ee3fd2132 in __clone
/build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95

0x6160007e86f0 is located 624 bytes inside of 632-byte region
[0x6160007e8480,0x6160007e86f8)
freed by thread T1981 (Pipe_normal [wo) here:
#0 0x55fc47aa680d in operator delete(void*)
(/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x3376e80d)
(BuildId: 865149e62959581e)
#1 0x55fc8059db84 in
std::default_delete<doris::pipeline::PipelineTask>::operator()(doris::pipeline::PipelineTask*)
const
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2
#2 0x55fc8059db84 in std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >::~unique_ptr()
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361:4
#3 0x55fc8059db84 in void
std::destroy_at<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >
>(std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >*)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15
#4 0x55fc8059db84 in void
std::_Destroy<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >
>(std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >*)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:138:7
#5 0x55fc8059db84 in void
std::_Destroy_aux<false>::__destroy<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask>
>*>(std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >*,
std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >*)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:152:6
#6 0x55fc8059db84 in void
std::_Destroy<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask>
>*>(std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >*,
std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >*)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:184:7
#7 0x55fc8059db84 in void
std::_Destroy<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >*,
std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >
>(std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >*,
std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >*,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > >&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:746:7
#8 0x55fc8059db84 in
std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >::~vector()
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:680:2
#9 0x55fc8052571c in void
std::destroy_at<std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >
>(std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >*)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15
#10 0x55fc8052571c in void
std::_Destroy<std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >
>(std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >*)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:138:7
#11 0x55fc8052571c in void
std::_Destroy_aux<false>::__destroy<std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > >
>*>(std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >*,
std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >*)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:152:6
#12 0x55fc8052571c in void
std::_Destroy<std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > >
>*>(std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >*,
std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >*)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:184:7
#13 0x55fc8052571c in void
std::_Destroy<std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >*,
std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >
>(std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >*,
std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >*,
std::allocator<std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > > >&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:746:7
#14 0x55fc8052571c in
std::vector<std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >,
std::allocator<std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > > >
>::_M_erase_at_end(std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >*)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1796:6
#15 0x55fc8052571c in
std::vector<std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >,
std::allocator<std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > > > >::clear()
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1499:9
#16 0x55fc8052571c in
doris::pipeline::PipelineFragmentContext::~PipelineFragmentContext()
/root/doris/be/src/pipeline/pipeline_fragment_context.cpp:142:12
#17 0x55fc47ad30cc in
std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:168:6
#18 0x55fc80658d57 in
std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count()
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:702:11
#19 0x55fc80658d57 in std::__shared_ptr<doris::TaskExecutionContext,
(__gnu_cxx::_Lock_policy)2>::~__shared_ptr()
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1149:31
#20 0x55fc80658d57 in
doris::pipeline::close_task(doris::pipeline::PipelineTask*,
doris::Status) /root/doris/be/src/pipeline/task_scheduler.cpp:100:1
#21 0x55fc8065aa17 in doris::pipeline::TaskScheduler::_do_work(int)
/root/doris/be/src/pipeline/task_scheduler.cpp:160:36
#22 0x55fc4c57cd19 in doris::ThreadPool::dispatch_thread()
/root/doris/be/src/util/threadpool.cpp:608:24
#23 0x55fc4c55395e in std::function<void ()>::operator()() const
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
#24 0x55fc4c55395e in doris::Thread::supervise_thread(void*)
/root/doris/be/src/util/thread.cpp:498:5
#25 0x7f9ee3d25608 in start_thread
/build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
lzyy2024 pushed a commit to lzyy2024/doris that referenced this issue Feb 21, 2025
@yiguolei @lzyy2024
[bugfix](memtable) arena is freed early and will cause use after free (…
193cecd 
…apache#46997)

### What problem does this PR solve?

Related PR: apache#40912

Problem Summary:

Do not reset _arena in MemTable::to_block(), because it is still used in
~MemTable() when releasing agg places

Fix the following use-after-free

Use:

==3628099==ERROR: AddressSanitizer: heap-use-after-free on address
0x52100381be60 at pc 0x5648f30893f8 bp 0x7f8842433310 sp 0x7f8842433308
READ of size 8 at 0x52100381be60 thread T4767 (wg_flush_broker)
#0 0x5648f30893f7 in
phmap::priv::raw_hash_set<phmap::priv::FlatHashSetPolicy<unsigned long>,
phmap::Hash<unsigned long>, phmap::EqualTo<unsigned long>,
std::allocator<unsigned long>>::destroy_slots()
doris/thirdparty/installed/include/parallel_hashmap/phmap.h:1992:14
apache#1 0x5648f30936f6 in
phmap::priv::raw_hash_set<phmap::priv::FlatHashSetPolicy<unsigned long>,
phmap::Hash<unsigned long>, phmap::EqualTo<unsigned long>,
std::allocator<unsigned long>>::~raw_hash_set()
doris/thirdparty/installed/include/parallel_hashmap/phmap.h:1236:23
apache#2 0x5648f3089276 in phmap::flat_hash_set<unsigned long,
phmap::Hash<unsigned long>, phmap::EqualTo<unsigned long>,
std::allocator<unsigned long>>::~flat_hash_set()
doris/thirdparty/installed/include/parallel_hashmap/phmap.h:4577:7
apache#3 0x5648f308922a in doris::BitmapValue::~BitmapValue()
doris/be/src/util/bitmap_value.h:824:7
apache#4 0x56490d319fa6 in
doris::vectorized::AggregateFunctionBitmapData<doris::vectorized::AggregateFunctionBitmapUnionOp>::~AggregateFunctionBitmapData()
doris/be/src/vec/aggregate_functions/aggregate_function_bitmap.h:127:8
apache#5 0x56490d49636a in
doris::vectorized::IAggregateFunctionDataHelper<doris::vectorized::AggregateFunctionBitmapData<doris::vectorized::AggregateFunctionBitmapUnionOp>,
doris::vectorized::AggregateFunctionBitmapOp<doris::vectorized::AggregateFunctionBitmapUnionOp>>::destroy(char*)
const doris/be/src/vec/aggregate_functions/aggregate_function.h:563:92
apache#6 0x5648f68376e9 in doris::MemTable::~MemTable()
doris/be/src/olap/memtable.cpp:159:27
Free:

0x52100381be60 is located 352 bytes inside of 4096-byte region
[0x52100381bd00,0x52100381cd00)
freed by thread T4767 (wg_flush_broker) here:
#0 0x5648f2f3ee46 in free (doris/output/be/lib/doris_be+0x57418e46)
(BuildId: 298b9c91a1ec8fe0)
apache#1 0x5648f3080dfc in DefaultMemoryAllocator::free(void*)
doris/be/src/vec/common/allocator.h:108:41
apache#2 0x5648f3080b3f in Allocator<false, false, false,
DefaultMemoryAllocator>::free(void*, unsigned long)
doris/be/src/vec/common/allocator.h:323:13
apache#3 0x5648f30b6dee in doris::vectorized::Arena::Chunk::~Chunk()
doris/be/src/vec/common/arena.h:77:31
apache#4 0x5648f30b6d1f in doris::vectorized::Arena::~Arena()
doris/be/src/vec/common/arena.h:151:16
apache#5 0x5648f30b695a in
std::default_delete<doris::vectorized::Arena>::operator()(doris::vectorized::Arena*)
const
env/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/unique_ptr.h:99:2
apache#6 0x5648f30b67c8 in std::__uniq_ptr_impl<doris::vectorized::Arena,
std::default_delete<doris::vectorized::Arena>>::reset(doris::vectorized::Arena*)
env/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/unique_ptr.h:211:4
apache#7 0x5648f30b5d8c in std::unique_ptr<doris::vectorized::Arena,
std::default_delete<doris::vectorized::Arena>>::reset(doris::vectorized::Arena*)
env/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/unique_ptr.h:509:7
apache#8 0x5648f684253b in
doris::MemTable::_to_block(std::unique_ptr<doris::vectorized::Block,
std::default_delete<doris::vectorized::Block>>*)
doris/be/src/olap/memtable.cpp:522:12
apache#9 0x5648f6842ac5 in
doris::MemTable::to_block(std::unique_ptr<doris::vectorized::Block,
std::default_delete<doris::vectorized::Block>>*)
doris/be/src/olap/memtable.cpp:528:5
apache#10 0x5648f6907a72 in
doris::FlushToken::_do_flush_memtable(doris::MemTable*, int, long*)
doris/be/src/olap/memtable_flush_executor.cpp:144:9
apache#11 0x5648f690932c in
doris::FlushToken::_flush_memtable(std::shared_ptr<doris::MemTable>,
int, long) doris/be/src/olap/memtable_flush_executor.cpp:183:16
apache#12 0x5648f6915d18 in doris::MemtableFlushTask::run()
doris/be/src/olap/memtable_flush_executor.cpp:60:20
lzyy2024 pushed a commit to lzyy2024/doris that referenced this issue Feb 21, 2025
@Gabriel39 @lzyy2024
[fix](scanner) fix shared rowset reader in different scanners (apache…
8491745 
…#47744)

*** Query id: b7482b4d3ab94bcf-acdb53e8a8fcff81 ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1739024048 (unix time) try "date -d @1739024048" if you
are using GNU date ***
*** Current BE git commitID: a733a56 ***
*** SIGSEGV address not mapped to object (@0x0) received by PID 5943
(TID 7905 OR 0x7fd0b0c80640) from PID 0; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int,
siginfo_t*, void*) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/common/signal_handler.h:421
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0]
in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
2# JVM_handle_linux_signal in
/usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 3# 0x00007FD4985D8520 in /lib/x86_64-linux-gnu/libc.so.6
4# doris::segment_v2::FileColumnIterator::next_batch(unsigned long*,
COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&,
bool*) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/column_reader.cpp:1271
5#
doris::segment_v2::SegmentIterator::_read_columns(std::vector<unsigned
int, std::allocator<unsigned int> > const&,
std::vector<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>,
std::allocator<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>
> >&, unsigned long) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1575
6# doris::segment_v2::SegmentIterator::_seek_and_peek(unsigned int) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1253
7#
doris::segment_v2::SegmentIterator::_lookup_ordinal_from_sk_index(doris::RowCursor
const&, bool, unsigned int, unsigned int*) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1148
8# doris::segment_v2::SegmentIterator::_get_row_ranges_by_keys() at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:432
9# doris::segment_v2::SegmentIterator::_lazy_init() at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:382
10#
doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*)
at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2012
11#
doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)
at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1920
12#
doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::vectorized::Block*)
at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.h:45
13#
doris::vectorized::VUnionIterator::next_batch(doris::vectorized::Block*)
at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/vgeneric_iterators.cpp:405
14# doris::BetaRowsetReader::next_block(doris::vectorized::Block*) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/beta_rowset_reader.cpp:376
15#
doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row()
at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/vcollect_iterator.cpp:509
16# doris::vectorized::VCollectIterator::Level0Iterator::init(bool) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/vcollect_iterator.cpp:462
17#
doris::vectorized::VCollectIterator::build_heap(std::vector<std::shared_ptr<doris::RowsetReader>,
std::allocator<std::shared_ptr<doris::RowsetReader> > >&) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/vcollect_iterator.cpp:126
18#
doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams
const&) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/block_reader.cpp:148
19#
doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams
const&) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/block_reader.cpp:222
20# doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:232
21#
doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>,
std::shared_ptr<doris::vectorized::ScanTask>) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:221
22# std::_Function_handler<void (),
doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>,
std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()()
const::{lambda()apache#1}>::_M_invoke(std::_Any_data const&) at
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
23# doris::ThreadPool::dispatch_thread() in
/mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be
24# doris::Thread::supervise_thread(void*) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/util/thread.cpp:499
25# start_thread at ./nptl/pthread_create.c:442
26# 0x00007FD4986BC850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83
lzyy2024 pushed a commit to lzyy2024/doris that referenced this issue Feb 21, 2025
@Mryange @lzyy2024
[fix](beut) fix core in exchange sink test (apache#47926)
2ceaac6 
### What problem does this PR solve?

In BE_TEST, a query context is not initialized, resulting in a core
dump.
```
==1732796==ERROR: AddressSanitizer: SEGV on unknown address 0x0000000001e0 (pc 0x55c664a74e74 bp 0x7ffcc76e9d20 sp 0x7ffcc76e9cf0 T0)
==1732796==The signal is caused by a READ memory access.
==1732796==Hint: address points to the zero page.
    #0 0x55c664a74e74 in std::__shared_ptr<doris::ResourceContext, (__gnu_cxx::_Lock_policy)2>::__shared_ptr(std::__shared_ptr<doris::ResourceContext, (__gnu_cxx::_Lock_policy)2> const&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/shared_ptr_base.h:1522:7
    apache#1 0x55c664a74e0e in std::shared_ptr<doris::ResourceContext>::shared_ptr(std::shared_ptr<doris::ResourceContext> const&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/shared_ptr.h:204:7
    apache#2 0x55c66878ff18 in doris::QueryContext::resource_ctx() /mnt/disk12/yanxuecheng/doris/be/src/runtime/query_context.h:292:62
    apache#3 0x55c69a2f2c19 in doris::vectorized::BlockSerializer::serialize_block(doris::vectorized::Block const*,
```

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [x] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [x] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [x] Confirm the release note
- [x] Confirm test cases
- [x] Confirm document
- [x] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
lzyy2024 pushed a commit to lzyy2024/doris that referenced this issue Feb 21, 2025
@xinyiZzz @lzyy2024
[fix](resource) Fix register ResourceContext to RuntimeQueryStatistic…
2265362 
…sMgr (apache#48004)

### What problem does this PR solve?

Wait for QueryTaskController to be initialized before register
ResourceContext

fix
```
/root/doris/be/src/runtime/workload_management/task_controller.h:59:44: runtime error: load of value 3200171710, which is not a valid value for type 'TQueryType::type'
    #0 0x555ff9336162 in doris::TaskController::query_type() /root/doris/be/src/runtime/workload_management/task_controller.h:59:44
    apache#1 0x555ff9336162 in doris::RuntimeQueryStatisticsMgr::report_runtime_query_statistics() /root/doris/be/src/runtime/runtime_query_statistics_mgr.cpp:346:50
    apache#2 0x555ff50d789d in doris::Daemon::report_runtime_query_statistics_thread() /root/doris/be/src/common/daemon.cpp:487:65
    apache#3 0x555ff9a3efae in std::function<void ()>::operator()() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
    apache#4 0x555ff9a3efae in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5
    apache#5 0x7f4935d00608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
    apache#6 0x7f4935fad132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95
```
lzyy2024 pushed a commit to lzyy2024/doris that referenced this issue Feb 21, 2025
@Gabriel39 @lzyy2024
[fix](scheduler) Fix invalid access after freed (apache#48168)
448aa8d 
WRITE of size 1 at 0x6160007e86f0 thread T1983 (Pipe_normal [wo)
#0 0x55fc8065b975 in std::__atomic_base<bool>::store(bool,
std::memory_order)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:457:2
apache#1 0x55fc8065b975 in std::__atomic_base<bool>::operator=(bool)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:349:2
apache#2 0x55fc8065b975 in std::atomic<bool>::operator=(bool)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/atomic:80:22
apache#3 0x55fc8065b975 in doris::pipeline::PipelineTask::set_running(bool)
/root/doris/be/src/pipeline/pipeline_task.h:192:47
apache#4 0x55fc8065b975 in
doris::pipeline::TaskScheduler::_do_work(int)::$_0::operator()() const
/root/doris/be/src/pipeline/task_scheduler.cpp:121:23
apache#5 0x55fc8065b975 in
doris::Defer<doris::pipeline::TaskScheduler::_do_work(int)::$_0>::~Defer()
/root/doris/be/src/util/defer_op.h:37:16
apache#6 0x55fc8065b975 in doris::pipeline::TaskScheduler::_do_work(int)
/root/doris/be/src/pipeline/task_scheduler.cpp:162:5
apache#7 0x55fc4c57cd19 in doris::ThreadPool::dispatch_thread()
/root/doris/be/src/util/threadpool.cpp:608:24
apache#8 0x55fc4c55395e in std::function<void ()>::operator()() const
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
apache#9 0x55fc4c55395e in doris::Thread::supervise_thread(void*)
/root/doris/be/src/util/thread.cpp:498:5
apache#10 0x7f9ee3d25608 in start_thread
/build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
apache#11 0x7f9ee3fd2132 in __clone
/build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95

0x6160007e86f0 is located 624 bytes inside of 632-byte region
[0x6160007e8480,0x6160007e86f8)
freed by thread T1981 (Pipe_normal [wo) here:
#0 0x55fc47aa680d in operator delete(void*)
(/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x3376e80d)
(BuildId: 865149e62959581e)
apache#1 0x55fc8059db84 in
std::default_delete<doris::pipeline::PipelineTask>::operator()(doris::pipeline::PipelineTask*)
const
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2
apache#2 0x55fc8059db84 in std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >::~unique_ptr()
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361:4
apache#3 0x55fc8059db84 in void
std::destroy_at<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >
>(std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >*)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15
apache#4 0x55fc8059db84 in void
std::_Destroy<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >
>(std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >*)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:138:7
apache#5 0x55fc8059db84 in void
std::_Destroy_aux<false>::__destroy<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask>
>*>(std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >*,
std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >*)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:152:6
apache#6 0x55fc8059db84 in void
std::_Destroy<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask>
>*>(std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >*,
std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >*)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:184:7
apache#7 0x55fc8059db84 in void
std::_Destroy<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >*,
std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >
>(std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >*,
std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >*,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > >&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:746:7
apache#8 0x55fc8059db84 in
std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >::~vector()
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:680:2
apache#9 0x55fc8052571c in void
std::destroy_at<std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >
>(std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >*)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15
apache#10 0x55fc8052571c in void
std::_Destroy<std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >
>(std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >*)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:138:7
apache#11 0x55fc8052571c in void
std::_Destroy_aux<false>::__destroy<std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > >
>*>(std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >*,
std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >*)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:152:6
apache#12 0x55fc8052571c in void
std::_Destroy<std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > >
>*>(std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >*,
std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >*)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:184:7
apache#13 0x55fc8052571c in void
std::_Destroy<std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >*,
std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >
>(std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >*,
std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >*,
std::allocator<std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > > >&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:746:7
apache#14 0x55fc8052571c in
std::vector<std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >,
std::allocator<std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > > >
>::_M_erase_at_end(std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >*)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1796:6
apache#15 0x55fc8052571c in
std::vector<std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >,
std::allocator<std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > > > >::clear()
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1499:9
apache#16 0x55fc8052571c in
doris::pipeline::PipelineFragmentContext::~PipelineFragmentContext()
/root/doris/be/src/pipeline/pipeline_fragment_context.cpp:142:12
apache#17 0x55fc47ad30cc in
std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:168:6
apache#18 0x55fc80658d57 in
std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count()
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:702:11
apache#19 0x55fc80658d57 in std::__shared_ptr<doris::TaskExecutionContext,
(__gnu_cxx::_Lock_policy)2>::~__shared_ptr()
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1149:31
apache#20 0x55fc80658d57 in
doris::pipeline::close_task(doris::pipeline::PipelineTask*,
doris::Status) /root/doris/be/src/pipeline/task_scheduler.cpp:100:1
apache#21 0x55fc8065aa17 in doris::pipeline::TaskScheduler::_do_work(int)
/root/doris/be/src/pipeline/task_scheduler.cpp:160:36
apache#22 0x55fc4c57cd19 in doris::ThreadPool::dispatch_thread()
/root/doris/be/src/util/threadpool.cpp:608:24
apache#23 0x55fc4c55395e in std::function<void ()>::operator()() const
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
apache#24 0x55fc4c55395e in doris::Thread::supervise_thread(void*)
/root/doris/be/src/util/thread.cpp:498:5
apache#25 0x7f9ee3d25608 in start_thread
/build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
@Gabriel39 Gabriel39 mentioned this issue Feb 26, 2025
Merged
16 tasks
Gabriel39 added a commit that referenced this issue Feb 26, 2025
@Gabriel39
[fix](schema scan) Fix invalid pointer access (#48313)
961a844 
Schema scanner runs on a separate thread which is executed
asynchronously. We should make sure all context used not be freed once
it is scheduled.

ERROR: AddressSanitizer: heap-buffer-overflow on address 0x613002f33eb2
at pc 0x55e085dccbe3 bp 0x7f345c0e1f10 sp 0x7f345c0e1f08
READ of size 1 at 0x613002f33eb2 thread T2776 (FragmentMgrAsyn)
#0 0x55e085dccbe2 in std::__atomic_base::load(std::memory_order) const
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:481:9
#1 0x55e085dccbe2 in std::atomic::operator bool() const
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/atomic:87:22
#2 0x55e085dccbe2 in
doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0::operator()()
const
/home/zcp/repo_center/doris_master/doris/be/src/exec/schema_scanner.cpp:118:5
#3 0x55e085dccbe2 in void std::__invoke_impl(std::__invoke_other,
doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
#4 0x55e085dccbe2 in std::enable_if, void>::type
std::__invoke_r(doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
#5 0x55e085dccbe2 in std::_Function_handler::_M_invoke(std::_Any_data
const&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
#6 0x55e050f081ca in doris::ThreadPool::dispatch_thread()
/home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:608:24
#7 0x55e050ede467 in doris::Thread::supervise_thread(void*)
/home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5
    #8 0x7f376ef5aac2 in start_thread nptl/pthread_create.c:442:8
#9 0x7f376efec84f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
github-actions bot pushed a commit that referenced this issue Feb 26, 2025
@Gabriel39
[fix](schema scan) Fix invalid pointer access (#48313)
3284277 
Schema scanner runs on a separate thread which is executed
asynchronously. We should make sure all context used not be freed once
it is scheduled.

ERROR: AddressSanitizer: heap-buffer-overflow on address 0x613002f33eb2
at pc 0x55e085dccbe3 bp 0x7f345c0e1f10 sp 0x7f345c0e1f08
READ of size 1 at 0x613002f33eb2 thread T2776 (FragmentMgrAsyn)
#0 0x55e085dccbe2 in std::__atomic_base::load(std::memory_order) const
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:481:9
#1 0x55e085dccbe2 in std::atomic::operator bool() const
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/atomic:87:22
#2 0x55e085dccbe2 in
doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0::operator()()
const
/home/zcp/repo_center/doris_master/doris/be/src/exec/schema_scanner.cpp:118:5
#3 0x55e085dccbe2 in void std::__invoke_impl(std::__invoke_other,
doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
#4 0x55e085dccbe2 in std::enable_if, void>::type
std::__invoke_r(doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
#5 0x55e085dccbe2 in std::_Function_handler::_M_invoke(std::_Any_data
const&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
#6 0x55e050f081ca in doris::ThreadPool::dispatch_thread()
/home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:608:24
#7 0x55e050ede467 in doris::Thread::supervise_thread(void*)
/home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5
    #8 0x7f376ef5aac2 in start_thread nptl/pthread_create.c:442:8
#9 0x7f376efec84f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
github-actions bot pushed a commit that referenced this issue Feb 26, 2025
@Gabriel39
[fix](schema scan) Fix invalid pointer access (#48313)
c01234d 
Schema scanner runs on a separate thread which is executed
asynchronously. We should make sure all context used not be freed once
it is scheduled.

ERROR: AddressSanitizer: heap-buffer-overflow on address 0x613002f33eb2
at pc 0x55e085dccbe3 bp 0x7f345c0e1f10 sp 0x7f345c0e1f08
READ of size 1 at 0x613002f33eb2 thread T2776 (FragmentMgrAsyn)
#0 0x55e085dccbe2 in std::__atomic_base::load(std::memory_order) const
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:481:9
#1 0x55e085dccbe2 in std::atomic::operator bool() const
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/atomic:87:22
#2 0x55e085dccbe2 in
doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0::operator()()
const
/home/zcp/repo_center/doris_master/doris/be/src/exec/schema_scanner.cpp:118:5
#3 0x55e085dccbe2 in void std::__invoke_impl(std::__invoke_other,
doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
#4 0x55e085dccbe2 in std::enable_if, void>::type
std::__invoke_r(doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
#5 0x55e085dccbe2 in std::_Function_handler::_M_invoke(std::_Any_data
const&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
#6 0x55e050f081ca in doris::ThreadPool::dispatch_thread()
/home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:608:24
#7 0x55e050ede467 in doris::Thread::supervise_thread(void*)
/home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5
    #8 0x7f376ef5aac2 in start_thread nptl/pthread_create.c:442:8
#9 0x7f376efec84f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
@Gabriel39 Gabriel39 mentioned this issue Feb 26, 2025
Merged
16 tasks
Gabriel39 added a commit that referenced this issue Feb 26, 2025
@Gabriel39
[fix](schema scan) Fix invalid pointer access (#48370)
a3eb715 
==14365==ERROR: AddressSanitizer: heap-buffer-overflow on address
0x613002873c70 at pc 0x5559a290ec05 bp 0x7f6c40385e90 sp 0x7f6c40385e88
READ of size 2 at 0x613002873c70 thread T4019 (FragmentMgrAsyn)
#0 0x5559a290ec04 in std::__atomic_base::load(std::memory_order) const
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:481:9
#1 0x5559a290ec04 in doris::AtomicStatus::update(doris::Status const&)
/home/zcp/repo_center/doris_master/doris/be/src/common/status.h:601:44
#2 0x5559d8f289c0 in
doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0::operator()()
const
/home/zcp/repo_center/doris_master/doris/be/src/exec/schema_scanner.cpp:118:5
#3 0x5559d8f289c0 in void std::__invoke_impl(std::__invoke_other,
doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
#4 0x5559d8f289c0 in std::enable_if, void>::type
std::__invoke_r(doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
#5 0x5559d8f289c0 in std::_Function_handler::_M_invoke(std::_Any_data
const&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
#6 0x5559a4140e4a in doris::ThreadPool::dispatch_thread()
/home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:608:24
#7 0x5559a4117877 in doris::Thread::supervise_thread(void*)
/home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5
    #8 0x7f7912f84ac2 in start_thread nptl/pthread_create.c:442:8
#9 0x7f791301684f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
github-actions bot pushed a commit that referenced this issue Feb 26, 2025
@Gabriel39
[fix](schema scan) Fix invalid pointer access (#48370)
116d912 
==14365==ERROR: AddressSanitizer: heap-buffer-overflow on address
0x613002873c70 at pc 0x5559a290ec05 bp 0x7f6c40385e90 sp 0x7f6c40385e88
READ of size 2 at 0x613002873c70 thread T4019 (FragmentMgrAsyn)
#0 0x5559a290ec04 in std::__atomic_base::load(std::memory_order) const
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:481:9
#1 0x5559a290ec04 in doris::AtomicStatus::update(doris::Status const&)
/home/zcp/repo_center/doris_master/doris/be/src/common/status.h:601:44
#2 0x5559d8f289c0 in
doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0::operator()()
const
/home/zcp/repo_center/doris_master/doris/be/src/exec/schema_scanner.cpp:118:5
#3 0x5559d8f289c0 in void std::__invoke_impl(std::__invoke_other,
doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
#4 0x5559d8f289c0 in std::enable_if, void>::type
std::__invoke_r(doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
#5 0x5559d8f289c0 in std::_Function_handler::_M_invoke(std::_Any_data
const&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
#6 0x5559a4140e4a in doris::ThreadPool::dispatch_thread()
/home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:608:24
#7 0x5559a4117877 in doris::Thread::supervise_thread(void*)
/home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5
    #8 0x7f7912f84ac2 in start_thread nptl/pthread_create.c:442:8
#9 0x7f791301684f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
github-actions bot pushed a commit that referenced this issue Feb 26, 2025
@Gabriel39
[fix](schema scan) Fix invalid pointer access (#48370)
63a5d04 
==14365==ERROR: AddressSanitizer: heap-buffer-overflow on address
0x613002873c70 at pc 0x5559a290ec05 bp 0x7f6c40385e90 sp 0x7f6c40385e88
READ of size 2 at 0x613002873c70 thread T4019 (FragmentMgrAsyn)
#0 0x5559a290ec04 in std::__atomic_base::load(std::memory_order) const
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:481:9
#1 0x5559a290ec04 in doris::AtomicStatus::update(doris::Status const&)
/home/zcp/repo_center/doris_master/doris/be/src/common/status.h:601:44
#2 0x5559d8f289c0 in
doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0::operator()()
const
/home/zcp/repo_center/doris_master/doris/be/src/exec/schema_scanner.cpp:118:5
#3 0x5559d8f289c0 in void std::__invoke_impl(std::__invoke_other,
doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
#4 0x5559d8f289c0 in std::enable_if, void>::type
std::__invoke_r(doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
#5 0x5559d8f289c0 in std::_Function_handler::_M_invoke(std::_Any_data
const&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
#6 0x5559a4140e4a in doris::ThreadPool::dispatch_thread()
/home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:608:24
#7 0x5559a4117877 in doris::Thread::supervise_thread(void*)
/home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5
    #8 0x7f7912f84ac2 in start_thread nptl/pthread_create.c:442:8
#9 0x7f791301684f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
@Gabriel39 Gabriel39 mentioned this issue Mar 13, 2025
Open
16 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@imay @cyongli @astor-oss @Knuth1989