Skip to content

[improve](statistic)Reduce call of olap table getVisibleVersion for cloud version. #53903

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jul 29, 2025

Conversation

Jibing-Li
Copy link
Contributor

What problem does this PR solve?

  1. Reduce call of olap table getVisibleVersion for cloud version, because it is an RPC in cloud mode.
  2. Improve auto collect job append logic, make very low level jobs could be executed more immediately

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Jibing-Li Jibing-Li marked this pull request as ready for review July 25, 2025 09:46
@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34088 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 40c34159b81043d4b2f627ee174f5cffd9b677b6, data reload: false

------ Round 1 ----------------------------------
q1	17583	5252	5198	5198
q2	1956	294	173	173
q3	10285	1270	722	722
q4	10214	973	520	520
q5	7472	2465	2274	2274
q6	176	169	132	132
q7	880	736	603	603
q8	9306	1299	1103	1103
q9	7241	5199	5101	5101
q10	6959	2374	1963	1963
q11	488	300	274	274
q12	369	349	222	222
q13	17771	3716	3151	3151
q14	231	234	220	220
q15	568	493	482	482
q16	427	424	383	383
q17	594	846	361	361
q18	7669	7090	7067	7067
q19	1643	963	552	552
q20	333	338	220	220
q21	3617	3233	2362	2362
q22	1065	1038	1005	1005
Total cold run time: 106847 ms
Total hot run time: 34088 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5269	5099	5118	5099
q2	241	319	228	228
q3	2143	2683	2323	2323
q4	1358	1820	1346	1346
q5	4208	4589	4511	4511
q6	216	172	131	131
q7	2023	1971	1781	1781
q8	2628	2572	2657	2572
q9	7300	7222	7375	7222
q10	3057	3322	2855	2855
q11	609	537	492	492
q12	734	770	644	644
q13	3623	4029	3340	3340
q14	301	299	275	275
q15	526	485	491	485
q16	449	481	456	456
q17	1200	1563	1413	1413
q18	8056	7806	7727	7727
q19	787	765	874	765
q20	2116	2007	1821	1821
q21	4795	4366	4296	4296
q22	1072	1003	1015	1003
Total cold run time: 52711 ms
Total hot run time: 50785 ms

@morrySnow morrySnow added usercase Important user case type label dev/3.0.x dev/3.1.x labels Jul 25, 2025
@doris-robot
Copy link

TPC-DS: Total hot run time: 187630 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 40c34159b81043d4b2f627ee174f5cffd9b677b6, data reload: false

query1	979	380	415	380
query2	6536	1733	1679	1679
query3	6757	219	217	217
query4	26459	23542	22926	22926
query5	4311	607	508	508
query6	312	221	216	216
query7	4626	492	293	293
query8	286	237	222	222
query9	8628	2880	2851	2851
query10	476	343	270	270
query11	15431	15029	14791	14791
query12	163	116	111	111
query13	1654	535	424	424
query14	8572	6000	5960	5960
query15	217	208	160	160
query16	7210	631	463	463
query17	1183	712	570	570
query18	1983	397	298	298
query19	182	185	171	171
query20	123	111	113	111
query21	214	122	105	105
query22	4262	4188	4312	4188
query23	34016	33014	33018	33014
query24	8001	2454	2370	2370
query25	531	475	399	399
query26	1238	275	158	158
query27	2745	505	362	362
query28	4374	2232	2206	2206
query29	761	544	459	459
query30	293	216	188	188
query31	918	824	777	777
query32	80	77	70	70
query33	548	376	334	334
query34	794	843	531	531
query35	799	811	755	755
query36	968	1038	905	905
query37	118	106	84	84
query38	4059	4170	4030	4030
query39	1459	1404	1388	1388
query40	222	124	116	116
query41	75	66	60	60
query42	123	113	114	113
query43	505	492	469	469
query44	1331	861	862	861
query45	173	166	169	166
query46	870	1015	645	645
query47	1727	1801	1745	1745
query48	369	424	310	310
query49	728	486	395	395
query50	674	682	404	404
query51	5537	5518	5633	5518
query52	122	110	103	103
query53	228	264	197	197
query54	584	595	534	534
query55	93	85	89	85
query56	322	319	329	319
query57	1161	1188	1155	1155
query58	281	271	268	268
query59	2542	2676	2476	2476
query60	356	340	337	337
query61	128	122	132	122
query62	806	738	670	670
query63	227	196	216	196
query64	4309	1002	703	703
query65	4239	4167	4197	4167
query66	1155	411	331	331
query67	15669	15659	15686	15659
query68	4035	867	590	590
query69	485	323	274	274
query70	1192	1154	1145	1145
query71	417	353	313	313
query72	5596	4808	4985	4808
query73	682	654	363	363
query74	9225	9002	8751	8751
query75	3133	3084	2633	2633
query76	3142	1126	732	732
query77	483	393	333	333
query78	9913	10641	9322	9322
query79	1532	870	593	593
query80	660	628	474	474
query81	466	253	229	229
query82	220	144	105	105
query83	253	259	248	248
query84	248	105	77	77
query85	772	355	346	346
query86	351	308	331	308
query87	4420	4478	4326	4326
query88	2886	2268	2249	2249
query89	386	314	288	288
query90	1793	230	224	224
query91	143	152	128	128
query92	73	76	69	69
query93	1132	986	641	641
query94	647	387	316	316
query95	411	333	328	328
query96	486	601	280	280
query97	2725	2705	2597	2597
query98	253	227	214	214
query99	1326	1393	1258	1258
Total cold run time: 266308 ms
Total hot run time: 187630 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.62 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 40c34159b81043d4b2f627ee174f5cffd9b677b6, data reload: false

query1	0.05	0.03	0.04
query2	0.09	0.04	0.04
query3	0.24	0.07	0.08
query4	1.62	0.11	0.11
query5	0.43	0.41	0.41
query6	1.18	0.65	0.68
query7	0.03	0.02	0.02
query8	0.05	0.04	0.03
query9	0.59	0.53	0.52
query10	0.58	0.56	0.57
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.65	0.61	0.60
query14	0.81	0.84	0.83
query15	0.92	0.90	0.88
query16	0.38	0.40	0.40
query17	1.06	1.09	1.10
query18	0.21	0.20	0.20
query19	1.99	1.82	1.87
query20	0.01	0.01	0.02
query21	15.39	0.96	0.53
query22	0.77	1.06	0.71
query23	15.02	1.39	0.63
query24	7.49	0.68	0.82
query25	0.50	0.22	0.09
query26	0.64	0.17	0.15
query27	0.05	0.05	0.05
query28	9.26	0.94	0.45
query29	12.60	3.92	3.30
query30	3.09	3.03	2.99
query31	2.83	0.58	0.38
query32	3.23	0.56	0.49
query33	3.09	3.05	3.11
query34	16.06	5.51	4.90
query35	4.89	4.88	4.92
query36	0.69	0.51	0.50
query37	0.10	0.07	0.07
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.18	0.14	0.13
query41	0.08	0.02	0.02
query42	0.04	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 107.32 s
Total hot run time: 32.62 s

@Jibing-Li
Copy link
Contributor Author

run p0

2 similar comments
@Jibing-Li
Copy link
Contributor Author

run p0

@Jibing-Li
Copy link
Contributor Author

run p0

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 29, 2025
Copy link
Contributor

PR approved by anyone and no changes requested.

@Jibing-Li Jibing-Li merged commit 189d621 into apache:master Jul 29, 2025
30 of 31 checks passed
@Jibing-Li Jibing-Li deleted the version branch July 29, 2025 06:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.x dev/3.0.x-conflict dev/3.1.x dev/3.1.x-conflict reviewed usercase Important user case type label
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants