graph/db: use /*SLICE:<field_name>*/ to optimise various graph queries
#10081
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @ellemouton, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request introduces significant optimizations to graph database queries by leveraging SQL's IN clause with a custom /*SLICE:<field_name>*/ directive. To manage potentially large parameter lists, a new ExecutePagedQuery helper has been implemented, allowing these batch queries to be processed in a paginated fashion. This enhancement aims to reduce the number of database calls and improve overall query performance, especially for operations involving multiple graph entities.
Highlights
- Database Query Optimization: Implemented the
/*SLICE:<field_name>*/directive to allow for efficientWHERE x IN <list>SQL queries, significantly reducing the number of database calls by batching multiple lookups into a single query. - Paginated Query Execution: Introduced a new
ExecutePagedQueryhelper function that enables queries with large lists of parameters to be executed in paginated batches. This prevents database parameter limits from being hit and ensures robust handling of extensive data sets. - Refactored Graph Store Queries: Applied the new paginated query mechanism to key methods within the
SQLStore, specificallyFetchChanInfosandPruneGraph, to improve their performance and efficiency when processing multiple channel IDs or outpoints. - SQLC Tooling Enhancement: Added a post-generation script to
scripts/gen_sqlc_docker.shto correctly handlesqlc.slice()directives for Postgres. This ensures proper parameter substitution in the generated Go code, addressing a limitation insqlc's default behavior forINclauses. - Test Suite Improvements: Updated various graph tests to use
require.NoError,require.Len, andrequire.Emptyfromtestify/requirefor more concise and robust error and assertion handling, improving test readability and maintainability.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This pull request introduces a generic, paginated query helper to optimize database queries. This is a great performance improvement that reduces the number of database calls by using IN clauses, while also preventing issues with an unbound number of parameters. The implementation is solid, with good test coverage for the new functionality.
ellemouton
force-pushed
the
graphSQLSlices
branch
5 times, most recently
from
b89b18b to
5d9bf9b
Compare
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces the use of the /*SLICE:<field_name>*/ directive to optimize graph queries by enabling the use of WHERE x IN <list of x type> SQL queries. It also adds an ExecutePagedQuery helper to handle these queries in a paginated fashion. The code changes include modifications to the graph database schema, query functions, and test cases to support this new feature.
ellemouton
force-pushed
the
graphSQLSlices
branch
from
5d9bf9b to
bf32adb
Compare
graph/db/sql_store.go
Outdated
| return fmt.Errorf("unable to fetch channel: %w", | ||
| err) | ||
| } | ||
| // Define the callback function for processing each channel |
There was a problem hiding this comment.
nit: missing punctuation in some comments here and below.
ellemouton
force-pushed
the
graphSQLSlices
branch
3 times, most recently
from
aa9d072 to
488b70a
Compare
yes indeed :) |
ellemouton
force-pushed
the
graphSQLSlices
branch
4 times, most recently
from
91f40c6 to
10379b5
Compare
This copies the workaround introduced in the taproot-assets code base and will allow us to use `WHERE x in <list>` type queries.
Along with a test for it. This helper will allow us to easily create a pagination wrapper for queries that will make use of the new /*SLICE:<field_name>*/ directive. The next commit will add a test showing this.
Here, a new query (GetChannelsByOutpoints) is added which makes use of the /*SLICE:outpoints*/ directive & added workaround. This is then used in a test to demonstrate how the ExecutePagedQuery helper can be used to wrap a query like this such that calls are done in pages. The query that has been added will also be used by live code paths in an upcoming commit.
This commit adds a new GetChannelsByOutpoints query which takes a slice of outpoint strings. This lets us then update PruneGraph to use paginated calls to GetChannelsByOutpoints instead of making one DB call per outpoint.
Remove a TODO by making use of the new sqldb.ExecutePagedQuery to fetch channels in batches rather than one by one.
Update it to use the new wrapped version of GetChannelsBySCIDWithPolicies to reduce the number of DB calls.
Use the new `SLICES` directive to add a DeleteChannels query which takes a set of DB channel IDs. Then replace all our calls to DeleteChannel with a paginated call to DeleteChannels.
ellemouton
force-pushed
the
graphSQLSlices
branch
from
10379b5 to
ddc0e95
Compare
In this PR, the
/*SLICE:<field_name>*/directive is made use of so that we can performWHERE x IN <list of x type>SQL queries. This then allows us to perform fewer DB calls.We dont want to give our SQL lib an unbound number of items in the parameter though so this
PR also adds an
ExecutePagedQueryhelper that gives us the ability to call these new types ofSQL queries in a paginated fashion,.