Redis catalog: How to use the raw decoder to correctly query the predefined BIGINT type value?

[zhangbutao]

Here is my test:

1. Set redis key/value through redis-cli

127.0.0.1:6379> SET testdb:testrawint:aa 123456789
OK

2. Create trino-redis catalog

connector.name=redis
redis.nodes=127.0.0.1:6379
redis.password=psswd
redis.table-description-cache-ttl=1s
redis.table-description-dir=/etc/redis_raw_test

redis.key-prefix-schema-table=true

3. Create redis raw decoder json definition file
   Refer to https://trino.io/docs/current/connector/kafka.html#raw-encoder
   cat /etc/redis_raw_test/testraw.json

{
   "tableName": "testrawint",
   "schemaName": "testdb",
   "key": {
       "dataFormat": "raw",
              "fields": [
            {
                "name":"redis_key",
                "type":"VARCHAR",
                "hidden":"false"
           }
       ]
   },
   "value": {
       "dataFormat": "raw",
       "fields": [
           {
                "name":"id",
                "mapping":"0",
                "dataFormat": "LONG",
                "type":"BIGINT"
           }

       ]
    }
}

4. Query this redis key/value using tino-cli

trino> select * from redisraw.testdb.testrawint;
      redis_key       |         id
----------------------+---------------------
 testdb:testrawint:aa | 3544952156018063160
(1 row)

You will find that id is 3544952156018063160 instead of 123456789

5. Change the json definition file to replace type BIGINT to VARCHAR

{
   "tableName": "testrawint",
   "schemaName": "testdb",
   "key": {
       "dataFormat": "raw",
              "fields": [
            {
                "name":"redis_key",
                "type":"VARCHAR",
                "hidden":"false"
           }
       ]
   },
   "value": {
       "dataFormat": "raw",
       "fields": [
           {
                "name":"id",
                "mapping":"0",
                "dataFormat": "BYTE",
                "type":"VARCHAR"
           }

       ]
    }
}

And then you can get the correct id value 123456789

trino> select * from redisraw.testdb.testrawint;
      redis_key       |    id
----------------------+-----------
 testdb:testrawint:aa | 123456789

Question

From the above test, i think trino-redis catalog can not use the raw decoder to query BIGINT/SMALLINT(or other numerical types) value.

This is because the Redis data in the test is always parsed into string type, and then the string type data is converted into bytes. If this byte is further converted into int type(or other numerical types), it will lead to incorrect results.

trino/plugin/trino-redis/src/main/java/io/trino/plugin/redis/RedisRecordCursor.java

Lines 225 to 235 in 2bc657e

	private void generateRowValues(String keyString, String valueString, @Nullable Map<String, String> hashValueMap)
	{
	byte[] keyData = keyString.getBytes(StandardCharsets.UTF_8);
	byte[] stringValueData = valueString.getBytes(StandardCharsets.UTF_8);
	// Redis connector supports two types of Redis values: STRING and HASH. HASH type requires hash row decoder to
	// decode a row from map, whereas for the STRING type decoders are optional. The redis keyData is always byte array,
	// so the decoder of key always decodes a row from bytes.
	Optional<Map<DecoderColumnHandle, FieldValueProvider>> decodedKey = keyDecoder.decodeRow(keyData);
	Optional<Map<DecoderColumnHandle, FieldValueProvider>> decodedValue = valueDecoder instanceof RedisRowDecoder redisRowDecoder
	? redisRowDecoder.decodeRow(hashValueMap)
	: valueDecoder.decodeRow(stringValueData);

trino/lib/trino-record-decoder/src/main/java/io/trino/decoder/raw/RawColumnDecoder.java

Line 206 in 2bc657e

return new RawValueProvider(ByteBuffer.wrap(value, start, actualEnd - start), fieldType, columnName, columnType);

trino/lib/trino-record-decoder/src/main/java/io/trino/decoder/raw/RawColumnDecoder.java

Lines 249 to 256 in 2bc657e

	public long getLong()
	{
	checkEnoughBytes();
	return switch (fieldType) {
	case BYTE -> value.get(start);
	case SHORT -> value.getShort(start);
	case INT -> value.getInt(start);
	case LONG -> value.getLong(start);

So, I think trino-redis can only use the raw decoder to query the predefined VARCHAR type value.
WDYT?

https://trino.io/docs/current/connector/redis.html#raw-decoder BTW, the doc shows the raw-decoder can support numerical values. I think it should be removed.