Skip to content

Added the two new types any_pointer and any_array #164

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 18 commits into
base: master
Choose a base branch
from
Open

Added the two new types any_pointer and any_array #164

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
cd76541
Added the two new types anyptr and anyarray
Oct 27, 2022
e147292
Cleaned up the review a bit
Oct 27, 2022
6ed6d76
Merge remote-tracking branch 'origin/master' into anyptr_and_anyarray
Oct 27, 2022
ec4197c
Merge remote-tracking branch 'origin/master' into anyptr_and_anyarray
Nov 2, 2022
cea669f
Renamed anyptr to any_pointer and anyarray to any_array
Nov 2, 2022
34a3eab
Fixed the docs as well
Nov 2, 2022
e61256e
Whitespace in README
Nov 2, 2022
a0bc08c
Merge branch 'master' into anyptr_and_anyarray
Nov 3, 2022
5f8587c
Merge branch 'official_master' into anyptr_and_anyarray
Nov 11, 2022
2ddf51e
Merge branch 'master' into anyptr_and_anyarray
Nov 11, 2022
b08dd2a
Redid the any_pointer and any_array json format
Apr 3, 2023
890bc01
And the implementation for the last change
Apr 3, 2023
c9e2d4d
Gcc giving other warnings than msvc
Apr 3, 2023
6d6d0a5
And one more warning
Apr 3, 2023
92205ac
Do I need to get WSL?
Apr 3, 2023
d78bc4b
Merge branch 'Official_master' into anyptr_and_anyarray
Tisten Feb 26, 2024
f65b499
Fixed bad merge
Tisten Feb 27, 2024
429cc83
Different warnings from cmake and bam
Tisten Feb 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 38 additions & 11 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,16 +60,18 @@ tool\

## Supported POD-Types in Defined Structs

| name | tld-type | C-storage | comment |
|----------------|------------------------------|---------------------------------------|-----------------------------------------------------------------|
| signed int | int8, int16, int32, int64 | int8_t, int16_t, int32_t, int64_t | |
| unsigned int | int8, uint16, uint32, uint64 | uint8_t, uint16_t, uint32_t, uint64_t | |
| floating point | fp32, fp64 | float/double | |
| bitfield | bitfield:<bits> | uint32_t bf : 3 | unsigned integer with specified amount of bits |
| string | string | const char* | utf8 encoded, null-terminated |
| inline_array | int8[20], some_type[20] | type[20] | fixed length, type can be any POD-type or user defined struct |
| array | int8[], some_type[] | struct { type* data; uint32_t count } | dynamic length, type can be any POD-type or user defined struct |
| pointer | some_type* | type* | pointer to any user-defined type |
| name | tld-type | C-storage | comment |
|----------------|------------------------------|---------------------------------------------|----------------------------------------------------------------------|
| signed int | int8, int16, int32, int64 | int8_t, int16_t, int32_t, int64_t | |
| unsigned int | int8, uint16, uint32, uint64 | uint8_t, uint16_t, uint32_t, uint64_t | |
| floating point | fp32, fp64 | float/double | |
| bitfield | bitfield:<bits> | uint32_t bf : 3 | unsigned integer with specified amount of bits |
| string | string | const char* | utf8 encoded, null-terminated |
| inline_array | int8[20], some_type[20] | type[20] | fixed length, type can be any POD-type or user defined struct |
| array | int8[], some_type[] | struct { type* data; uint32_t count } | dynamic length, type can be any POD-type or user defined struct |
| pointer | some_type* | type* | pointer to any user-defined type |
| any_pointer | any_pointer | struct { void* ptr; dl_typeid_t tid; } | pointer to any struct type, where the type is decided on assignment |
| any_array | any_array | struct { dl_array array; dl_typeid_t tid; } | all elements are of the same type, the type is decided on assignment |

## DL-JSON

Expand Down Expand Up @@ -104,6 +106,10 @@ and DL will always error out if the value do not fit the type.
* scientific float notation, such as 3.2e+12, 1.2e-5
* constants `inf`, `infinity` and `nan` ( ignoring case! )

### Any types

These types are based on un-typed pointers and the actual type is set when assigning the data.

## TLD (Type Library Definition) Format

Type-libs in text-format follows this format.
Expand Down Expand Up @@ -286,6 +292,22 @@ left untouched!",
// inline arrays are just arrays with the correct type ...
"inline_array_member1" : [1, 2, 3],

// an any_pointer can be null, and then it is typeless
"any_pointer_member1": null,
// but when it has a value then that will be stored in the "__subdata" section, and there will be a "__any_type_" entry right after the member.
"any_pointer_member2": "ptr1234"
"__any_type_any_pointer_member2" : "my_struct",

// any_array members will also have the "__any_type_" entry right after the member
"any_array_member": [
{
"x" : 1,
"y" : 2
},
]
"__any_type_any_array_member" : "my_struct",


// ... it could be a struct with x and y members.
"inline_array_member2" : [
{
Expand Down Expand Up @@ -328,7 +350,12 @@ left untouched!",

"another_sub_data" : {
"some_float" : 1.23
}
},

"ptr1234" : {
"x" : 1,
"y" : 2
},
}
}
```
Expand Down
2 changes: 2 additions & 0 deletions include/dl/dl.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,8 @@ typedef enum
DL_TYPE_STORAGE_STR,
DL_TYPE_STORAGE_PTR,
DL_TYPE_STORAGE_STRUCT,
DL_TYPE_STORAGE_ANY_POINTER,
DL_TYPE_STORAGE_ANY_ARRAY,

DL_TYPE_STORAGE_CNT
} dl_type_storage_t;
Expand Down
167 changes: 147 additions & 20 deletions src/dl.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,9 +72,9 @@ static dl_error_t dl_ptr_chain_patching( const dl_data_header* header, uint8_t*
}

dl_error_t dl_instance_load( dl_ctx_t dl_ctx, dl_typeid_t type_id,
void* instance, size_t instance_size,
const unsigned char* packed_instance, size_t packed_instance_size,
size_t* consumed )
void* instance, size_t instance_size,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR contains some "space to tab" conversions to minimize the "strange indent here" feedback.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll try to get to fixing a clang-format file :)

const unsigned char* packed_instance, size_t packed_instance_size,
size_t* consumed )
{
dl_data_header* header = (dl_data_header*)packed_instance;

Expand Down Expand Up @@ -152,7 +152,7 @@ struct CDLBinStoreContext
dl_binary_writer_init( &writer, out_data, out_data_size, is_dummy, DL_ENDIAN_HOST, DL_ENDIAN_HOST, DL_PTR_SIZE_HOST );
}

uintptr_t FindWrittenPtr( void* ptr )
uintptr_t FindWrittenPtr( const uint8_t* ptr )
{
if( ptr == 0 )
return (uintptr_t)0;
Expand All @@ -165,7 +165,7 @@ struct CDLBinStoreContext
return (uintptr_t)0;
}

void AddWrittenPtr( const void* ptr, uintptr_t pos )
void AddWrittenPtr( const uint8_t* ptr, uintptr_t pos )
{
written_ptrs.Add( { pos, ptr } );
}
Expand All @@ -183,8 +183,8 @@ struct CDLBinStoreContext

struct SWrittenPtr
{
uintptr_t pos;
const void* ptr;
uintptr_t pos;
const uint8_t* ptr;
};
CArrayStatic<SWrittenPtr, 128> written_ptrs;
CArrayStatic<uintptr_t, 256> ptrs;
Expand Down Expand Up @@ -224,11 +224,11 @@ static void dl_internal_store_string( const uint8_t* instance, CDLBinStoreContex
dl_binary_writer_write( &store_ctx->writer, &offset, sizeof(uintptr_t) );
}

static dl_error_t dl_internal_instance_store( dl_ctx_t dl_ctx, const dl_type_desc* type, uint8_t* instance, CDLBinStoreContext* store_ctx );
static dl_error_t dl_internal_instance_store( dl_ctx_t dl_ctx, const dl_type_desc* type, const uint8_t* instance, CDLBinStoreContext* store_ctx );
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "store" functions now treat the data as const


static dl_error_t dl_internal_store_ptr( dl_ctx_t dl_ctx, uint8_t* instance, const dl_type_desc* sub_type, CDLBinStoreContext* store_ctx )
static dl_error_t dl_internal_store_ptr( dl_ctx_t dl_ctx, const uint8_t* instance, const dl_type_desc* sub_type, CDLBinStoreContext* store_ctx )
{
uint8_t* data = *(uint8_t**)instance;
const uint8_t* data = *(const uint8_t* const*)instance;
uintptr_t offset = store_ctx->FindWrittenPtr( data );

if( data == 0x0 ) // Null-pointer, store pint(-1) to signal to patching!
Expand Down Expand Up @@ -265,7 +265,7 @@ static dl_error_t dl_internal_store_ptr( dl_ctx_t dl_ctx, uint8_t* instance, con
return DL_ERROR_OK;
}

static dl_error_t dl_internal_store_array( dl_ctx_t dl_ctx, dl_type_storage_t storage_type, const dl_type_desc* sub_type, uint8_t* instance, uint32_t count, uintptr_t size, CDLBinStoreContext* store_ctx )
static dl_error_t dl_internal_store_array( dl_ctx_t dl_ctx, dl_type_storage_t storage_type, const dl_type_desc* sub_type, const uint8_t* instance, uint32_t count, uintptr_t size, CDLBinStoreContext* store_ctx )
{
switch( storage_type )
{
Expand Down Expand Up @@ -297,18 +297,77 @@ static dl_error_t dl_internal_store_array( dl_ctx_t dl_ctx, dl_type_storage_t st
return err;
}
break;
case DL_TYPE_STORAGE_ANY_POINTER:
for( uint32_t elem = 0; elem < count; ++elem )
{
dl_typeid_t tid = *reinterpret_cast<const dl_typeid_t*>( instance + ( ( elem * 2 + 1 ) * sizeof( void* ) ) );
sub_type = dl_internal_find_type( dl_ctx, tid );
dl_error_t err = dl_internal_store_ptr( dl_ctx, instance + ( elem * sizeof( void* ) * 2 ), sub_type, store_ctx );
if( DL_ERROR_OK != err )
return err;
dl_binary_writer_write_uint32( &store_ctx->writer, tid );
if DL_CONSTANT_EXPRESSION(sizeof(uintptr_t) == 8)
dl_binary_writer_write_zero( &store_ctx->writer, 4 );
}
break;
default: // default is a standard pod-type
dl_binary_writer_write( &store_ctx->writer, instance, count * size );
break;
}
return DL_ERROR_OK;
}

static dl_error_t dl_internal_store_member( dl_ctx_t dl_ctx, const dl_member_desc* member, uint8_t* instance, CDLBinStoreContext* store_ctx )
static dl_error_t dl_internal_store_anyarray( dl_ctx_t dl_ctx, const uint8_t* data_ptr, uint32_t count, CDLBinStoreContext* store_ctx )
{
for (uint32_t i = 0; i < count; ++i)
{
uintptr_t sub_pos = dl_binary_writer_tell( &store_ctx->writer );
dl_binary_writer_seek_end( &store_ctx->writer );

const uint8_t* sub_data_ptr = data_ptr + i * 3 * sizeof(void*);
const uint8_t* sub_data = *(const uint8_t* const*)sub_data_ptr;
uint32_t sub_count = *(const uint32_t*)( sub_data_ptr + sizeof(void*) );
dl_typeid_t sub_type_id = *(const dl_typeid_t*)( sub_data_ptr + 2 * sizeof(void*) );
const dl_type_desc* sub_type = dl_internal_find_type( dl_ctx, sub_type_id );
size_t sub_size = dl_internal_align_up( sub_type->size[DL_PTR_SIZE_HOST], sub_type->alignment[DL_PTR_SIZE_HOST] );
dl_binary_writer_align( &store_ctx->writer, sub_type->alignment[DL_PTR_SIZE_HOST] );

uintptr_t sub_offset = dl_binary_writer_tell( &store_ctx->writer );

// write data!
dl_binary_writer_reserve( &store_ctx->writer, sub_count * sub_size ); // reserve space for array so subdata is placed correctly

dl_error_t err = dl_internal_store_array(dl_ctx, DL_TYPE_STORAGE_STRUCT, sub_type, sub_data, sub_count, sub_size, store_ctx);
if (DL_ERROR_OK != err)
return err;
dl_binary_writer_seek_set( &store_ctx->writer, sub_pos );

if( !store_ctx->writer.dummy )
store_ctx->ptrs.Add( dl_binary_writer_tell( &store_ctx->writer ) );

// make room for ptr
dl_binary_writer_write( &store_ctx->writer, &sub_offset, sizeof(uintptr_t) );

// write count
dl_binary_writer_write( &store_ctx->writer, &sub_count, sizeof(uint32_t) );
if DL_CONSTANT_EXPRESSION(sizeof(uintptr_t) == 8)
dl_binary_writer_write_zero( &store_ctx->writer, sizeof(uint32_t) );

// write type id
dl_binary_writer_write( &store_ctx->writer, &sub_type_id, sizeof(uint32_t) );
if DL_CONSTANT_EXPRESSION(sizeof(uintptr_t) == 8)
dl_binary_writer_write_zero( &store_ctx->writer, sizeof(uint32_t) );
}
return DL_ERROR_OK;
}

dl_error_t dl_internal_store_member( dl_ctx_t dl_ctx, const dl_member_desc* member, const uint8_t* instance, CDLBinStoreContext* store_ctx )
{
dl_type_atom_t atom_type = member->AtomType();
dl_type_storage_t storage_type = member->StorageType();

if (storage_type == DL_TYPE_STORAGE_ANY_ARRAY)
atom_type = DL_TYPE_ATOM_ARRAY; // To avoid code duplification and simplify implementation
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

☝️

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you prefer the code duplification here?

switch ( atom_type )
{
case DL_TYPE_ATOM_POD:
Expand All @@ -320,7 +379,7 @@ static dl_error_t dl_internal_store_member( dl_ctx_t dl_ctx, const dl_member_des
const dl_type_desc* sub_type = dl_internal_find_type( dl_ctx, member->type_id );
if( sub_type == 0x0 )
{
dl_log_error( dl_ctx, "Could not find subtype for member %s", dl_internal_member_name( dl_ctx, member ) );
dl_log_error( dl_ctx, "Could not find struct subtype %X for member %s", member->type_id, dl_internal_member_name( dl_ctx, member ) );
return DL_ERROR_TYPE_NOT_FOUND;
}
dl_error_t err = dl_internal_instance_store(dl_ctx, sub_type, instance, store_ctx);
Expand All @@ -331,12 +390,31 @@ static dl_error_t dl_internal_store_member( dl_ctx_t dl_ctx, const dl_member_des
case DL_TYPE_STORAGE_STR:
dl_internal_store_string( instance, store_ctx );
break;
case DL_TYPE_STORAGE_ANY_POINTER:
{
dl_typeid_t tid = *reinterpret_cast<const dl_typeid_t*>( instance + sizeof( void* ) );
const dl_type_desc* sub_type = nullptr;
if( *reinterpret_cast<const void* const*>( instance ) )
{
sub_type = dl_internal_find_type( dl_ctx, tid );
if( sub_type == 0x0 )
{
dl_log_error( dl_ctx, "Could not find any_pointer subtype %X for member %s", tid, dl_internal_member_name( dl_ctx, member ) );
return DL_ERROR_TYPE_NOT_FOUND;
}
}
dl_error_t err = dl_internal_store_ptr( dl_ctx, instance, sub_type, store_ctx );
if( DL_ERROR_OK != err )
return err;
dl_binary_writer_write( &store_ctx->writer, &tid, sizeof(dl_typeid_t) );
}
break;
case DL_TYPE_STORAGE_PTR:
{
const dl_type_desc* sub_type = dl_internal_find_type( dl_ctx, member->type_id );
if( sub_type == 0x0 )
{
dl_log_error( dl_ctx, "Could not find subtype for member %s", dl_internal_member_name( dl_ctx, member ) );
dl_log_error( dl_ctx, "Could not find pointer subtype %X for member %s", member->type_id, dl_internal_member_name( dl_ctx, member ) );
return DL_ERROR_TYPE_NOT_FOUND;
}
dl_error_t err = dl_internal_store_ptr( dl_ctx, instance, sub_type, store_ctx );
Expand Down Expand Up @@ -366,7 +444,7 @@ static dl_error_t dl_internal_store_member( dl_ctx_t dl_ctx, const dl_member_des
return DL_ERROR_TYPE_NOT_FOUND;
}
}
else if( storage_type != DL_TYPE_STORAGE_STR )
else if( storage_type != DL_TYPE_STORAGE_STR && storage_type != DL_TYPE_STORAGE_ANY_POINTER)
count = member->size[DL_PTR_SIZE_HOST];

return dl_internal_store_array( dl_ctx, storage_type, sub_type, instance, count, 1, store_ctx );
Expand All @@ -377,7 +455,7 @@ static dl_error_t dl_internal_store_member( dl_ctx_t dl_ctx, const dl_member_des
uintptr_t size = 0;
const dl_type_desc* sub_type = 0x0;

uint8_t* data_ptr = instance;
const uint8_t* data_ptr = instance;
uint32_t count = *(uint32_t*)( data_ptr + sizeof(void*) );

uintptr_t offset = 0;
Expand All @@ -396,6 +474,45 @@ static dl_error_t dl_internal_store_member( dl_ctx_t dl_ctx, const dl_member_des
size = dl_internal_align_up( sub_type->size[DL_PTR_SIZE_HOST], sub_type->alignment[DL_PTR_SIZE_HOST] );
dl_binary_writer_align( &store_ctx->writer, sub_type->alignment[DL_PTR_SIZE_HOST] );
break;
case DL_TYPE_STORAGE_ANY_ARRAY:
storage_type = DL_TYPE_STORAGE_STRUCT;
if (member->AtomType() == DL_TYPE_ATOM_POD)
{
dl_typeid_t type_id = *(dl_typeid_t*)( data_ptr + 2 * sizeof(void*) );
sub_type = dl_internal_find_type( dl_ctx, type_id );
size = dl_internal_align_up( sub_type->size[DL_PTR_SIZE_HOST], sub_type->alignment[DL_PTR_SIZE_HOST] );
dl_binary_writer_align( &store_ctx->writer, sub_type->alignment[DL_PTR_SIZE_HOST] );
}
else if( member->AtomType() == DL_TYPE_ATOM_INLINE_ARRAY )
{
dl_binary_writer_seek_set( &store_ctx->writer, pos );
return dl_internal_store_anyarray( dl_ctx, data_ptr, member->inline_array_cnt(), store_ctx );
}
else if( member->AtomType() == DL_TYPE_ATOM_ARRAY )
{
dl_binary_writer_align( &store_ctx->writer, sizeof(void*) );
offset = dl_binary_writer_tell( &store_ctx->writer );
dl_binary_writer_reserve( &store_ctx->writer, count * 3 * sizeof(void*) ); // reserve space for array so subdata is placed correctly
dl_error_t err = dl_internal_store_anyarray( dl_ctx, *(const uint8_t* const*)data_ptr, count, store_ctx );
if (DL_ERROR_OK != err) return err;

dl_binary_writer_seek_set( &store_ctx->writer, pos );

if( !store_ctx->writer.dummy )
store_ctx->ptrs.Add( dl_binary_writer_tell( &store_ctx->writer ) );

// make room for ptr
dl_binary_writer_write( &store_ctx->writer, &offset, sizeof(uintptr_t) );

// write count
dl_binary_writer_write( &store_ctx->writer, &count, sizeof(uint32_t) );
if DL_CONSTANT_EXPRESSION(sizeof(uintptr_t) == 8)
dl_binary_writer_write_zero( &store_ctx->writer, sizeof(uint32_t) );
return DL_ERROR_OK;
}
else
DL_ASSERT(false && "Invalid ATOM-type!");
break;
case DL_TYPE_STORAGE_PTR:
sub_type = dl_internal_find_type( dl_ctx, member->type_id );
/*fallthrough*/
Expand All @@ -417,14 +534,24 @@ static dl_error_t dl_internal_store_member( dl_ctx_t dl_ctx, const dl_member_des
dl_binary_writer_seek_set( &store_ctx->writer, pos );

if( !store_ctx->writer.dummy )
store_ctx->ptrs.Add( dl_binary_writer_tell( &store_ctx->writer ) );
store_ctx->ptrs.Add( dl_binary_writer_tell( &store_ctx->writer ) );
}

// make room for ptr
dl_binary_writer_write( &store_ctx->writer, &offset, sizeof(uintptr_t) );

// write count
dl_binary_writer_write( &store_ctx->writer, &count, sizeof(uint32_t) );
if DL_CONSTANT_EXPRESSION(sizeof(uintptr_t) == 8)
dl_binary_writer_write_zero( &store_ctx->writer, sizeof(uint32_t) );

if (member->StorageType() == DL_TYPE_STORAGE_ANY_ARRAY)
{
// write type id
dl_binary_writer_write( &store_ctx->writer, (data_ptr + 2 * sizeof(void*)), sizeof(uint32_t) );
if DL_CONSTANT_EXPRESSION(sizeof(uintptr_t) == 8)
dl_binary_writer_write_zero( &store_ctx->writer, sizeof(uint32_t) );
}
}
return DL_ERROR_OK;

Expand All @@ -440,7 +567,7 @@ static dl_error_t dl_internal_store_member( dl_ctx_t dl_ctx, const dl_member_des
return DL_ERROR_OK;
}

static dl_error_t dl_internal_instance_store( dl_ctx_t dl_ctx, const dl_type_desc* type, uint8_t* instance, CDLBinStoreContext* store_ctx )
static dl_error_t dl_internal_instance_store( dl_ctx_t dl_ctx, const dl_type_desc* type, const uint8_t* instance, CDLBinStoreContext* store_ctx )
{
bool last_was_bitfield = false;

Expand Down Expand Up @@ -519,9 +646,9 @@ dl_error_t dl_instance_store( dl_ctx_t dl_ctx, dl_typeid_t type_id,
dl_binary_writer_update_needed_size( &store_context.writer );

dl_binary_writer_reserve( &store_context.writer, type->size[DL_PTR_SIZE_HOST] );
store_context.AddWrittenPtr( instance, header_plus_alignment ); // if pointer refers to root-node, it can be found at offset "sizeof(dl_data_header)" plus alignment
store_context.AddWrittenPtr( (const uint8_t*)instance, header_plus_alignment ); // if pointer refers to root-node, it can be found at offset "sizeof(dl_data_header)" plus alignment

dl_error_t err = dl_internal_instance_store( dl_ctx, type, (uint8_t*)instance, &store_context );
dl_error_t err = dl_internal_instance_store( dl_ctx, type, (const uint8_t*)instance, &store_context );

// write instance size!
dl_binary_writer_seek_end( &store_context.writer );
Expand Down
Loading