Limitation on the size of dynamic allocations

[devreal]

The overall size of memory allocatable in DART-MPI through dart_memalloc is currently limited to 16MB. The limit is imposed by the size of the window backing these dynamic allocations, which is allocated during initialization and is then chunked up using the infamous buddy allocator. Although no one could have expected anyone to ever require more than these 16MB we have come to witness the unexpected: @tuxamito requires larger chunks of dynamic memory for testing his sparse matrix implementation.

I see two possible ways to deal with this:

1. Get rid of the buddy allocator and use dynamic windows to which malloced memory is attached. To me this seems to be the cleaner solution but it has two major downsides: we lose the ability to use shared memory optimization on these allocations and (more importantly) have to rely on dynamic windows, which (as we have seen) may have a significant performance impact (mainly due to the lack of registration of the memory with the network device and thus inhibits the use of RDMA capabilities of some modern interconnects).
2. Increase the size of the backing window and make it configurable through an environment variable. This seems like a less clean fix but it retains our ability to use RDMA capabilities.

While I would like to favor point 1 I am concerned about the performance impact, a sacrifice we should not make easily.

Any input is appreciated. I think we should

[fmoessbauer]

Can we implement it in a way that the user instantiates an allocator with a size of his choice (backed by one window) and then he can allocate using this window?

See this example:

alloc_t * dart_allocator;
// collective operation
dart_alloc_init(dart_allocator, 1024*1024); // size in kb => 1 GB
void * mem =dart_memalloc(sizeof(int)*8, dart_allocator);
// ...
// release all allocations (probably _finalize)
dart_alloc_release(dart_allocator);

[devreal]

@fmoessbauer I missed your post earlier, sorry. That would certainly be an option as well, yes. I like that better than environment variables 👍