overwrite malloc/free in dlopen-ed library

Question

I have a shared library - plugin.so, which is dlopen-ed by the host program with flag RTLD_LOCAL, I have my own memory operation functions defined in that library:

void *plugin_malloc(size_t size) { /* ... */ }
void plugin_free(void *ptr) { /* ... */ }

What I need is to replace ALL malloc/free calls in plugin.so with my own plugin_malloc/plugin_free, I tried using GCC's alias attribute extension:

void *malloc(size_t) __attribute__((alias("plugin_malloc"), used))
void free(void*) __attribute__((alias("plugin_free"), used))

However, this only works when the library is linked into the host program, but not work with the dlopen way.

I'm on Linux with compiler GCC-4.8.5, and I have the source code of plugin.so and can modify it as I like, but I cannot modify the host program, and replace malloc/free not only in plugin.so but also in entire program is also acceptable.

So, is there any solution? Thanks.

---

EDIT: I also have no permission to modify the host program's startup arguments, environment variables, what I can do is just providing the plugin.so to guys who own the host program, and they run the host program and dlopen my plugin.so.

Answer 1

Considering you provide a pair of functions named malloc and free (instead of plugin_malloc for example), in a shared library compiled with -fPIC, then you just need to LD_PRELOAD it when calling your client app:

LD_PRELOAD=/path/mymalloc.so executable

or export it before calling your client:

export LD_PRELOAD=/path/mymalloc.so
executable

more details:

• man ld.so
• The LD_PRELOAD trick

[UPDATE]

Considering you cannot change the environment but only replace the dynamic library and nothing else then you may be able to:

• find where malloc/free are in memory
• replace the original functions entry code for a jmp to your own functions

Your library gonna need an initialization function to do this dirty work. Check the constructor attribute here. Replacing the code may not be allowed, though.

Another possibility to explore(oit?): if the code is using glibc you can try to provide a __malloc_hook in your library instead.

Yet another: seize the application control in your library init function never returning from it and then execute the application again with your custom setup.

I cannot come out with any other possibility, for now, considering your constraints.

Answer 2

What I need is to replace ALL malloc/free calls in plugin.so with my own plugin_malloc/plugin_free,

This is trivial to do.

Suppose you have foo.o and plugin_malloc.o that are linked into plugin.so. The foo.o uses malloc and free, while plugin.o defines plugin_malloc and plugin_free.

Then:

objcopy --redefine-sym malloc=plugin_malloc --redefine-sym free=plugin_free foo.o foo2.o
gcc -shared -fPIC plugin.o foo2.o -o plugin.so

Voila: all references to malloc and free in foo.o have been replaced. Enjoy.

Update:

if I call a glibc function which allocates memory and needs to be freed in my code, the program crashes. e.g. char *s = strdup("hello"); free(s); because strdup calls glibc's malloc, but the later free is my plugin_free

There are a few ways, only some of which satisfy your other constraints:

1. you must replace all mallocs and frees with your own for the entire program (e.g. via LD_PRELOAD, or by statically linking malloc implementation into the main executable), or
2. you must ensure that you don't call any function that allocates memory via malloc that you expect to free later, or
3. for any such calls to stdup or asprintf etc., when you want to deallocate this memory, you must call __libc_free rather than plugin_free, or
4. in plugin_malloc, pad all memory you allocate with 16 extra byte header (for alignment) write magic number to the start of the block, and return a pointer past the header to the caller. In plugin_free, check for proper alignment, then check the header for magic number. If it's there, subtract 16 from the pointer and use the rest of plugin_free to free the memory. If the magic number is not there, assume the pointer didn't come from plugin_malloc, and call __libc_free on it instead.
5. In plugin_malloc keep track of all the blocks you've ever allocated. In plugin_free check that list, and use __libc_free if the pointer is not on the list.

Since you have all sources for your plugin, either solution 2 or 3 should be doable, but obviously requires that you audit every call to free to see where the memory came from.

Variants 4 and 5 don't require such audit, but don't cleanly separate memory allocated by plugin from that allocated by the main program.

Don't forget about other ways to allocate memory: realloc, memalign, posix_memalign, etc.