You should be ok to swap out the mame files, the only thing that might happen is when we make a new update, AGS may overwrite your custom build.
We're planning on adding features to allow power users to customize more without AGS borking it in every update, but for now I'd recommend keeping a backup of your custom build.
Also, you may be interested in an "adding emulators" post that another user made - check it out!