After your hints and some further investigation, I believe I've figured out why my two systems behave differently. It turns out that either allow_execmem or allow_execstack is enough for firefox to run. Since the denial was for execmem, I didn't investigate allow_execstack at first. But if I turn off both on the fresh install, I trigger the problem there too. Both were disabled on the system I upgraded.
Dominick Grift:
You can change the context of the firefox executable to execmem_exec_t
It works, and it sounds like the least intrusive change. I still have the protection on the rest of the system. I'll make a bugzilla asking if that maybe would be the default. (I guess firefox is one of the important targets for attacks though. So having to do this looses a bit of protection.)
drago01:
Its the JS JIT, pre firefox4 it was only available on i686 starting with firefox4 it works on x86_64 too.
Ah! That explains why this started to happen after the upgrade.
Dominick Grift:
Strange, as i never noticed this issues on any of my x86_64 systems
Are you running with default settings? Unless I'm mistaken, the default is for both allow_execmem and allow_execstack to be enabled, and the problem won't appear.
It is possible to silently deny this access
This is not just about an annoying alert. The denial does prevent firefox from running. Firefox crashes when it happens.