My brain is furry at the moment (gypsum drywall/cement dust and sleep deprivation), but I don't know any way without experimenting.
As you've figured out, what one gets is the class instance object, not the __init() return value. If the object is not instantiated the 'test' variable is nil (which can be different from false if one explicitly tests for false, I vaguely recall -- default 'if not test' should be true for either nil or false).
Some sort of "delete self on fail" probably would not be processed in time for the object check to show nil in the routine just after the instantiation. Or would it?
You could look into the C++/Lua interface support, maybe query an upvalue or even rewrite the constructor code to permit return values. (I really am tired.)
Maybe passing in a reference as an argument to the constructor and testing that would not be that much of a burden on processing.
The rest of my thinking is nonsense even to me, so I'll just wish you well in your search and go get some rest.