Malware analysis is a dark art, and several tools exist to automate the process. They can be as simple as a quick static analysis using MS strings and other such tiny tools to look over an executable, or as complex as a virtual environment that spins up multiple copies of the file and monitors system calls.
But ultimately, if any attempt to execute a file results in “What’s the password?”, you’re not going to get much more than a exit code. There’s no vulnerability here – if there was, you’d likely be upset that password protecting anything does diddly-squat. I’m sure the next call would be to WinRAR (if they even have a phone!) asking why they can be opened up so easily.
Instead of trying to take on the mammoth task of making a vendor change their code, my advice would be to adopt a policy of blocking these files if you’re the paranoid type. It’s far easier to drop unknown, locked out files then attempt to pry them open.