If you value your time I highly recommend buying this software. No it is not free, no it is not open-source, but I don't care. I paid 36 euro (48 dollar) including taxes. Fortunately, there is such a product for Mac OS X, it's called Mac Blu-ray Player and it's a paid application. I got so fed-up with this process that I started searching for any product that just allows me to point it to a folder with Blu-ray content and friggin' play it. select the biggest m2ts file and open it in the appropriate player.
To play a Blu-ray movie, you have to perform these steps: I've been using Plex, XBMC and VLC but these free open-source products are all a usability nightmare. I find playing a Blu-ray movie on my Mac cumbersome.