I am trying to recreate a site’s filterable gallery [MIC]. I know it uses fancybox.js but I can’t tell what else is running it.

It’s basically using ajax requests to do a WP_Query using the data supplied from the tick boxes to add as a tax_query..
If your going to do it yourself, you’ll need to brush up a bit of JS to send the data to a PHP function that will return it to the JS to be re-rendered.