Filter to load plugin only on pages using Add to Any?

Yep, you can use the addtoany_script_disabled filter to programmatically remove all AddToAny JS & CSS.

Just note that this also disables AddToAny ability to load on dynamically inserted content such as with infinite scrolls, etc.

Usage example:

add_action( 'wp_enqueue_scripts', function() {
    // Your conditional goes here. For example:
    // Allow only on single posts (of any post type).
    if ( is_singular() ) return;

    // Remove AddToAny CSS/JS scripts.
    // Note: This disables AddToAny's ability to load dynamically.
    add_filter( 'addtoany_script_disabled', '__return_true' );
});