Thanks for reaching out. I am hesitant to add this functionality since the current implementation matches Core panels that support adding additional functionality. The Typography or Dimension panels on most Core blocks are good examples.
That said, you can disable all default controls in the plugin settings.
That way, the Visibility panel will be empty until the user explicitly adds a Visibility condition with the “+” icon.
