How can I block accessing my site with https + IP address?

I have a wordpress site made in amazon lightsail, let's say it "website.com". And I don't want my site to be accessible from static IP. So I edited httpd-vhosts.conf like this:

<VirtualHost \*:80>
ServerName XXX.XXX.XXX.XXX
<Location />
Order deny,allow
Deny from all
</Location>
</VirtualHost>

But this doesn't block access when I visit site with https://XXX.XXX.XXX.XXX. What should I do to block access when I visit site with https and IP address too?