Preventing strangers from registering using custom question

I have set up a site for sharing stories and pictures from family holidays. I want only people that know me to be able to register. I want to avoid strangers registering. I am trying to do this using code snippets by adding a custom field to the registration form. I ask a question and if the answer during registration doesn’t match my desired answer I give an error. I assumed that the user would then not be created. However, I can see that if I answer wrongly, even though an error is correctly returned, the user is still created. I want it so that if the user fails to answer the question correctly, they get an error message and no user is created.

What am I doing wrong? This is the code I use:

add_action( 'register_form', 'add_register_field' );
function add_register_field() { ?>
	<p>
		<label><?php _e('<hr><br />To keep strangers away, prove you know me by answering this:<br /><br />Where do I live?') ?><br />
		<input type="text" name="user_proof" id="user_proof" class="input" size="25" tabindex="20" /></label>
	</p>
<?php }

add_filter( 'registration_errors', 'add_register_field_validate', 10, 3 );
function add_register_field_validate( $errors, $sanitized_user_login, $user_email ) {
	if (!isset($_POST[ 'user_proof' ]) || empty($_POST[ 'user_proof' ])) {
		$errors->add( 'proofempty', '<strong>Error</strong>: You must answer where I live'  );
	} elseif ( strtolower( $_POST[ 'user_proof' ] ) != 'dagobah' ) {
		$errors->add( 'prooffail', '<strong>Error</strong>: You can not register, if you do not know where I live'  );
	}
	return $errors;
}