After a bit of playing about, I found the posts_where filter a bit too intrusive for what I wanted to do, so I came up with an alternative. As part of the 'save_post' action that I attached for my custom post type, I added the following logic;
$visibility = isset($_POST['visibility']) ? $_POST['visibility'] : '';
$protected = get_option('__protected_posts', array());
if ($visibility === 'password' && !in_array($post->ID, $protected)) {
array_push($protected, $post->ID);
}
if ($visibility === 'public' && in_array($post->ID, $protected)) {
$i = array_search($post->ID, $protected);
unset($protected[$i]);
}
update_option('__protected_posts', $protected);
What this does is hold an array of post id's in the options table where the post is protected by a password. Then in a custom query I simply passed this array as part of the `post__not_in` option e.g.
$query = new WP_Query(array(
'post_type' => 'my_custom_post_type',
'post__not_in' => get_option('__protected_posts'),
));
This way I could exclude the protected posts from an archive page but still allow a user to land on the password protected page to enter the password.