NDP and Green voters are not the people the Conservatives are looking to. There is a group of people in the centre who sometimes vote Liberal and sometimes Conservative, depending on leaders, policies and situations. Those are the people that gave Harper his majorities, and which gave Trudeau his majority. I think it unlikely they would completely abandon Trudeau so soon but it wouldn't take a lot of them shifting to Scheer to drop the Liberals to a minority government, especially if the NDP gets its act together.