Freedom of movement from the EU was one of the biggest factors behind the Brexit vote. A Brexit without a clear end to free movement in its current form is neither possible nor desirable.

But contrary to the scaremongering voiced by Vince Cable and others, ending free movement will not turn Britain into a kind of European North Korea with our young people all sitting miserably at home while the rest of Europe goes to the dance.

In fact, I predict that not very much will change in the general movement of people to and from Europe, the one caveat being that if people from the EU want to work in the UK (and vice versa) they will need a work permit.

As there are about 35million arrivals each year to the UK from EU countries, both the immigration authorities and the tourism industry have a strong interest in retaining visa-free travel for tourism and short visits (and the same is true in the rest of Europe). So don’t throw away the back-pack and railcard.

And in a report published today for the think tank Policy Exchange I further propose that we should offer continuity in arrangements for EU students in terms of fees and access to the government loan system (in the expectation that European colleges will do the same).

The number of undergraduates from the EU is not large, about 25,000 a year, and it would send a helpful signal about the UK wanting to remain the leading European centre for higher education, innovation and research (nearly half of EU students are postgraduates).

I also propose that we should extend the current Youth Mobility Visa that offers two year access to the UK for 18-30 year olds from places like Australia and Taiwan to all EU states, something that should allay the fears of the hospitality sector.

So, a balanced outcome is posseible which allows for gradual reductions in inflows, especially of low skill EU workers, while retaining a high degree of continuity in other areas. We can respond to the legitimate anxiety about over-rapid change and labour market competition while remaining an open, hub economy and country, especially in relation to skilled professionals and students from the EU. 

My report advocates several other steps to achieve this balanced outcome. They include: no automatic right of residence for EU citizens coming in the future; a “light-touch” five year work-permit for future EU professionals to be cleared in less than a month; more restrictive two year permits for unskilled workers with preference for those willing to work anti-social hours; creation of new temporary work programmes including in agriculture and for young people.

There is a bigger opportunity available too. The creaking British growth model has been based in the past generation on easy hire and fire and constant expansion of the labour supply, and it has thus become dependent on high levels of immigration (most from the EU is of low and middling skill level).

Ending free movement in a measured way is just what the low-productivity British economy needs. EU citizens make up about 7% of the UK workforce, rising to 17% in London, and some sectors such as food manufacturing (30%) and London house-building (56%) have become damagingly over-dependent on EU labour. 

These inflows over the last 15 years have underpinned economic growth but at some cost to British people in the bottom half of the income and educational spectrum: some irresponsible employers have sharply cut training budgets, while others have filled entire factories and warehouses with people from eastern Europe. 

And so long as already trained labour was flowing in from abroad there has been insufficient incentive for government and business to sort out our own education and training systems, especially for people at the bottom end of society (many from ethnic minorities). Yet this is key to both higher productivity and to renewing the national social contract with some of our most powerless citizens. We need to focus more on raising “the general competence of society” (as Christopher Lasch put it in his book The Revolt of the Elites) rather than sending everyone to university.

As Chris Bickerton of Cambridge University has written: “Immigration was a big issue in the referendum not because British people are xenophobic but because immigration is at the heart of the British growth model. As a result the UK experiences life in the EU single market through the prism of EU nationals coming to live and work in the UK.

“Other EU states, with different growth models, experience the EU single market in other ways: through violations of the labour code or through high levels of emigration.”

In this country there is a cultural aspect to this too. Freedom of movement has created a “neither one thing nor the other” category of resident: someone who is neither a temporary visitor/guest to a country, such as a tourist, nor someone who is making a permanent commitment to a new country in the manner of the traditional immigrant. The openness of free movement has also made it very hard for local and national authorities to plan for future population growth and infrastructure needs.

Nevertheless apart from that work permit requirement—light touch for skilled, much less so for unskilled—much continuity is possible and even when it comes to jobs and social rights future EU citizens should continue to have some limited special access to the UK labour market and welfare state as a symbol of the “deep and special” relationship the UK is seeking.

Britain will remain a welcoming society to future EU citizens, on top of the three million already here. Ending free movement is, however, a necessary (though not sufficient) condition for healing domestic wounds and nudging us onto a new growth path, ironically one that’s more like the higher-productivity continental economies.

David Goodhart is Head of Demography, Immigration and Integration at Policy Exchange and the author of Immigration After Brexit, published this week