This morning in Vienna, David Davis brushed off working people’s fears that their rights at work will be undermined after Brexit. He insists that Britain will be leading a “race to the top” on standards. But he’s done nothing to set my mind at ease.

For one thing, Davis’s promise that he’s not building a “Mad Max-style dystopia” says a lot for how low the Brexit bar has been set. And even then, the government might not clear it if the extreme fringe of the cabinet have their way. Too many of Davis’s colleagues are champing at the bit, looking for any opportunity to use Brexit to slash workers’ rights. Just before Christmas, Boris Johnson and Michael Gove briefed the papers on a plan to attack the Working Time Directive, which safeguards paid holidays, rest breaks and reasonable working hours.

Whatever Davis promises, that Johnson felt able to speak out on this worries me. So do his more recent proclamations about regulatory “divergence”, “flexibility” and “agility” – code words for scrapping hard-won rights at work, along with environmental and safety standards.

The foreign secretary is on record as an opponent of paid holiday, parental rights and fair redundancy pay. He’s said that the EU regulations that protect workers are “back-breaking”. Other senior ministers – Liam Fox and Michael Gove among them – have echoed the same views.

And when the EU Withdrawal Bill was going through the House of Commons, the government refused to back amendments that would lock in protections for rights at work.

So trade unions aren’t crying wolf here. We have serious, well-founded concerns about what Brexit will do those rights for workers that came from the EU. And no one in government has given us the guarantees that we need.

Sure, Theresa May has offered warm words. But then, she also promised to put workers on boards before backing down under pressure from her party colleagues. And she talked a good game on tackling insecure work – but her government’s response to the Taylor Review went nowhere near far enough to support struggling workers in the gig economy.

So why should we believe this will be any different? Especially when Davis himself has mooted the possibility of a Canada-style deal with Europe, which wouldn’t provide any security for workers’ rights or any means to enforce them. The Canada-EU deal gives one-sided protections to foreign investors, putting their needs way ahead of the wellbeing of working people.

The British people may have voted for Brexit, but they didn’t vote to lose their rights at work. Just today, new polling shows that 73% of the public want to see the Working Time Directive maintained or even strengthened after Brexit.

So the TUC’s position is clear: Brexit can’t be used as an excuse to cut people’s rights at work, in service of an extreme deregulation agenda. Already we have an imperfect and incomplete patchwork of rights that doesn’t provide all the protections we need. So whatever the Brexit deal, trade unions will keep fighting for more and better rights at work, including a ban on zero-hours contracts and equal rights for agency workers. A decent Brexit deal is the bare minimum, not the be-all and end-all.

That said, hardworking Brits need a guarantee that, after Brexit, their rights at work will keep pace with the rest of Europe. Currently, the best way to provide that guarantee is by sticking with the single market. So if the prime minister and Brexit secretary want us to trust their promises on workers’ rights, they should put all options back on the negotiating table.

It’s the only way to deliver a “race to the top”, and to protect working people’s rights long-term.