Tenant demanding deposit back unfairly?

We have a prospective tenant that failed the referencing due to a CCJ they apparently didn’t know about (we refunded their holding deposit). We liked them and tried to work with them and offer an olive branch through them finding a suitable guarantor.

They accepted and found someone suitable and paid the holding deposit again (through bank transfer). We held the property an extra week after the referencing passed for them to move in on a more convenient date. Then, the night before they were due to move in the guarantor changed their mind and pulled out when viewing our guarantee agreement which states it’s for the full term of the tenancy and not just 6 months only (which we assumed they would’ve known).

They subsequently told us this and pulled out on the morning they were due to move in! Now they are demanding their full deposit money back and stating it was our fault, because of our strict insurance which required a guarantor.

We are almost certain this is unfair on us as we now incur additional costs for the property for several weeks whilst we find a new tenant.

However, now they are saying they will raise a dispute through OpenRent (even though the deposit was never taken through this website and it was paid direct to our bank)

Do we have to give their money back?

Are we breaking the law by not returning it or can we be taken to court like I believe they will threaten?

Many thanks

Pete