Generally speaking you can dismiss an employee up until they have worked for you for 12 months and they cannot challenge it as unfair dismissal. You need to ensure that the dismissal is not discriminatory ie based on race, sex etc.
There are also circumstances in which a dismissal would be regarded as automatically unfair. These include, for example, dismissing an employee for seeking to be a member of a union, refusing to be a member of a union etc.
The post I've linked below lists all of these types of reasons (from Business Link).
Automatically Unfair Dismissal
I'm not a lawyer by any means but in the situation where this employee has worked for you for only 3 weeks I think you could go either route. If you do the performance management route then keep records and follow the exact procedures that would apply if they had worked for you for over 12 months. The other obvious thing is to not allow it to continue beyond where they have worked for you for 12 months!
If you want to go the boot 'em out the door route then I think all you need say is words to the effect. 'I'm afraid this is not working out and we will not be continuing your employment'. To keep it watertight give them the statutory minimum notice - either (probably) paying them in lieu or allowing them to work out their notice.
From what I can gather an employee of 3 weeks without an employment contract/statement has no statutory notice period! After one month's work they have a statutory right to one week's notice.