An interesting historical point about all these books is the way that the names of counties / regions change at regular and rapid intervals, particularly with respect to Scotland! There are also quite a few variations in the spelling of castle names. I have generally used the first variation that I came across.

The Castles of Great Britain - Sidney Toy - Heinemann 1966 (First published 1953)

This covers the complete development of the castle from prehistory to almost the twentieth century, with lots of information on architectural details, ground plans etc. Most official guide books still rely on Toy for their plans if not all the content. Illustrated with black and white photographs.

The Scottish Castle - Stewart Cruden - Nelson 1963 (First published 1960)

Very similar approach to Toy but more specific to Scottish sites. Cruden is particularly strong on the development of the Scottish towerhouse. Illustrated with black and white photographs.

Castles of Britain and Ireland - Plantagenet Somerset Fry - Abbeville Press 1997

Several well-illustrated chapters (with colour photographs) describing the development and history of the castle, followed by a fairly comprehensive gazetteer for each country. (I say "fairly comprehensive" because Fry concentrates on the period from the Norman Conquest to late Medieval. He does not include Henry VIII's artillery forts/castles, for instance.)

The English Castle - François Matarasso - Cassell 1995

Another well-illustrated (with line drawings rather than photographs) history of the subject.

Additional source material for Irish sites:

Ireland - The Complete Guide and Road Atlas - Appletree Press 1989

Great Britain and Ireland - Phaidon Cultural Guides 1986

Ireland - The Rough Guide - Harrap-Columbus 1991

Ireland - Lonely Planet Publications 2000