The Hong Kong compound where Mr Yee (Tony Leung) and his wife (Joan Chen) live in Hong Kong is a rather enigmatic place along Victoria Road that is usually referred to as The White House. The location is along Victoria Road at the base of Mount Davis between Kennedy Town and Pok Fu Lam.
Centamap has it marked as part of an old disused battery but in fact it was originally a mess building for the Royal Engineers. During the 1950's it was given over to Special Branch and used as a detention centre for suspected spies. This included it being used during the 1967 riots as a holding place for trouble makers - it was remote enough to hold people without any interference while the authorities waited for the troubles to cool down. More recently there are rumours that it was used as a staging point and debriefing centre for Tiananmen Square protesters who had managed to flee the military crackdown on the Mainland and were spirited away with the help of the HK authorities and other international jurisdictions including the USA (see Operation Yellowbird here).
These days it is currently being redeveloped into an international University campus for MBA teaching. The company responsible for its redevelopment has promised to protect as much of it as possible but I guess we will have to wait and see how it turns out.