The film starts off with an illicit exchange taking place seemingly in the middle of nowhere, but anyone who has visited the area in question will no doubt immediately recognise it as the road that leads up to Tsing Shan Monastery in Tuen Mun. Back in 1963 it was the area still referred to as Castle Peak and had yet to be develop into Tuen Mun new town and this was a full ten years before Bruce Lee would make the place famous in Enter the Dragon.
Note the background in this lower picture. It shows Castle Peak Bay as it was before the massive reclamation project filled it all in. This whole view is now of high rises. However, you'll be pleased to know that the ornamental gate is still the same. Here is a picture I took of it a few years ago, although I was up the road looking down - the same side the two lower screen grabs were done on.
Later on in the film, one of the main actors returns to the monastery looking for someone. I suspect the scenes were filmed on the same day but the lower screen grabs were from much later in the film.
It's not easy to tell from the black and white pictures but it looks as though the main entrance gate we see above might be the same yellow colour that still adorns its wall. The picture below is also from a few years back.