That means the ferry is actually hiding the public pier and the end of Public Square Street that terminated at the water. The Govt archive has a direct comparison (albeit the following picture is from 1976 - 17 years later - but you can still see the white building on the right). If you look below you can see the same three blocks and the white Police Station building at the top right of the coastline.
Trying to compare the locale with modern day Streetview pictures is an exercise in the futile because it has changed so much. The typhoon shelter is all but gone now - filled in to create the West Kowloon Reclamation. Ferry Street is about the closest we can get to the old coastline (I suppose you can guess where it got its name from) and you can see another brief glimpse of this same area in The World of Suzie Wong (filmed a year later). However, here are some screencaps that show you what it used to be like with some modern comparison to look at.
The space we see behind Noel Purcell (top picture - he played the ship's engineer) was filled up by the Ferry Point Estate (building commenced in the early 60's).
As you can see the old public pier used in the film to stand in for the HK & Macau ferry pier, has gone the way of the rest of the typhoon shelter. Believe it or not, the coastline here - already moved west by reclamation by 1959 - is no longer within sight (it's actually about 800 metres further out). The view looking back the other way has also changed a fair amount too. The old blocks replaced by the previously mentioned Prosperous Garden estate. The YMT Police Station being pretty much the only surviving relic from the period.
The old spot occupied by the ferry on film now sits under the West Kowloon Corridor (that's the flyover you can see above). I know this is all about progress but it's comparisons like these that make me wish Hong Kong had a little less concrete and a little more open space.
I'll leave you with the best illustration of how development has affected this area. Pictures say a thousand words after all.