Ah, the Japanese and English...Japan is probably the only place in the world where the people can accurately write an English sentence in the present perfect tense, but cannot understand an American when he asks in English: "Which train goes to Kyoto?"
And no one is more aware of this irony than the Japanese people themselves, because they slavishly studied the English language for years in school; when push comes to shove, they really can't speak or understand English in any practical way.
Some of this may be the fault of the written Japanese language itself, which is composed of Chinese characters (kanji) along with two kinds of Japanese characters to accommodate foreign words, concepts, etc. The way that Japanese young children learn to read and write Japanese is by methodically memorizing large groups of characters until they have a working vocabulary.
In many ways it's this method of learning the Japanese language that has informed the way that the Japanese people learn English.
Japanese people (who do not speak English) are largely responsible for teaching young Japanese English. And they do this by emphasizing grammar almost to the complete exclusion of speaking. For as one wise man once said, "you can't pass on what you don't have. "
Exposure to "spoken English" might be from JET program teachers, who are largely American ESL teachers employed by the state to swoop into junior high classes and have fun with the students using English. It's all fun and games, but Japanese young people are mostly concerned with passing their high school and university entrance exams, which require an expertise grammar knowledge.
All and all, the Japanese people are slightly uncomfortable in situations where they might be expected to use English with a foreigner. Perhaps they feel they were short changed when it came to their adolescent English lessons, or perhaps it's more about Japanese "shame culture," where losing face is a dreaded prospect.
In the end, the best bet when you're lost somewhere in Japan is to bumble through using your travel book Japanese, and hope that a sympathetic Japanese person realizes that their unused English is positively Shakespearean compared to your mangled Japanese.