Well, again if I'm right about Rory getting killed at the end, that line may well have been an important way of setting up that situation. It'll now feel plausible if that's exactly why he dies.

Then again, it certainly isn't very Moffatish to go killing off characters, especially ones we've got to know over the course of an entire season. It'll be quite a change of form if Rory does get killed.