Cracks in the Logic of LLMs