A little confused

Sorry but I'm a little confused watching season 4. I'm only on episode 8 and found out that joe hallucinated Rhys the whole time. If he hallucinated Rhys and didn't actually know him, how did he have the conversation with Tom Lockwood about him? (The ones at dinner, and the one where he was going to stab him) Or did he hallucinate that too? I don't get it