The religious leaders wanted to arrest Jesus because they realized he was telling the story against them—they were the wicked farmers. But they were afraid of the crowd, so they left him and went away.