I think it's because you associate it to bad things, numbers aren't evil or good, it's about how they are used.
I don't trust dreams that much, once I had a vision about 5+6=10, it sounded so real, the emotion was like I found real occult stuff and secrets. Then I woke up, thought about that...