Should I take my baby to the doctor?

Okay so Sunday night my baby woke up every 2-3 hours when usually he either sleeps through the night or wakes up only once. Then Monday night I had a hard time getting him to nurse (when usually he loves to nurse) but I did and got him to sleep. He slept through the night. Tuesday night when I tried to get him to sleep he screamed for 2 hours and I noticed he felt hot so I took his temp and it was 101. So I gave him tylenol he calmed down I nursed him to sleep he slept through the night. The next morning I called the doctor and they said they didnt need to see him because he slept through the night. They said bring him in if he has a fever 3 days in a row. So then last night same thing happened he is screaming 101 temp i gave him tylenol he only woke up once last night. He does not run a fever during the day. I think it is an ear infection and hate to wait until he has another night like the last 2. Plus he may have had a fever the other two nights. He is 5 months old tomorrow. The nurse acted like nothing was wrong but I know my baby and this is not normal. I read that ear infections can go away on their own. Should I wait for that? Is it dangerous if he has an untreated ear infection bc I think he has one.