Normal body temperature is between 36.5°c - 37.5°c with a digital thermometer irrespective of what you feel with your hands. But if above, it's fever. You can tepid sponge and give paracetamol and if it persists, take baby to a paediatrician.
The brain does a lot of work so their is a lot of blood flow to the head which can make it a little warm to touch.
However, that does not mean fever or sickness. As long as the temperature in your thermometer reads normal, their is really no need to worry.