Solstice musings

Today is the summer solstice, the longest day of the year. It's also referred to as the first day of summer.

Tell me, why is it that after the first day of summer the days start getting shorter, starting the march towards autumn? Why isn't this day known as the middle of the summer? Oh, look - it is also known as Midsummer in some places. That makes much more sense to me.