he didn’t use calculus. Anyway, here’s my idea. My start is exactly the same as andy’s approach. Let the square be ABCD. It is quite clear that we can assume one of the vertices of the triangle is an vertex of the square. (One will need to consider the orientation of the triangle to prove this fact.) Let this vertex be A. Next, for the maximality of its area, the triangle should have at least one more vertex on the side of the square. Let this vertex be E and let this to be on BC.

If angle EAB > 30, then angle DAE < 60, so the remaining vertex must be outside the square. Thus angle EABAE until angle EAB = 15. At this moment, AF = AE. Then AEF is the required triangle and it is easy to find its area.