I've heard mentioned that you should try to include at least one angle in the design to eliminate the possibility of standing waves inside the enclosure, but the general consensus around here is that the shape doesn't matter.

Is a bigger subwoofer enclosure better?

By making the enclosure a bit bigger it will become more efficient in many cases, increasing output. ... As the size of the enclosure increases, the mechanical limits of the woofer will be easier to reach. If the box is too small (by a reasonable amount) add power.

Does a sub box have to be square?

It can be round, square, triangle, rectangle, or any other shape as long as the total area is equivalent to what is required for that enclosure.