It is possible to make circular PCB outlines, using SHIFT + SPACE to change corner mode while re-defining board shape.
Another option is to draw your outline using lines, arcs, circles; select the drawn objects; use Home | Board Shape > Define From Selected Objects to create the board outline.