After creating the triangle with the polygon tool, use the rotate tool (press R to access) to click in the triangle.
Drag the teal-colored icon to the pinnacle of the triangle.
Hold ALT/OPTION and click again until the rotate dialog appears.
Type in a number of degrees to rotate (this may take experimentation, depending on the triangle dimensions) and press Copy.
Transform Again (Sequence) by right clicking the triangle or press CTRL/CMND > ALT/OPTION 4.
After a circle of triangles are made, group them and position them on top of the rectangle so that they hang out over the rectangle.
Cut the triangles - CTRL/CMND X or Edit menu > Cut. Then select the rectangle and select the Edit menu > Paste Into.