While you're creating the rectangle, or immediately after creating it, use the arrow keys - up & down - to change the curvature.
There's no way to make rounded-corner polygons other than drawing them by hand as Richard suggests.
There's more info here: https://help.sketchup.com/en/article/3000199
-m