Besides smooth woven fabric, you also have the option of wearing more casual knit shirts. These can be polos (in short or long sleeves), henleys, quarter zips and turtlenecks. All of these look fine ...