I have particle system, and I want to the individual particles to take the shape of a star. Which should be faster: Drawing the 10 polygons using a flat shader, or 2 polygons (a square) with a partially transparent texture?
10 flat shaded polygons is pretty much guaranteed to be faster than the 2 transparent textures from a GPU pov. Especially true when you don't have unified shader architecture, as you'll almost always be bottlenecked by pixel shader throughput for particle systems. Unless you're doing anything fancy, your vertex shader in this case is likely fewer cycles than a single pixel, and that doesn't even account for the alpha blend.
Two polygons with a texture is faster.
Because it is the typical way to do things. If you're working in 2D then you can just use
You'll also be saving your artists time. Making a texture is much easier than making a model, even a simple one.
... oh you meant runtime performance ...
I'm just going to chime in with my POV on this.
As you mentioned in a comment, there is a 4-triangle star,
Now the question is whether to use this 12-vertex, 4-face star (I wouldn't use an index buffer because that would only be a savings of 2 vertices/prim the way I've constructed it here), or to use a square polygon with 2 triangles and use a texture on top.
I'd bet your horses that untextured polys will give you much better performance. But going with the textured quad is probably a better idea for the reason of flexibility alone: Using a texture gives you additional flexibility to change your mind about the shape of the object, with no runtime performance change, if you were to change the texture.
You could also use your same particle engine with different textures to achieve different effects. For example, the traditional "ball-shaped" particle will plug into your texture-based particle engine quite easily without requiring you to make many changes. So using a texture makes your particle engine easier to reuse in that way.
As Sam said, the triangle shown above infamous T-junctions. Look at the big triangle in the red-star. If you shade the leftmost vertex blue and the rightmost vertex green, what color could you shade the left/right vertices of the TOP triangle of the star? The gradient will look very wrong at the T-junctions, because you'd have to guess the interpolated color at the T-junctions, and that's not going to look good.
To eliminate T-junctions, you only need 8 triangles: