Michael, the triangle or arrowhead in the left margin appears next to each paragraph that has a heading style, only when the mouse cursor hovers somewhere in that paragraph. Its function is to expand or collapse all the text (and other objects) that follow until the next heading of the same level.
When the arrowhead points down and right, the material after the heading is expanded, and you can click the arrowhead to collapse the material. Then the arrowhead points to the right, and you can click it to expand the material.
What you call an "apostrophe-looking symbol" is really the combination of the expand/collapse arrowhead and a small square. That square appears when the show/hide paragraph mark is turned on and the paragraph has one of these options in the Line and Page Breaks tab of the Paragraph dialog checked: "Keep with next", "Keep lines together", or "Page break before".

All of the "Heading" styles include "Keep with next" and "Keep lines together" as part of their default definitions, so the square will appear there. It will also appear for a paragraph to which you've manually added one of the three options.