NEW YORK—Wall Street turned in a mixed performance Monday as investors watched the price of oil regain ground and decided to cash in some of their gains from the stock market’s big rally last week.

While the market’s major indexes showed modest losses, the number of stocks advancing outpaced decliners by about 2 to 1 on the New York Mercantile Exchange, and by about 4 to 3 on the Nasdaq Stock Market.

The tame session unfolded as oil rose on concerns that the threat of new sanctions against Iran over its nuclear program could escalate tensions in the Middle East. Light, ...