For a game, your primary goal is to meet the target framerate on the target minimum spec machine (and possibly a maximum load time, etc).
In order to do that, no, you do not have to worry after every line.
You have to worry early whenever you select a particular strategy, algorithm, or container. If you make it literally impossible to meet the target by making inadequate design decisions, then any optimization that you may do later is to no avail.
Next, you have to worry whenever something is parallel or parallelizable. Games are massively parallel, if for no other reason, then because graphics are.
Therefore, parallel doesn't just mean "threads" but also for example the graphics API, disk access, or network. Whenever you miss the opportunity of having something that could easily and natively be parallelized run in parallel, for example due to synchronizing badly (or due to not using an asynchronous API at all), you lose more than you can ever optimize by other means.
You also have to worry whenever something is well-known to be a bottleneck or a source of stalls, or a hindrance to scaling. Such as, for example, switching render states, draw calls, reading back from the GPU, or opening files.
Last, when you are done, and testing shows that you do not meet the target frame rate, you need to optimize. Find the one biggest bottleneck that takes 90% of the time, and optimize that. If this is not enough, find the seconds biggest.
If you do meet the target, congratulations. Move on and forget it.