I think the intimidation factor may arise when you start dealing with more complicated operations such as normalization, dot and cross products, and using multiple coordinate systems with matrices to transform between them. These are not necessarily easy to understand at first, even if you have a strong geometry and algebra background.
Also, at least in the US, people who have gone through the typical high-school math sequence are accustomed to thinking about geometry in terms of lines, slopes, angles, etc. They have to unlearn that stuff to some extent, and learn to think about it in terms of vectors and matrices instead. It's not that the concepts of linear algebra are such a stretch, but that they are a somewhat different set of concepts than the ones used in classical geometry, which people are likely to have learned in school.
BTW, the distinction between vectors and points lies in the operations you can perform on them. Although both are represented (in a particular coordinate system) by a list of components, and therefore look "the same", the operations allowed are not the same. For instance, you can add two vectors, or multiply a vector by a scalar. You can't do that with points - or at least, it doesn't make any sense to do so. But you can subtract two points, and the result is a vector from one point to the other. You can also add a point to a vector to get a new point.
Points and vectors also behave differently with respect to transformations. Namely, points are subject to translation, while vectors are not. Consider the example of an object moving with a position (point) and a velocity (vector); if you translate the object to a different place, you alter its position, but not its velocity.
In fact, furthering this line of reasoning, there are not just vectors; there are other entities like covectors and bivectors, which may also "look like" a vector in terms of having a list of components in a coordinate system, but which behave differently in terms of the operations available and the way they react to transformations. These all belong to a field of math called Grassmann algebra. Beyond that, one can be even more general and consider tensor algebra. This is advanced stuff though.