Tring to answer to the appropriate comment of Natan, I did some consideration that can be useful to understand what really happens when you use vectors in Affine Space to represent 3D vectors in the standard Euclidean Space.
First I will call vector whatever has coordinates, so a point and a vector are the same entity; you can see a vector as difference of two point: V = B - A; V moves
A in B because A + V = A + B - A = B. Put A = 0 (the origin) and you will get that V = B - 0 = B: the point B and the vector that moves 0 to B are the same thing.
I will call "vector" - in the sense used in the majority of 3D libraries - when a vector of the affine space has w = 0.
The matrix are used because they let you to represent a linear function in a compact/elegant/efficient form, but linear functions has the major disadvantage that can't transform the origin: F(0) = 0 if F wants to be linear (amog other thing such F(λX) = λF(X) and F(A + B) = F(A) + F(B) )
This means that you can not construct a matrix that do a translation since you will never move the 0 vector. Here comes into play the Affine Space. The affine space adds a dimension to the euclidean space so traslantions can be done with scaling and rotations.
The Affine Space is a projective space in the sense that you can construct a equivalence relation between Affine and Euclidean vectors so you can confuse them (as we did with poins and vectors). All the affine vectors that projects to the origin with the same direction can be seen as the same euclidean vector.
This means that all the vectors that have the same proportions in the coordinates can be considered equivalent:
i.e. every affine vector can be reduced to a canon version where w=1 (we choose among every equivalent vector the one we like best).
Visually (2D euclidean - 3D affine):
hence the mean of "projective" space; You should notice that here the euclidean space is 2D (the cyan region)
There is a particular set of affine vectors that can't be put in their canonical version (with ease) the one that lies on the (hyper)plane w=0.
We can show it visually:
what you (should) see is that while w -> 0 then the projected vector into the Euclidean space goes to the infinite but to the infinite in a particular Direction.
Now is clear that adding up two vectors in the projective space can lead to problems when you consider the sum vector as a projected vector in the euclidean space, this appends because you will sum the W components in the affine space and then project them to the euclidean (hyper)plane.
This is why you can sum only "points" to "vectors" because a "vector" will not change the w coordinate of the "point" this is true only for "points" where w =1:
As you see the green point is the one obtained adding the two affine vectors that represent the cyan "point" and the V "vector", but if you apply V to every affine vector in a form different by the canon one, you will obtain a wrong results (the red ""point"").
You see that Affine Space can't be used transparently to describe operation on Euclidean Spaces and the misuse of the term "vector" has sense under the (strict) constraint of compute sums only on canon projective vectors.
Said that, is quite reasonable to think that the GPU assumes that a Vector4 has to have w=0 or w=1, unless you really know what you are doing.