The subject which studies matrices and vectors is called Linear Algebra, and you can find some good resources here.
Khan Academy
MIT OpenCourseWare
If you're looking for some examples of matrices to experiment with you can find a list here with a number of examples which can help build a gestalt for what's going on.
Rotations - http://en.wikipedia.org/wiki/Rotation_matrix
Axis permutation - http://en.wikipedia.org/wiki/Permutation_matrix
Axis Scaling - http://en.wikipedia.org/wiki/Scaling_(geometry
Shear - http://en.wikipedia.org/wiki/Shear_matrix
Coordinate Systems - http://www.math.hmc.edu/calculus/tutorials/changebasis/
Translation - http://en.wikipedia.org/wiki/Affine_transformation#Representation
For a better understanding of why they use a 4x4 matrix for their transform rather than a 3x3 matrix it helps to understand how vector addition (ie. translation) can be done using a 4x4 matrix and how the extra dimension can be used to assist with the calculation of perspective views. The OpenGL standard for computer graphics which is used by virtually all CAD systems uses the same 4x4 matrix representation so it's no surprise that Sketchup uses them as well. Information on those ideas can be found here.
http://en.wikipedia.org/wiki/Affine_transformation
http://en.wikipedia.org/wiki/3D_projection#Perspective_projection
Also, it's good to know what a "change of basis" represents, as it's a core concept to understanding "local" vs "global" coordinate systems and can both be represented and accounted for using matrices which is really useful when dealing with components and groups.
A good site for general computational geometry and from a computer science standpoint you can check out these course notes. Note that it assumes a certain amount of elementary linear algebra and calculus, but it is geared toward an entry level audience.
http://www.cs.mtu.edu/~shene/COURSES/cs3621/NOTES/
Note that it assumes a certain amount of elementary linear algebra to get started but it's done with a beginners audience in mind and I found it very insightful as to why certain objects are distinct like points vectors and vertices are distinct within the object model as well as why faces, vertices, and edges have no explicit constructors and must be added to the entities collection.
For more advanced study I highly recommend Fundamentals of Computer Graphics which gives a broad survey CAD system topics from linear transformations, interpolation, boolean operations on solids, color theory, rendering, orthogonal and perspective views, and algorithmic considerations on all those operations.
Also, the Sketchup Transform object has a couple quirks compared to "traditional" matrix arithmetic, most notably that for scaling the last entry in the 4x4 matrix, transform.to_a[-1] instead of the 3x3 matrix representing the linear transformation. The entry is called the homogeneous coordinate and it plays an important role in both translation transforms and the transforms related to perspective views. If we have a scale factor of F, then the homogeneous coordinate becomes 1/F. It's important to consider that when working with the matrices directly and when translation the matrices for use in other systems.