Inverse of a Matrix: Efficient Code (timewise)

[Lindley]

Usually the fastest approach involves some form of matrix decomposition. LU, SVD, or Cholesky decompositions are common.

As noted, these are probably overkill for small matrices.