the easiest way is to write a trait class

Code:
template <class T>
struct grad_type {};

template <>
struct grad_type<Scalar> { typedef Vector type; };

template <>
struct grad_type<Vector> { typedef Tensor type; };

template <class T>
void test(T x) {
    typename grad_type<Y>::type  Y = grad(x)
}
of course, you need to specialize the trait for each possible parameter type. It would be easier if Scalar,Vector,Tensor had some high-level info in them ( like a rank grading ), in this way the trait could compute the grad_type automatically at compile time.

Anyway, I think emulating "auto"/"decltype" is possible in this case via some template machinery like SFINAE, but I wouldn't advise it if you don't know what you're doing ...

moreover, another simple solution could be to refactor the code, if possible; something like

Code:
template <class T,class G>
void test_impl(T& x,G y) {
	// ...
}

template <class T>
void test(T x) {
    test_impl( x, grad(x) );
}