Except when it is the operand of the sizeof or unary & operator, or is a string literal being used to initialize another array in a declaration, an expression of type "N-element array of T" is converted ("decays") to an expression of type "pointer to T", and the value of the expression is the address of the first element of the array.
If the declaration of the array being passed is
int S[4][4] = {...};
then when you write
rotateArr( S );
the expression S has type "4-element array of 4-element array of int"; since S is not the operand of the sizeof or unary & operators, it will be converted to an expression of type "pointer to 4-element array of int", or int (*)[4], and this pointer value is what actually gets passed to rotateArr. So your function prototype needs to be one of the following:
T rotateArr( int (*arr)[4] )
or
T rotateArr( int arr[][4] )
or even
T rotateArr( int arr[4][4] )
In the context of a function parameter list, declarations of the form T a[N] and T a[] are interpreted as T *a; all three declare a as a pointer to T.
You're probably wondering why I changed the return type from int to T. As written, you're trying to return a value of type "4-element array of 4-element array of int"; unfortunately, you can't do that. C functions cannot return array types, nor can you assign array types. IOW, you can't write something like:
int a[N], b[N];
...
b = a; // not allowed
a = f(); // not allowed either
Functions can return pointers to arrays, but that's not what you want here. D will cease to exist once the function returns, so any pointer you return will be invalid.
If you want to assign the results of the rotated array to a different array, then you'll have to pass the target array as a parameter to the function:
void rotateArr( int (*dst)[4], int (*src)[4] )
{
...
dst[i][n] = src[n][M - i + 1];
...
}
And call it as
int S[4][4] = {...};
int D[4][4];
rotateArr( D, S );