Reasons are
1. scanf("%c%[^\n]",&a); needs two parameters. Remove %[^\n].
2. \n character left behind by the previous scanf on pressing Enter key. Next scanf will read this \n character in the buffer. You need to consume this \n. Use a space before %c specifier to eat up this \n.
Try this:
scanf(" %c",&a);
↑ A space before %c specifier
A space before %c specifier is able to eat ant number of newline characters.
Your code after modification:
#include<stdio.h>
int main(void)
{
char a;
int n;
do
{
printf("enter the number\n");
scanf("%d",&n);
printf("the squre is %d\n",n*n);
printf("want any more so Y for yes N for no\n");
scanf(" %c",&a);
}while(a=='Y');
return 0;
}