POSIX requirements to ulimit(UL_SETFSIZE) are as follows:

  UL_SETFSIZE
    Set the file size limit for output operations
    of the process to the value of the second argument,
    taken as a long, multiplied by 512.
    ...
    The return value shall be the integer part of
    the new file size limit divided by 512.


glibc implementation of this function for Linux 
libc/sysdeps/unix/sysv/linux/ulimit.c
returns just results of __setrlimit() call, which is 0 in the case of success.

The code demonstrating the problem is below.

#include <ulimit.h>
#include <stdio.h>
	
int main( int argc, char ** argv ) 
{
	long res;
	res=ulimit(UL_GETFSIZE); 
	printf("Result of ulimit(UL_GETFSIZE): %ld\n", res);

	res=ulimit(UL_SETFSIZE, 10000); 
	printf("Result of ulimit(UL_SETFSIZE, 10000): %ld\n", res);

	res=ulimit(UL_GETFSIZE); 
	printf("Result of ulimit(UL_GETFSIZE): %ld\n", res);

	return 0;
}