No reason you couldn’t build it flat on a dirt floor, it just takes time.
The big issue is that with casters, everytime you move it, you will have to dial it in again for the best flatness. Ideally you would have it on stationary feet and bring the work to the table.