To flip the overlay horizontally, put a negative sign in front of the first scale size:
@overlay OVERLAYNAME scales to -1.0 1.0
To flip the overlay upside down, put a negative sign in front of the second scale size:
@overlay OVERLAYNAME scales to 1.0 -1.0