Put the filter after the zooms.
EXT. CAMPUS GARDEN NIGHT
@LIV2 spot 0.963 215 141 in zone 2
@LIV2 changes into OLIVEflashback2
@YOUNGLUCAS spot 1.007 167 136 in zone 2
@YOUNGLUCAS moves to layer 5
@LIV2 moves to layer 4
&cut to zone 2
&zoom on 505 273 to 638% in 0
@set hsl 5 -50 8 colorize with blendMode NORMAL to 100%
(Dialogue and character movements go here)
EXT. MANOR GARDENS DAY
@LUCAS spot 0.635 33 293 in zone 1
@OLIVE spot 0.599 47 299 in zone 1
&cut to zone 1
&zoom on 12 311 to 372% in 0
@reset hsl