There's a great deal of movement from infant to primary school across the UK, and even small schools with established classes often increase their class size from, say, 30 to 34 (the maximum) in Year 3 (Primary). Just three or four new pupils in a class can dramatically change the dynamic -- and sometimes for the better if it's been, for example, a majority boy cohort.
It's interesting because I speak to a lot of parents about schools and the general consensus seems to be that, while you think it's the end of the world to move schools or make changes when they are little, it really isn't. Big changes have big impact as they get older, and, even then, it depends on the child. So I hear parents say that they thought the entrance to Reception was a huge decision until they got the secondary choices and that REALLY WAS A HUGE DECISION! Sixth form can be even more crucial as it can affect their university or FE choices.
My advice as a parent would be that the cohort is key. If your lb is in a very strong cohort where he is settled and challenged then, yes, it will be a wrench to then move him. You might argue staying in a strong cohort in a less good (on paper) school is better than moving to a better school with a weaker cohort. When I say cohort I mean class group. This can very quite significantly from year to year; with some 'good' years and others less so.
If you can, research via Parent Reviews or school gate conversations what the cohort is likely to be at each option.
Hope that helps!