I argue that this is actually memorization from the model. After a certain threshold, it looks like magic or emergent behaviour when it's not really.

If we really had emergent behaviours, why does GPT-4 fails a simple test like this:

https://chat.openai.com/share/122bae00-3885-47b9-90b6-fb0c28bc7d41