The most annoying world in Super Mario Bros. 3 is hands-down world three. Plumbers apparently are terrible swimmers, and navigating underwater is only marginally less frustrating than killing off all four ninja turtles in two minutes. The designers felt a frog suit would be a nice bone to toss at gamers, but it’s barely an improvement underwater, and a nightmare when back on land.
So after surviving World 3, I was hoping for something a little breezier, and drier.
Death By Troggles 
God, I loved giant world. As I remember, though, it’s easy to skip with a warp zone. I don’t remember 3 as much as most people, though, as it was probably my least favorite of the series (I know…I’m that guy).
I remember the first few times I encountered enemies there. My brother and I laughed like hell.
I never skipped over this world because I also loved it. I also decided at some point in my Mario gaming days that it was more fun to try to go through the entire game rather than just warp past everything.
Your memory is correct. Though, it’s fairly easy to skip the entire game if you get all the whistles from levels 1 and 2.
I remember going through that door that turned everything small, deciding that it sucked, and turning everything big again.