How he chose those designs seems to be simple (from what I've read): user experience. He wanted to make products people wanted to use. If there was something clunky, or if something had a rough edge, he'd hammer away at his staff until it was smooth, like the corners on an iPod.
Look at the typical MP3 player, certainly before the iPod, and it had loads of small, fiddly buttons with complex, multi-layered menus that you'd get hideously lost in. The first iPod still had buttons, but laid out logically for control similar to how the, now famous, click-wheel would work on later models. The menus were easy to read, the controls simple to operate. Once they found the method they didn't change much, another benefit over most manufacturers, who seem to change the way their hardware works completely with every release.
To load music onto the iPod you still had to go the old route of finding your CD, ripping it onto the computer and then copying it over to the device. The sync functionality in the associated software was designed to get around some of the mess in that process (finding the tracks you've ripped and copying them over). iTunes was the solution to the rest, instead of having to go out, buy the CD and rip it into the correct format, you just went onto iTunes, did a quick search, downloaded it and voila, it synced to your device. Loading media was a rough edge and the iTunes store was a solution to it, not just another way to make money from the hardware.
The same is true of many of Apple's other devices. They took the button heavy, confusing, clunky designs that everyone else's engineers came up with and they threw them out. They designed with simplicity and ease of use in mind, everything from the packaging to the store experience is a search for simplicity. And jobs wouldn't stop until they found it.
Isaacson says Jobs decided on the iPad after a friend who worked at Microsoft said they were building a tablet but that it used a stylus. The ill-fated Newton, started while Jobs was away from Apple and cancelled not long after his return, may have had an impact on his dislike of them, but is there anyone who liked using them, they were though to be a necessary evil. Instead of just accepting it, Jobs got his engineers to solve the problem, he wanted to use his finger, they delivered the solution. You can see Jobs' derision of the stylus when he announced the iPhone.
I always like the often-quoted story of Jobs redesigning the DVD burning app for OS X:
"Then Steve comes in," [Mike] Evangelist recalls. "He doesn't look at any of our work. He picks up a marker and goes over to the whiteboard. He draws a rectangle. 'Here's the new application,' he says. 'It's got one window. You drag your video into the window. Then you click the button that says burn. That's it. That's what we're going to make.' "
There's no doubt Jobs was a visionary, he saw things others didn't, but what he was best at was delivering great products, great because they were products people could use, that people wanted to use, instead of clunky crap. He didn't limit himself to what was possible today, he drove his company to find the right solution, not just a solution. One that was right for people using