This scheme was started by Nintendo to counter the effects of the video game crash of 1983.
There were several reasons for the crash, but the main cause was supersaturation of the market with hundreds of mostly low-quality games which resulted in the loss of consumer confidence....
In 1986, Nintendo president Hiroshi Yamauchi noted that "Atari collapsed because they gave too much freedom to third-party developers and the market was swamped with rubbish games." In response, Nintendo limited the number of titles that third-party developers could release for their system each year.
It continues today because it's a lucrative deal for console manufacturers, allowing them to sell devices below-cost to customers, and make money licensing the SDK to developers, often taking a cut of revenue as well. The benefit to consumers today is highly questionable, especially compared to more-open-but-still-gated models like Apple's App Store.
The difference between the free and proprietary SDK varies by vendor. In all cases, the free option is more technologically limited - XNA limits you to C#, for example, where the full XDK allows you to write in any language that can target the Xbox 360's PPC-based architecture. The same is true of the hardware. The Xbox debugging units contain larger hard drives, can connect to SMB drive shares, and connect to a special version of Xbox Live. They usually also come with customer support contracts and access to other developer tools like better profilers or test harnesses.