There is a big difference between alternate universes and alternate history. Alice fell through the rabbit hole into an alternate universe. Robert Silverberg explored the ramifications of alternate history in Up the Line. One lets you go anywhere from the starting place of the "real" Earth (or whatever planet you started on); the other is bound by the strictures of actual history. These are examples of alternate universes:

• Alice in Wonderland and Through the Looking Glass by Lewis Carroll
• Red Moon and Black Mountain by Joy Chant
• The Dragon and the George by Gordon R. Dickson, and its sequels:

  The Dragon Knight
  The Dragon and the Djinn
  The Dragon on the Border

• The Mirror of Her Dreams and A Man Rides Through It by Stephen R. Donaldson
• The Riftwar series by Raymond Feist, which also includes the trilogy written with Janni Wurtz. These two series are interesting in that they examine the war from the POV of the opposing sides:
  
  From the Krondor side:

  Magician: Apprentice
  Magician: Master
  Silverthorn
  A Darkness at Sethanon
  Prince of the Blood
  The King's Buccaneer

  From the Tsurani side:

  Daughter of the Empire
  Servant of the Empire
  Mistress of the Empire

• The Broken Citadel, Castledown, and The Great Wheel by Joyce Ballou Gregorian
• The Guardians of the Flame series by Joel Rosenberg, comprising:

  The Sleeping Dragon
  The Sword and the Chain
  The Silver Crown
  The Warrior Lives
  The Road to Ehvenor

• Barbara Hambly's Darwath series, including

  The Time of the Dark
  Armies of Daylight
  The Walls of Air
  Mother of Winter
  Icefalcon's Quest