The Elmbridge Phoenix Swimming Club was originally named the Walton Swimming Club and was founded in 1908. Before the Walton-on-Thames Pool was built in the 1960s, the club held races, river sports and fun galas in the River Thames. Victoria Cross and sculling races, water jousting, costume races and Tug-o-war were interspersed with handicap and scratch swimming Events.

By the early 21st century the old 33m pool in King's Close was becoming more expensive to maintain, and the council thought the land would be better used for housing.

In 2006 the Leisure Centre at Waterside Drive was rebuilt to include an eight-lane 25m competition pool and was renamed as the Xcel Leisure Complex. The club moved into the new centre even before its public opening and shortly afterwards the old pool was demolished.