Varley was born in England in 1881, and went to the Sheffield School of Art. From 1900 to 1902 he attended the Académie royale des beaux-arts in Antwerp, Belgium. In 1912, Varley moved to Canada, settling in Toronto, where he worked for the commercial art firms Grip Ltd. and Rous & Mann. A member of the Group of Seven, he had met fellow Group artists at the Arts and Letters Club in Toronto. In 1914 he joined Tom Thomson for a sketching trip to Algonquin Park with A.Y. Jackson and Arthur Lismer.

In 1918 Varley was commissioned to paint for the Canadian War Records, traveling to England and France. In 1920 he returned to Canada and exhibited in the first Group of Seven show. The summer of that year he painted at Georgian Bay in northern Ontario. Varley exhibited work at the influential Wembley Show in England in 1924.

From 1925 to 1926 he taught at the Ontario College of Art, then in 1926 moved to Vancouver to teach at the Vancouver School of Decorative and Applied Arts. He lived near Jericho Beach and, inspired by the lush landscape, painted in the North Shore mountains and along the Fraser River. He also traveled to Garibaldi Park to paint in 1927, 1929, 1932 and 1934. Varley was also a perceptive figurative painter, most notably of his muse Vera Weatherbie, and also did portrait commissions. In 1933, Varley resigned from the Vancouver School of Art and founded the British Columbia College of Arts with Jock Macdonald. In 1934 Varley moved to Lynn Valley in North Vancouver, painting many fine landscapes of the area. The closure of the College in 1935 made Varley look east again.

In 1936 Varley moved to Ottawa to teach at the Ottawa Art Association, and exhibited in the Group of Seven show at the National Gallery that year. In 1938 his work was exhibited at the Tate Gallery in London, and that summer he went on a painting trip to the Arctic on the government supply ship Nascopie. After leaving his position at the Ottawa Art Association in 1939, he moved to Montreal in 1940. The year 1943 saw him back in Ottawa, and 1945 he moved to Toronto. From 1948 to 1949 he taught summers at the Doon School of Fine Arts near Kitchener, Ontario. Varley continued his travels with a 1955 painting trip to Cape Breton, Nova Scotia and in 1957 made the first of several summer trips to the Kootenay area of BC, with his last trip to BC occuring in 1967.

More recognition came to Varley with a 1953 National Film Board on his work, and in 1954 a retrospective exhibition was held originating at the Art Gallery of Toronto, which traveled to the National Gallery of Canada, the Montreal Museum of Fine Arts and Western Canada.