There were plenty of peers, even UK and European ones, that opposed the US colonial project. Read Losurdo - Liberalism, a counter-history if you want an in-depth look at the debates of the time.