Since I didn’t find it right away, here’s the solution. The default notation “server:port” doesn’t work here (for God knows what reason). You need to seperate the hostname and port with a comma. Here’s an example.

SQL Server Alternate Port
What ever happened to the default point seperation? :-(