I managed to solve it and will answer my own questions if somebody else might have similiar problems.
1. Yes, you can use self signed certificates with the Collaboration server. However, it was a bit of a challange to set up an ssl certificate and configure the local network so that both the openLCA client and the internet browsers will work, since they have different rules. The problem described above was likely because openLCA could find the server, but there was a name mismatch with the certificate.
2. If the certificate and the local network configured correctly, the certificate will be added as trusted by following the instructions in openLCA described in the question.