Core component of SQL Server for storing, processing, and securing data
Things are a little different with SQL 2025, since it is now using the MSOLEDBSQL19 provider, which defaults to require an encrypted connection. (Just like all other client APIs from Microsoft.) This can be tackled in more than one way. One is to install a proper certificate on the SQL 2017 instance and which is trusted on SQL 2025 instance. This can be a certificate that you buy from a provider whose root certificate in the trusted certificate store on SQL 2025 - and quite a few vendor's root certificates comes with Windows. You can also create a self-signed certificate that you distribute yourself, see this article for how to do this: https://codekabinett.com/rdumps.php?Lang=2&targetDoc=create-install-ssl-tls-certificate-sql-server. (Note that this discussion is by no means specific to linked servers, but it applies to all connections to SQL Server, no matter they are from SSMS, applications or other SQL Server instances.)
If you follow this path, you create the linked server in the same way as before, for instance:
EXEC sp_addlinkedserver 'MYSERVER'
If you are not worried about man-in-the-middle attacks, you can opt to trust the self-signed certificate that SQL Server creates when it starts up. In this case, you need to cater for this when you create the linked server:
EXEC sp_addlinkedserver 'MYSERVER', '', 'MSOLEDBSQL19', @datasrc = 'MYSERVER', @provstr = 'TrustServerCertificate=yes'
Note here that the first MYSERVER is the alias that you will use locally for the linked server, whereas the argument to @datasrc is the actual network name for the server, which may include instance name (or port number). That is, they do not have to be the same.
However, the error message you get suggests some different issue, which may be related to the communication with the AD, but set up the linked server with trust for the certificate first, before we dig into deeper troubleshooting.