VPS is an acronym which stands for Virtual Private Server. A VPS hosting environment mimics a dedicated server within a shared hosting environment. VPS Hosting is one of several types of web hosting accounts you can choose to host your website online. The technology behind VPS hosting is similar to that of VMware or Virtual Box. These programs allow you to run several virtualized operating systems on one machine.
While VPS is a server on its own and you have a dedicated IP and administrator access, Remote Desktop Protocol (RDP) is a Microsoft protocol designed to facilitate application, data transfer, security and encryption between client users, devices and a virtual network server. It enables a remote user to add a graphical interface to the desktop of another computer. Each RDP Server is shared among ~10 remote desktop accounts. You don't get administrator access in the server but all the works you do in your RDP Account are 100% private.
In brief, RDP provides remote display and input capabilities over network connections for Windows-based applications running on a server, while VPS is a virtual machine that you can use to host your data and since VPS runs its own OS, you can get a Windows or Linux VPS server.
Which is better VPS or RDP?
That totally depends on your needs. If you want to host websites or want complete control of the Server, you should probably go for VPS. If you want a shared server with high configuration and without administrator access you should go for RDP Account.