Virtual Machine Remote Control (protocol)
VMRC is a pile of garbage which is modified VNC with NTLM and Kerberos authentication. Ain't that fun?
Used in Virtual Server 2005.
The protocol is a modified version of the Remote Framebuffer Protocol, which is referenced in this specification. When detail or behavior is not described here, it should be assumed to be the same as documented for RFB 3.3 in the RFB Specification.
Handshake
ProtocolVersion
As in RFB, the server begins the handshake with a ProtocolVersion message. However, it reports itself as RFB version 3.6:
| No. of bytes | Value |
|---|---|
| 12 | "RFB 003.006\n" (hex 52 46 42 20 30 30 33 2e 30 30 36 0a)" |
The client is expected to reply with this same ProtocolVersion packet; the VMRC server is not backwards-compatible with standard RFB versions.
Security Types
As in RFB 3.3, the server now selects the security type by sending a single word. However, VMRC does not support any of the standard RFB security types, and instead provides its own:
| Number | Name |
|---|---|
| 4 | NTLM |
| 5 | Reconnect |
| 6 | Negotiate |
Authentication
As in RFB 3.3, after the server selects the security type, authentication data specific to that type follows. These are documented in the following subsections.
NTLM
The NTLM security type provides authentication via the NT LAN Manager Authentication Protocol.
All NTLM messages are sent with the following shape:
| No. of bytes | Type | Description |
|---|---|---|
| 4 | U32 | message-length |
| message-length | NTLM_MESSAGE | message |
The authentication flow follows the connection-oriented NTLM flow, specified as linked above. Namely, the client starts the flow with a NegotiateMessage, the server responds with a ChallengeMessage, and the client completes the process with a AuthenticateMessage.
The official VMRC client always uses a value of 2718478855 (U32 LE) for the NegotiateFlags value. I have not tested with other values.
On success, the server sends a constant word before the SecurityResult. I do not know what this is for.
| No. of bytes | Type | Value | Description |
|---|---|---|---|
| 4 | U32 | 0x00 | success |
On failure, the server closes the connection without any response.
Reconnect
 | This section (Reconnect) is a stub. You can help the Computernewb Wiki by expanding or deleting the section (Reconnect). |
0x05 always will connect twice if detected, and changes the authentication method to:
- 0x04: NTLM (NTLMSSP_NEGOTIATE flags set to 0xA2088207.)
- 0x05: Kerberos.
- 0x06: NTLM/Negotiate (NTLMSSP_NEGOTIATE flags set to 0xE2088297.)
Negotiate
TODO: No idea what this is... the client calls it NTLM but the structure is way different.
Post Authentication
After authentication, the server sends a SecurityResult, and then the client and server exchange a standard RFB ClientInit and ServerInit.
The official client always sets the shared-flag to 1, and the official server always uses a client-name of "Virtual Server"
Capability Negotiation(?)
After ServerInit, the client sends a message that is believed to be some sort of capability negotiation. I do not currently know what the values mean, but this is required.
| No. of bytes | Value |
|---|---|
| 4 | hex 07 00 00 00 00 00 00 01 2a |
The server sends two messages in response to this, both beginning with a message-type of 0x05. The first is 396 bytes total and seems to contain a list of capabilities offered by the server (as ASCII strings) along with some binary values. The second is only 13 bytes and contains more unknown binary values.
VMRC Connect
The client now sends a message to connect to a VM:
| No. of bytes | Type | Value | Description |
|---|---|---|---|
| 1 | U8 | 0x07 | message-type |
| 7 | observed hex 02 00 00 00 00 00 02 | unknown | |
| 4 | U32 | vm-length | |
| vm-length | string | vm |
vm should be set to the Virtual Machine the client wishes to connect to. To instead display the VM list, vm should be omitted and vm-length set to 0.
After sending this, the server will respond with a VMRC Display Changed message. The handshake is now complete and normal RFB protocol behavior takes place.
Server to Client messages
These can be sent at any point after the handshake.
VMRC Display Changed
The server sends this message when the client connects to a VM, returns to the list, or when the VM resolution changes. After the header, it follows the same format as a standard RFB ServerInit.
| No. of bytes | Type | Value | Description |
|---|---|---|---|
| 1 | U8 | 0x04 | message-type |
| 3 | padding | ||
| at least 24 | ServerInit | server-init |