read

A few of my recent projects have involved interfacing with Steam via SteamKit and through that, the Dota 2 game coordinator. In this article I will attempt to explain the interaction between the various elements of the Steam network, and how these are harnessed in SteamKit to create powerful and helpful bots on the network.

Protocol Buffers: The Internal Binary Serialization

For those new to interfacing with Valve's internal Steam and Game Coordinator networks with SteamKit, the entire system uses Google's ProtoBuf system for message serialization and communication, and all game coordinator messages go through the connection to the Steam network, not a secondary direct connection to the game's servers. In this way games that use Valve's Game Coordinator system rely on Steam to be online, and a valid Steam account signed in to access the system.

Valve typically includes the ProtoBuf protocol definition files in the compiled library resources, which allows us to regularly dump them using a tool written by the SteamRE team. The result is a tree of .proto files, for example the Dota protocol definitions. Due to the nature of ProtoBuf, even if these are slightly out of date, they will continue to function properly with reasonably minor revisions to the system, and thus don't need to be dumped very frequently.

As all protocol messages are detailed in these files, and we connect to the Steam and Game Coordinator networks in the same way as the official client, bots written using SteamKit can perform any action the normal clients can, in the exact same way, without limitations.

A typical protobuf message definition will look like the following:

message CSODOTAPartyInvite {
     message PartyMember {
        optional string name = 1;
        optional fixed64 steam_id = 2;
    }

    optional uint64 group_id = 1 [(key_field) = true];
    optional fixed64 sender_id = 2;
    optional string sender_name = 3;
    repeated .CSODOTAPartyInvite.PartyMember members = 4;
    optional uint32 team_id = 5;
    optional bool low_priority_status = 6;
    optional bool as_coach = 7;
}

Typically most fields are denoted as optional which allows the protocol to be mutable over time, should they be removed, clients will simply see them as missing and not strictly refuse to deserialize the message as incomplete. This allows the protocol to change while still allowing older versions of the client to interface with the network gracefully. Thus backwards compatibility is default, and the latest protocol version can always be used, without implementing multiple handlers, one for each version. Finally, repeated is the way to represent arrays in ProtoBuf, while still allowing the field to be empty (represented as an empty array) maintaining backwards compatibility.

These messages are converted into data structure representations for a variety of languages, using associated generators. Here are a few:

  • protobuf-net - C#
  • node-steam - Javascript
  • protoc - c, c++, java, python

The representations of these messages is different for every language, but the ProtoBuf system will always serialize them to identical binary representations regardless of language. This allows easy communication between many different systems and languages.

When the client receives a message from the server, it will come in as a raw binary array. This data could be any of these messages - ProtoBuf does not have any built-in message identifiers in serialized data - so every message must be exactly the same. Steam's implementation is a bit complex, however, put simply - every message is deserialized to a container message containing an integer message identifier and a byte array for the binary message. The client then knows what message definition to use to deserialize the byte array and get the final message. Thus, every message has an identifier, which is contained in enumerations like this one:

enum EDOTAGCMsg {
    k_EMsgGCDOTABase = 7000;
    k_EMsgGCGeneralResponse = 7001;
     k_EMsgGCGameMatchSignOut = 7004;
    ...
}

To create a psuedo-client for writing bots for these game coordinators, one must implement handlers for every message they're interested in observing and reacting to. Overall, this ends up being thousands of messages, many of which are never relevant to the operation of a typical bot. Simply implementing handlers as they become relevant is a good time-efficient mode of development.

Observing Communication at Runtime

To implement handlers for these messages, it's often helpful to watch the communication between the official client and network. This is what the NetHook program is used for. To use the nethook dumper, compile it and inject the DLL into the Steam client (not the game!!). The system will dump all of the observed messages to a directory.

To analyze these messages, use the NetHookAnalyzer. This will display every message in sequence, which is useful to reverse engineer the message sequence for initializing and connecting to the network, as well as performing tasks.

Most game coordinators will require a "hello" message, which it will respond to with a "Welcome" message containing the version number of the GC among other things. From this point onward the GC will accept any normal messages from the client, and allow you to perform operations.

In the case of the Dota 2 client, the client will request a bunch of data regarding live matches and rich presence of friends (rendered as "Playing a game" or "as Sand King (level 1)"). This is completely unnecessary, however, and can be done at any time.

Performing GC Operations: DOTA 2 Lobby Example

The source code for a full implementation I've written of the DOTA handlers is available in my SteamKit fork.

Use one of the SteamKit examples to get an account signed in and ready.

First, you will want to request a connection to the Game Coordinator by telling Steam that we're playing Dota 2 (this will change the status of the bot from "Online" to "Playing Dota 2"). Notice it's an array - you can specify multiple games played, but as far as I know the first one will always be the "primary" played game.

var playGame = new ClientMsgProtobuf<CMsgClientGamesPlayed>(EMsg.ClientGamesPlayed);

playGame.Body.games_played.Add(new CMsgClientGamesPlayed.GamePlayed
{
    game_id = new GameID(570),
});

// notice here we're sending this message directly using the SteamClient
this.Client.Send(playGame);

Next, wait a few seconds, and send out the "hello" message to the GC:

var clientHello = new ClientGCMsgProtobuf<CMsgClientHello>((uint)EGCBaseClientMsg.k_EMsgGCClientHello);
Send(clientHello, 570);

After receiving the k_EMsgGCClientWelcome message, you can now create a lobby like so:

        var create = new ClientGCMsgProtobuf<CMsgPracticeLobbyCreate>((uint) EDOTAGCMsg.k_EMsgGCPracticeLobbyCreate);
        create.Body.pass_key = "helloworld";
        create.Body.lobby_details = new CMsgPracticeLobbySetDetails();
        Send(create, 570);

You can set any options you want in lobby_details. It's very easy to set up the lobby however you want from here.

I've reverse engineered how the process of creating a lobby works - the client will receive a cache subscribe and later cache updates with the full lobby object when it is updated - but I will leave the implementation explanation for another date.

Performing Steam Actions: Friends Example

Steam has a variety of subsystems: one of them is the "Friends" subsystem. The friends system handles your name, profile, persona state, etc. SteamKit comes with built in handlers for most Steam functions, so you can use them quite easily:

client = new SteamClient();
friends = client.GetHandler<SteamFriends>();

Later on, once you've initialized the client:

friends.SetPersonaState(EPersonaState.Online);
friends.SetPersonaName("My Cool Bot");

Interacting with these elements of the system is quite trivial as nice abstractions in the form of handlers have already been written for you. I encourage you to check out the SteamKit examples.

Conclusion

All of the elements are here to build a complex system - absolutely (almost) anything the DOTA or Steam clients can do can be replicated in a bot. I encourage you to explore this functionality, but not abuse it, as Valve has allowed the SteamRE project to continue without restricting the access to the protobuf definitions in the binaries nor applying additional security against illegitimate clients thus far. Please use this knowledge in moderation.

Like anything in programming, I view development of this sort of system like a stack, or an onion - it comes in layers. SteamRE provides the lower level layers to interface with the Steam network, handlers are built on top of that, and your app can be built on top of that as well, in increasingly high-level task management.

A couple of projects I'm using this for:

  • Subscriber Games - an automated Twitch inhouse system
  • (unreleased) a web league system coming up later this year.

A couple cool projects SteamKit has been used for in the community:

Blog Logo

Christian Stewart


Published

comments powered by Disqus
Image

Christian Stewart

Also known as Quantum and Paralin.

Back to Overview