Complexity Science » Warwick Annual Retreat Projects 2017 » Some Reinforcment Learning Problem

Some Reinforcment Learning Problem

Gian Lorenzo Spisso — Tue, 09 May 2017 10:36:27 GMT

It's great you've found this, was quite interesting to read. It is also nice that we basically got all the feature of the game right (altough we were a bit reinventing the wheel). Player 1 seemed quite obvious, but apparently player 2 has several mixed strategies he could follow. We would expect any computer learner program to be able to settle down in the optimal strategy for player 1 and on one of the mixed strategies of player 2.

It is interesting how the author mentions that Player 2 is the advantaged one (as he can choose more then one strategy which might yield some advantage if P1 makes a mistake (at least that is my understanding of its final comment).

It would be interesting to see if our simple learner can:

1) Actually obtain the optimal strategies

2) Win consistently against a human

Since we have to present these results tomorrow:

1) I can lay down today an outline of the content of the slides

2) Can someone else put it into some slide format?

3) Robert: is the code up and running? Do you think by tomorrow it would be feasible to have a run of the two players playing against each other?

Some Reinforcment Learning Problem

Yihe Lu — Fri, 05 May 2017 17:13:27 GMT

Hi all,

Found a theoretic paper (Schwartz, 1959) on the solution to 2-player spoof (generally with n coins each player, all other rules the same).

I have attached it in case you want to have a look.

Otherwise, here is the keypoints you may want to take home (with potentially any possible misunderstanding or intrinsic bias of mine):

1. it is a fair game.

2. both players would play mixed strategies and they are optimal as if you try to change it, you would suffer from loss (in the sense of expectation).

3. the proof entirely uses the very classical (old-fasioned) game theoretical approach, which is just listing all possible strategies with payoff functions.

At least, hopefully you could see something similar in the convergent of your AI's strategies in the game of two players.

Attachments (follow link to download)

1959.pdf (846 KB)

Some Reinforcment Learning Problem

Robert Gowers — Wed, 22 Mar 2017 22:22:25 GMT

In this post Gian Lorenzo Spisso wrote:

Hey Robert, this was a very interesting read! Thanks for sharing this!

On the nose of it, it feels that the any paradoxical result rests on the definition of the game and I would tend to side with those claimnig that its ill posedeness is the source of paradox. But even if that's not the case, it seems that it might be a defining exactly what kind of game should be played. I'd say we'd like to keep this as tractable as possible so that we can code something and get some results in the short time available at the retreat. I might be wrong though and would love to chat about this at the retreat.

Also, I wanted to add: a summary of the game of spoof can be found here.

Discussin with Ayman we might actually go the way of making the guessing simultaneous as a starter to make it even easier.

Yes, often the problem has often been ill-posed. My intention would be to avoid the philosophical confusion by avoiding having a "perfect" predictor and simply having the predictor as another agent with their own strategy who makes a secret prediction. Reading up on spoof, I would say that is a similar type of problem, but with a greater number of variables (it seems like there should be a way to make two-player spoof equivalent to the Newcomb problem, the number of coins you put in your hand is like the prediction stage).

Some Reinforcment Learning Problem

Gian Lorenzo Spisso — Wed, 22 Mar 2017 16:19:24 GMT

*defining

This is getting out of hand quickly.

Some Reinforcment Learning Problem

Gian Lorenzo Spisso — Wed, 22 Mar 2017 16:18:51 GMT

*it might be hard definng exactly...

Apologies, I like my English like I like my operative system: obscure and unfriendly to the user.

Some Reinforcment Learning Problem

Gian Lorenzo Spisso — Wed, 22 Mar 2017 16:15:15 GMT

Hey Robert, this was a very interesting read! Thanks for sharing this!

On the nose of it, it feels that the any paradoxical result rests on the definition of the game and I would tend to side with those claimnig that its ill posedeness is the source of paradox. But even if that's not the case, it seems that it might be a defining exactly what kind of game should be played. I'd say we'd like to keep this as tractable as possible so that we can code something and get some results in the short time available at the retreat. I might be wrong though and would love to chat about this at the retreat.

Also, I wanted to add: a summary of the game of spoof can be found here.

Discussin with Ayman we might actually go the way of making the guessing simultaneous as a starter to make it even easier.

Some Reinforcment Learning Problem

Robert Gowers — Wed, 22 Mar 2017 14:12:35 GMT

I have added a WARP topic about Newcomb's problem, which is potentially a reinforcement learning problem. It's possible that we could integrate the two WARPs, depending what is decided for this WARP.

Some Reinforcment Learning Problem

Ayman Boustati — Sat, 11 Mar 2017 22:33:24 GMT

This sounds good!I am guessing the game isn't too difficult to code up? We could try it and see how we go, if it is an easy problem, we can look at other games like it to try. I think this should be fun and we can learn a lot since RL and Game Theory are share a lot of ideas.

Some Reinforcment Learning Problem

Gian Lorenzo Spisso — Sat, 11 Mar 2017 17:10:15 GMT

It should be a feasible/interesting task for reinforcement learning!

Some Reinforcment Learning Problem

Gian Lorenzo Spisso — Sat, 11 Mar 2017 17:09:40 GMT

Ehi Ayman, we were thinking together with Giovanni of trying to teach a very simple game to a computer. The game is Spoof, a drinking game where everyone picks a number of coin from 0 to 3 and in turn people try to guess the total amount available. Game should be simple enough, but at the same time the incomplete information components makes it very interesting!

Some Reinforcment Learning Problem

Ayman Boustati — Thu, 09 Mar 2017 15:51:54 GMT

I found there RL environemts we can try out:

OpenAI Gym: this seems like the easiest and most traight forward to use.
DeepMind Lab: this one focuses on 3D navigation puzzles.
Project Malmo: this one focuses on building RL agents for Minecraft.

We could either try to play around with one of these, or come up with our own problem to solve.

Some Reinforcment Learning Problem

Ayman Boustati — Fri, 03 Mar 2017 11:47:46 GMT

I am quite interested in learning more about Reinforcement Learning. Reincforcement Learning is a field that combines machine learning and game theory (link). I think it will be a good idea to explore the basics of reinforcement learning during the retreat, in the form of a simple (but cool!) project. At the moment, all I have in mind is the multi-armed bandit problem, which Sophie kindly sent me a few resources on. However, I am open to any suggestions for other interesting problems, provided they fit the timescale allocated to the WARPS (i.e. approx 7 hours). I want to start the conversation about possible projects on this forum or in person in the department, so we can have a clear idea on what to do during the retreat.

Looking forward for suggestions!