As mentioned previously, an exploitive bot needs to learn how the opposition plays and develop counter strategies to their style. The first thing we need to do to achieve this is gather information on our opposition. There is already software out there to do this for you such and Hold'em Manager, Poker Tracker and Poker Office, you can then tap into their databases. The other option is to create your own database and write a hand importer yourself, the only advantages of this being it's more flexible and you can use a more powerful database engine and tools such as Microsoft Sql Server.
Once you've done this you need hand histories to import, you may have many hand histories of your own, but I'd recommend starting out at the lowest stakes, so unless you've been playing $2 games for a while you're unlikely to have enough. There are various sites online where you can buy hand histories, and several places you can pick up obfuscated hand histories for free.
Once you've got all your hand histories and started talking to your database you can start the process of working with the data. There are two approaches here, a statistics based approach, or a machine learning approach.
The statistics based approach is simplest, it basically says if player x bets in this situation 15% of the time, then we'll assume they're doing this with the top 15% of hands, it's pretty crude, but effective.
With the machine learning approach we're starting to get into the world of Skynet. I'm not going to go into too much detail here, as there is plenty of information on Wikipedia and for more poker specific articles you can take a look at PokerAI.
The general idea behind the machine learning approach is that the computer is going to learn how other players play based on their statistics, and we can use this information to predict how they'll act is certain situations and respond to that. Clever stuff hey?! Machine learning is still very basic however. Artificial neural networks, for example, try to mimic the human brain's method of reinforcement learning. But... the human brain has an estimated 100 billion neurons and is effectively running millions of parallel CPU cores, we'll be working with networks with a number of neurones that you can count on your hands and toes, and unless you have a lot of money, you're not going to be working with millions on parallel CPUs either. So it's a bit like trying to teach poker to a toddler, sometimes they just don't understand!
Saturday, 6 November 2010
Tuesday, 2 November 2010
Where do I start?
I'm not going to be dealing with the poker bot basics in this blog. If you've got it into your head that you want to have a shot a writing one, but don't know where to start, then take a look at this Coding The Wheel article. That'll introduce you to the concepts of interfacing and interacting with the poker client of your choice.
In this blog I'm going to be picking up from where I'm currently at, there will be various tips and tricks that developers may find useful as well as code snippets from what I'm working on and an overview of how my bot and AI work.
In this blog I'm going to be picking up from where I'm currently at, there will be various tips and tricks that developers may find useful as well as code snippets from what I'm working on and an overview of how my bot and AI work.
Sunday, 31 October 2010
Where can I buy a winning poker bot?!
Ok, so no one has asked me this question yet via this blog, but in anticipation for the moment that they do I thought I'd write this...
Ask yourself this. If someone is making a decent living off a poker bot then why would they sell it?
Yes, you might be able to buy bots that can beat the 1c/2c games straight out the box. The reason the developer is selling these bots is because they figure they can make more money from selling the bot than running it.
If someone already has a decent winning bot, then do you really think they're going to want to introduce the hassle of customers? Are they going to want to increase the risk of being caught, as any poker site can pose as a customer, buy the software, and then just look for indicators of that software being run. Either looking for the process, or patterns that are specific to that bot.
Let's say poker bots are priced the same way as a business, so about 3.5x the annual profit is a fairly standard metric. That means a poker bot that makes $50,000 a year, is worth about $175,000. Would you pay $175,000 for a poker bot? If so, drop me an email! ;-) But seriously, no one is going to be able to write a poker bot unless they are intelligent, and no intelligent person is going to sell a decent winning poker bot for $50. Free money is worth a lot more than that.
Because of this, and the fact that winning poker bots are near impossible to write, winning bots are never likely to be a widespread problem in the world of online poker. So if you're browsing the internet and have stumbled across this page while looking for a poker bot that can let you retire and buy yourself a yacht in the Caribbean, then save yourself some time and money and give up now. You're not going to find one.
Ask yourself this. If someone is making a decent living off a poker bot then why would they sell it?
Yes, you might be able to buy bots that can beat the 1c/2c games straight out the box. The reason the developer is selling these bots is because they figure they can make more money from selling the bot than running it.
If someone already has a decent winning bot, then do you really think they're going to want to introduce the hassle of customers? Are they going to want to increase the risk of being caught, as any poker site can pose as a customer, buy the software, and then just look for indicators of that software being run. Either looking for the process, or patterns that are specific to that bot.
Let's say poker bots are priced the same way as a business, so about 3.5x the annual profit is a fairly standard metric. That means a poker bot that makes $50,000 a year, is worth about $175,000. Would you pay $175,000 for a poker bot? If so, drop me an email! ;-) But seriously, no one is going to be able to write a poker bot unless they are intelligent, and no intelligent person is going to sell a decent winning poker bot for $50. Free money is worth a lot more than that.
Because of this, and the fact that winning poker bots are near impossible to write, winning bots are never likely to be a widespread problem in the world of online poker. So if you're browsing the internet and have stumbled across this page while looking for a poker bot that can let you retire and buy yourself a yacht in the Caribbean, then save yourself some time and money and give up now. You're not going to find one.
Friday, 29 October 2010
Types of poker bot - The Exploitive Bot
The exploitive bot is a very different animal to the optimal bot, and far more human like in it's play. The optimal bot removes all aspects of psychology from the game, it's purely a formula that is fortunately near impossible to calculate, otherwise online poker would be ruined for ever.
The exploitive bot works by analysing other players, identifying their weaknesses and then working out counter-strategies to their style of play. These bots will adapt to players and tables, always trying to give themselves an edge. This is far more like the style of play of a poker professional. The problem with this style of bot is that unlike the optimal bot, where you have one strategy that's played against all players, you have a potentially infinite number of strategies. No two poker players play exactly the same way, and therefore you need to develop a different strategy for each one.
The advantage of this permanently shifting strategy is that it makes the bot far less predictable. If the optimal bot wasn't quite optimal, and a hole was found in it's strategy, then a player could exploit this over and over and it would never adapt. The exploitive bot however will not only adapt to exploit others, it will also adapt to others exploiting it.
The problem with this method is not only do you have to calculate your exploitive strategies on the fly, as you clearly can't precalculate an infinite number of strategies, the bot also needs to understand human thought and develop good hand reading skills. With this method of play, we're trying to develop a learning, thinking machine that adapts to what's going on around it. In my opinion, the optimal bot isn't intelligence, but more a complex set of rules. The exploitive bot is true artificial intelligence, and a much more interesting project in my opinion, and what this blog is going to be concentrating on.
So while calculating the perfect exploitive strategy would have the same complexities as creating the perfect optimal strategy. The fact that we're constantly adapting our strategy and are less predictable means that our strategy doesn't need to be quite as perfect as the optimal strategy to still be able to pull a profit.
The exploitive bot works by analysing other players, identifying their weaknesses and then working out counter-strategies to their style of play. These bots will adapt to players and tables, always trying to give themselves an edge. This is far more like the style of play of a poker professional. The problem with this style of bot is that unlike the optimal bot, where you have one strategy that's played against all players, you have a potentially infinite number of strategies. No two poker players play exactly the same way, and therefore you need to develop a different strategy for each one.
The advantage of this permanently shifting strategy is that it makes the bot far less predictable. If the optimal bot wasn't quite optimal, and a hole was found in it's strategy, then a player could exploit this over and over and it would never adapt. The exploitive bot however will not only adapt to exploit others, it will also adapt to others exploiting it.
The problem with this method is not only do you have to calculate your exploitive strategies on the fly, as you clearly can't precalculate an infinite number of strategies, the bot also needs to understand human thought and develop good hand reading skills. With this method of play, we're trying to develop a learning, thinking machine that adapts to what's going on around it. In my opinion, the optimal bot isn't intelligence, but more a complex set of rules. The exploitive bot is true artificial intelligence, and a much more interesting project in my opinion, and what this blog is going to be concentrating on.
So while calculating the perfect exploitive strategy would have the same complexities as creating the perfect optimal strategy. The fact that we're constantly adapting our strategy and are less predictable means that our strategy doesn't need to be quite as perfect as the optimal strategy to still be able to pull a profit.
Thursday, 28 October 2010
Types of poker bot - The Optimal Bot
There are two distinctly different types of poker bot, the exploitive bot, and the optimal bot. These are very different ideas and require very different approaches.
Anyone who has played many poker tournaments will have heard of the Nash Equilibrium tables for poker. Nash Equilibrium is a game theory concept that says every game with 2 or more players has an optimal strategy, and no matter what someone else does, they will not be able to exploit that. This is the idea behind the optimal player, the optimal player is unexploitable, however, the flip side of this is that the optimal player does not exploit. "So that means it's possible to make an unbeatable poker bot" I hear you scream! Well, yes... in theory, and that's the important point here. Why do you think that the Nash tables have only been calculated for 2 player push / fold games? The amount of computing power and time required to calculate these strategies is huge, and with every bit of complexity you add, the required power increases exponentially. I doubt there is enough computing power in the universe to calculate these strategies for 3 player No-Limit, let alone anything beyond that.
The problem with No-Limit is that every different bet amount creates a different game state, so where as with Limit, you can fold, call or bet with No-Limit you can fold, call, or bet $1, or bet $2, or bet $1.33. If you've got $100 left if your stack, and the big blind is $1, that's 9,900 different possible game states created from different bet amounts. Now if every player at the table can create 10,000 possible different game states on every go, that's 1000000000000000000000000 possible game states with 6 players pre flop, and that's before we take into account every single possible combination of cards that can be dealt, future turns, and every possible combination of board cards. Abstractions can be used to simplify the game, but with every abstraction you're losing detail and are less likely to be able to find the true optimal solution. If short handed No-Limit hold'em is solved in my lifetime, I'll be very surprised.
The University of Alberta have been working on producing a world class poker bot for well over a decade now, and while their Heads-up limit bot is pretty damn good, and their Head up No-Limit bot isn't bad, anything beyond that is no where near being able to compete on the world stage yet.
So, optimal bots are a great theory, that's all they are, and when it comes to No-Limit Hold'em, that's all they'll ever be.
Anyone who has played many poker tournaments will have heard of the Nash Equilibrium tables for poker. Nash Equilibrium is a game theory concept that says every game with 2 or more players has an optimal strategy, and no matter what someone else does, they will not be able to exploit that. This is the idea behind the optimal player, the optimal player is unexploitable, however, the flip side of this is that the optimal player does not exploit. "So that means it's possible to make an unbeatable poker bot" I hear you scream! Well, yes... in theory, and that's the important point here. Why do you think that the Nash tables have only been calculated for 2 player push / fold games? The amount of computing power and time required to calculate these strategies is huge, and with every bit of complexity you add, the required power increases exponentially. I doubt there is enough computing power in the universe to calculate these strategies for 3 player No-Limit, let alone anything beyond that.
The problem with No-Limit is that every different bet amount creates a different game state, so where as with Limit, you can fold, call or bet with No-Limit you can fold, call, or bet $1, or bet $2, or bet $1.33. If you've got $100 left if your stack, and the big blind is $1, that's 9,900 different possible game states created from different bet amounts. Now if every player at the table can create 10,000 possible different game states on every go, that's 1000000000000000000000000 possible game states with 6 players pre flop, and that's before we take into account every single possible combination of cards that can be dealt, future turns, and every possible combination of board cards. Abstractions can be used to simplify the game, but with every abstraction you're losing detail and are less likely to be able to find the true optimal solution. If short handed No-Limit hold'em is solved in my lifetime, I'll be very surprised.
The University of Alberta have been working on producing a world class poker bot for well over a decade now, and while their Heads-up limit bot is pretty damn good, and their Head up No-Limit bot isn't bad, anything beyond that is no where near being able to compete on the world stage yet.
So, optimal bots are a great theory, that's all they are, and when it comes to No-Limit Hold'em, that's all they'll ever be.
Wednesday, 27 October 2010
Hello, and welcome to The Poker Bot Blog
Poker bots, poker bots, poker bots... it's a hot topic at the moment in the world of online poker. Full Tilt just shut down (probably hundreds) of accounts suspected of using the Shanky Bot. People are talking about poker bots being the death of online poker, and sites are investing more and more resources into keeping control of the problem as people are trying to get what they see as free and easy money.
So, where do I fit into this? I'm a professional software developer who got bored of work. I had an interest in poker, so quit work and decided to try and knock up a winning poker bot.
I've always had an interest in artificial intelligence, and thought this poker bot stuff would be pretty easy. I was wrong, so very wrong.
Poker involves 3 things.
I've been working hard in the world of poker botting for over a year now, and have learnt a lot in the process. The purpose of this blog is to explain the world to the public, and publicise my research. I will explain why poker bots are not to be feared, and encourage discussions on the morality of the subject.
Hopefully some of you will find this interesting, stay tuned.
So, where do I fit into this? I'm a professional software developer who got bored of work. I had an interest in poker, so quit work and decided to try and knock up a winning poker bot.
I've always had an interest in artificial intelligence, and thought this poker bot stuff would be pretty easy. I was wrong, so very wrong.
Poker involves 3 things.
- Maths - Computers are pretty good at this, as a matter of fact, this is their greatest strength. If they know everyone's cards then they can play mathematically perfect poker. Unfortunately, playing your cards in poker doesn't win you money. Luck evens out in the end, if everyone just played their cards the same way no one would win anything in the long run. To win in the long run, you need to be able to win with the worst hand, this involves bluffing, and that involves psychology!
- Psychology - Computers are very good at things we're not, like maths, and very bad and things we do without thinking, like seeing and hearing. Voice recognition and image recognition are still in their infancy, even after many years of research. Poker involves 'thought recognition', this is a hard concept for people to get their head around, let alone a computer. How do you get a computer to understand what someone is thinking?!
- Luck - Humans and computers have got a fairly level playing field here, neither can control luck. Computers do however have one slight advantage. While a human player could easily be put on tilt by a piece of bad luck, a computer doesn't have emotions, it can just play the same steady game no matter what happens
I've been working hard in the world of poker botting for over a year now, and have learnt a lot in the process. The purpose of this blog is to explain the world to the public, and publicise my research. I will explain why poker bots are not to be feared, and encourage discussions on the morality of the subject.
Hopefully some of you will find this interesting, stay tuned.
Subscribe to:
Posts (Atom)