Almost two years since I started this project page... It's time for a touch-up on this!
I'm creating the perfect Blackjack player! This Raspberry Pi-powered robot will identify the cards in its hand and the dealer's upcard, and use a Hit or Stand lookup table to determine the best play to make. It will also be able to count cards and implement card counting strategies like the "Illustrious 18". So far, I've made a card detector program that uses a trained machine learning object detection model (YOLO v3) that works extremely well at identifying cards.
For the next phase of the project, I'm working on coding RAIN MAN to play through actual hands of blackjack. I'll have to implement a state machine that brings him through different phases of a round of blackjack: reading initial deal, making play decisions, and resolving the hand.
Details
Here's a video showing how the machine learning-based card detector works! The robust detection model allows RAIN MAN 2.0 to see cards in a variety of conditions, and even when the cards are overlapping.
List of Project Tasks to Complete:
Software
Get card detector program to work with overlapping cards. (Going to try to implement convolutional neural network for detecting the cards) - DONE!
Add card counting capability to card detector - DONE!
Get card counting to work with multiple decks (this will require re-training the detection model)
Add ability to determine correct play (hit, stand, split, double down) for a given blackjack hand + dealer upcard
Add ability to play through a hand of blackjack
Hardware
Create an initial 3D model of what the physical blackjack bot will look like - use best guesses on what the dimensions will be
Create servo-driven arm that taps the card table twice for a "hit", or waves side to side for a "stand". Figure out best way to wire servo up to Raspberry Pi
Integrate a small LCD display that shows what cards the bot is seeing
This is just an initial list! As the project develops, I will undoubtedly find more things I need to do.
Over the past couple months, I've been tinkering with machine learning to try and train an object detection neural network that can detect playing cards. The OpenCV algorithm I used (described in this video) works great at detecting cards, but it doesn't work if the cards are overlapping even the slightest bit. Unfortunately, blackjack is always dealt with the cards overlapping. If my blackjack robot is going to work, it needs to be able to count cards even when they're overlapping.
Someone told me that I might be able to train an object detection classifier (a type of neural network) to recognize the cards even if they're partially obscured or overlapping. Object detection classifiers recognize patterns to identify objects, so they only need to see a portion of the object to detect it. I decided to use Google's TensorFlow machine learning framework to train a playing card detection classifier.
I've spent lots of time learning about machine learning (enough to make a tutorial showing how to train your own) and I've taken hundreds of pictures of playing cards to feed to the training API. Unfortunately, it's starting to seem like machine learning isn't going to be the silver bullet I hoped it would be. The trained playing card detector just doesn't work very well.
For the most part, it works great when it has a clear view of the cards (so does my OpenCV algorithm):
And it even works if the cards are overlapping:
But if I deal some actual blackjack hands in front of the camera, the way it would be done in a casino, it isn't able to detect all the cards. The cards are too overlapped for it to see all the cards.
Right now, it isn't trained well enough to distinguish that there are two cards in each hand. It only sees the top card. It's possible that if I fed the trainer hundreds more clearly labeled pictures of overlapping cards, it might be able to see both the cards. I've already given it 367 training pictures, but maybe it will work better if I give it 1,000 more. Also, it is still a little inaccurate and sometimes incorrectly identifies cards. More training data might help with this, too.
However, there are some other problems. My classifier is trained off Google's Faster RCNN Inception model, which takes lots of processing power. I want to run my blackjack robot on a Raspberry Pi, which has limited processing power. I tried using the lower-power MobileNet-SSD model, but it doesn't work very well at identifying individual cards. I need to find a way to keep the processing requirements low while still having good accuracy.
Also, I only have the detector trained to recognize card ranks nine, ten, jack, queen, king, and ace. It will take lots more training pictures to get it to work with every card rank. I have a sneaking suspicion that it won't work as well on the lower numbers (four is very similar to five, etc.).
However, I'm still going to try! My next step is to train the detector to recognize ALL cards, not just nine through ace. Then, I'll run it on a Raspberry Pi and see if it's still able to detect cards fast enough, and make a YouTube video about it.
I'm still trying to think of how I might be able to get it to work with the cards overlapping. I think the solution will involve a combination of machine learning and some image processing with OpenCV. Please let me know if you have any ideas!
I work in Vegas, in surveillance, the program it's self would be awesome to have to run down players with. If you could get it to spit the count out on to a spread sheet that an agent could add the rest of the needed info to I think you'd have a million dollar product
Is it possible to perform the detection on the computer and use a raspberry pi as a controller for motors? I am trying to sort the playing cards into 4 baskets of the 4 suits using a simple 2 motor mechanism. one that will rotate and stop at 4 different angles and another that will just turn 180 degrees to drop the playing card.
You should persist on the path of machine learning. It works very well, at least on a gpu. For the cards overlapping, just focus the training on the card corners. And you are right about the training data: you need lots more. But the chance with the detection of cards problem (vs detection of dogs for instance) is that you can artificially generate yourself your data. That's what I did to train a YOLO neural net. You can find more info on this video I've made : https://youtu.be/pnntrewH0xg
Thanks!! I wish I had seen this comment when you posted it two months ago. Your card detector works amazingly well! I'll see if I can re-create your wonderful work.
To solve the occlusion accuracy problem have you considered training and recognizing just card corners, their left sides, or just the text rather than the entire card?
Evan Juras
Svavar Konradsson
Lars Knudsen
gmccauley
I work in Vegas, in surveillance, the program it's self would be awesome to have to run down players with. If you could get it to spit the count out on to a spread sheet that an agent could add the rest of the needed info to I think you'd have a million dollar product