RainMan 2.0 - Blackjack Robot

A Raspberry Pi-powered robot that plays Blackjack and counts cards. The perfect Blackjack player!

Similar projects worth following
I'm creating the perfect Blackjack player! This Raspberry Pi-powered robot will identify the cards in its hand and the dealer's upcard, and use a Hit or Stand lookup table to determine the best play to make. It will also be able to count cards and implement card counting strategies like the "Fab 4" and the "Illustrious 18". So far, I've made a card detector program using Python-OpenCV that runs on a Raspberry Pi with attached PiCamera to detect and identify playing cards on a poker table.

For the next phase of the project, I'm working on getting the card detector to work on overlapping cards (which is how they are dealt in blackjack). I'm attempting to use machine learning to achieve this: training an object detection Convolutional Neural Network to recognize cards. Then, I'll work on implementing a card counting mechanism and getting the program to play through a hand of blackjack.

Here's a video showing how the card detector works!

And here's a link to the source code for the card detector:

List of Project Tasks to Complete:


  • Get card detector program to work with overlapping cards. (Going to try to implement convolutional neural network for detecting the cards)
  • Add card counting capability to card detector
  • Add ability to determine correct play (hit, stand, split, double down) for a given blackjack hand + dealer upcard
  • Add ability to play through a hand of blackjack


  • Create an initial 3D model of what the physical blackjack bot will look like - use best guesses on what the dimensions will be
  • Create servo-driven arm that taps the card table twice for a "hit", or waves side to side for a "stand". Figure out best way to wire servo up to Raspberry Pi
  • Integrate a small LCD display that shows what cards the bot is seeing

This is just an initial list! As the project develops, I will undoubtedly find more things I need to do.

  • Machine learning object detector isn't as effective as I hoped it would be!

    Evan Juras02/20/2018 at 15:36 3 comments

    Over the past couple months, I've been tinkering with machine learning to try and train an object detection neural network that can detect playing cards. The OpenCV algorithm I used (described in this video) works great at detecting cards, but it doesn't work if the cards are overlapping even the slightest bit. Unfortunately, blackjack is always dealt with the cards overlapping. If my blackjack robot is going to work, it needs to be able to count cards even when they're overlapping.

    Someone told me that I might be able to train an object detection classifier (a type of neural network) to recognize the cards even if they're partially obscured or overlapping. Object detection classifiers recognize patterns to identify objects, so they only need to see a portion of the object to detect it. I decided to use Google's TensorFlow machine learning framework to train a playing card detection classifier.

    I've spent lots of time learning about machine learning (enough to make a tutorial showing how to train your own) and I've taken hundreds of pictures of playing cards to feed to the training API. Unfortunately, it's starting to seem like machine learning isn't going to be the silver bullet I hoped it would be. The trained playing card detector just doesn't work very well. 

    For the most part, it works great when it has a clear view of the cards (so does my OpenCV algorithm):

    And it even works if the cards are overlapping:

    But if I deal some actual blackjack hands in front of the camera, the way it would be done in a casino, it isn't able to detect all the cards. The cards are too overlapped for it to see all the cards.

    Right now, it isn't trained well enough to distinguish that there are two cards in each hand. It only sees the top card. It's possible that if I fed the trainer hundreds more clearly labeled pictures of overlapping cards, it might be able to see both the cards. I've already given it 367 training pictures, but maybe it will work better if I give it 1,000 more. Also, it is still a little inaccurate and sometimes incorrectly identifies cards. More training data might help with this, too.

    However, there are some other problems. My classifier is trained off Google's Faster RCNN Inception model, which takes lots of processing power. I want to run my blackjack robot on a Raspberry Pi, which has limited processing power. I tried using the lower-power MobileNet-SSD model, but it doesn't work very well at identifying individual cards. I need to find a way to keep the processing requirements low while still having good accuracy.

    Also, I only have the detector trained to recognize card ranks nine, ten, jack, queen, king, and ace. It will take lots more training pictures to get it to work with every card rank. I have a sneaking suspicion that it won't work as well on the lower numbers (four is very similar to five, etc.).

    However, I'm still going to try! My next step is to train the detector to recognize ALL cards, not just nine through ace. Then, I'll run it on a Raspberry Pi and see if it's still able to detect cards fast enough, and make a YouTube video about it.

    I'm still trying to think of how I might be able to get it to work with the cards overlapping. I think the solution will involve a combination of machine learning and some image processing with OpenCV.  Please let me know if you have any ideas!

View project log

Enjoy this project?



siddhantssangal wrote 03/19/2019 at 11:34 point

Is it possible to perform the detection on the computer and use a raspberry pi as a controller for motors? I am trying to sort the playing cards into 4 baskets of the 4 suits using a simple 2 motor mechanism. one that will rotate and stop at 4 different angles and another that will just turn 180 degrees to drop the playing card.

  Are you sure? yes | no

geaxgx wrote 06/07/2018 at 04:48 point

You should persist on the path of machine learning. It works very well, at least on a gpu. For the cards overlapping, just focus the training on the card corners. And you are right about the training data: you need lots more. But the chance with the detection of cards problem  (vs detection of dogs for instance) is that you can artificially generate yourself your data. That's what I did to train a YOLO neural net. You can find more info on this video I've made :

  Are you sure? yes | no

Evan Juras wrote 08/15/2018 at 05:26 point

Thanks!! I wish I had seen this comment when you posted it two months ago. Your card detector works amazingly well! I'll see if I can re-create your wonderful work.

  Are you sure? yes | no

murray.macdonald wrote 05/20/2018 at 01:50 point

To solve the occlusion accuracy problem have you considered training and recognizing just card corners, their left sides, or just the text rather than the entire card?

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates