• By - ev-


Full game on lichess : [https://lichess.org/0KqgCkzl](https://lichess.org/0KqgCkzl)Guess what match? :)Thinking of building this as a mobile app - would anyone be interested in using something like this? Edit 2021-01-11: The beta is almost ready! If you want to use the app, signup at https://knightvision.io/


I'm officially a chess nerd. I recognized it without having to look it up. >!The Evergreen Game: Adolf Anderssen vs Jean Dufresne!< https://www.youtube.com/watch?v=uqzxnz6d7JM


Well done!


I'm curious, at what point did you realize what game it was?


Not him, but I realised it when white played nf6+, because I only remember what the final position looked like


Actually about the same for me. I remember first watching the game thinking white was dead when the black Q captured the knight on f3 with the threat if mate in 1. Then that insane forced checkmate line just stuck with me.


Man I'm MORE than interested ! This could be so cool to easily record the games we play with friends ! Very cool, thanks for your work !


Thank you for the kind words! I’ll let you know when there is an app to download! :)


please ping me when its ready also. will this be android or ios?


Will do! Hopefully both!


Hey man, if r/chess mods delete your post with the app announcement (they can say it's self-promotion), can you post it in your profile so that we could find it? I'm gonna monitor your profile from time to time, because I really want to try this app!


I definitely will do that. Thank you so much!


Ping me please!


Please ping me too


Ping me to this is amazing!


Will do, thank you!


I’d like to get pinged for this too please!


Awesome. Ping me please (Y)


I can see this being useful for IRL streamers and that kind of thing but I feel like it would be way more interesting to have at chess tournaments to broadcast them. Like if you're playing at a lower board on a chess mat, just set up a webcam of phone camera and you have a full record of the game. Awesome project!


I'd love a ping as well


Amazing! Any rough eta so that I can have the remind me bot ping me at ~the right time. In the meantime RemindMe! 1 month "chess vision app"


You should ask [chessvision.ai](https://chessvision.ai) to partner with you! You might be able to do something with them or sell your design to them.


YES! I’m a shitty, shitty chess player. But my five year old is amazing (well, as far as five year olds go—his OTB rating is 571. He’s been playing 5 months). Anyway, ever see a little kid try to notate? Handwriting is terrible. Writing letters in reverse (b,d). Slow. This would be a great way for younger players to keep track of their OTB games. I could see coaches recommending such an app to their students (and their students’ parents). Even older players who may struggle with notation—dysgraphia, tremors—could use this to quickly keep track of their OTB games. I can see a lot of usefulness for this.


Great feedback and that's amazing to hear about your son. I have a 4.5 year old that refuses to learn the rules(His own rules include the knight being thrown into the ground from time to time). All the best wishes for your son! I started playing when I was 15 / can't imagine how much better I would have been if I started that young!


To call a spade a spade, the kid is profoundly gifted. It’s been hard dealing with him because he’s so bored all the time, but chess has been his siren’s song. I hope he sticks with it. He even loses graciously (95% of the time lol) because he’s intrigued by how he lost. It’s delightful. Just don’t be that parent that pushes their kid to hard. I see so many parents at the chess center yelling at their 5-6 year olds for blunders and such. I’ve been asked what I do to incentivize my son and make him a good player. People are crazy. I’m sure you’re not doing that, but yeah. And sometimes kids fall in love with things you’re interested in when they’re a touch older! My 8 year old just started coding video games. That’s also awesome. 4.5 is still so young. Have you seen that story time chess set? Or No stress chess? No stress chess is fun because even once you know how to play, having to move based on what a card tells you to do can make for some interesting games! Good luck with the app development! I look forward to seeing it come out!


I could have written this post. This would be amazing for my kid too!


I'm very interested in this. Would be cool to play faster time controls without having to log. But yeah, would have to be a mobile app. What language did you use for your current program?


Hi, I'm one of the developers behind this. It's all written in a relatively new language called Julia which I'm a big fan of.


Will you make it open source?


Wow, never heard of that, but looks really cool. At first glance, it reminds me a lot of Go (not the syntax, so much), but maybe better for data applications.


I hate that Julia seems to be python++++++, has all the nice features without any of the drawbacks *cough*speed*cough*


Nice Julia is really taking off! We are using it at work (engineering related)


Yes, my chess club would love this


Very impressive. if you used TF it shouldn't be too complicated to transfer the model to a mobile app. keep us updated!


I would pay ($3-5) for a tool like this.


so did you make the program, or are you planning to make it in the future?


The video is made from the program. It works great, but it is not a mobile phone app yet.


Very cool. Do you have a desktop Version, or a github where we could track the progress?


how accurate is it? does it make mistakes occasionally?


I feel like it must: surely it at least gets confused if it sees something that is not a chess piece and possibly may be incompatible with certain chess piece designs?


Depends on how many images it was trained on, and how varied they were.


Very interested!


Would absolutely love something like that!


subscribe! feels like the best games my friends and I play are OTB, having a tool like this would be amazing! using pure memory to remember interesting endgame positions to analyse is tough (for a bunch of 1300-1600s at lesst)


I have been looking for exactly this for months! Most of the OTB games I play nobody else feels like documenting, but I feel like I would learn so much from them!


I can totally relate! I'll let you know when there is an app available!


This would make a really useful mobile app! Let us know if you release the source code, I'd be curious to try to make it work with React Native or something.


This would be awesome. As others mentioned, it would be great with friends. When I play OTB with a friend I like to record the moves on the Lichess mobile app or something so I can study it later, so this would be a life saver


I would love it!


> Thinking of building this as a mobile app - would anyone be interested in using something like this? Absolutely yes. I think you'd have a lot of people interested in this.


Man ping me please when the app is testable! I dont know match about AI but Im a Game Developer so if you need something just ask


Will do, thank you!


Can it record the time used per move too?


Yes in theory. It recognizes when a move is made, so could record the time as well.


Super interested


Definitely. A friend and I came up with exactly the same idea but never implemented it.


You are using a very nice chess set in the video, with quite standard piece shapes. I suppose you have considered the fact that app users will have a wide range of piece types.


Yes of course. The one in the video is a classic board with Staunton pieces. We have a set of other boards that are quite different and not as “nice”, and it works well for those as well. I believe we can get the program very stable for most boards, but of course this is a early release!


Dude this is incredible. Could you technically stream from a phone to a computer to let it do the processing?


Thank you!:) Yes the video streams from the phone to a desktop-app right now, but it could also be made directly on the phone.


Super impressive stuff, huge practical potential for larger scale adoption. Awesome work! Imagine everything from better tournament broadcasting, to people going to their first otb tournament to record their own game and be able to run analysis right after without inputting it themselves. Streamers with a better board view during irl actions… seriously cool project. I guarantee chesscom would love to have their hands on this software.


Something tells me that having a phone recording a tournament game would somehow be against the rules. But outside of tournaments, I agree that it has great real-world potential!


The arbiter can authorize it, but agree that it's better to use a webcam though.


This has a lot of potential for organizers that cannot afford magnetic boards to stream the moves. Would be great to make chess games more accessible in general.


DGT gonna show up to your house


Please post git link


Well, if this is real and actually works, you just made DGT boards obsolete.


The problem is reliability, computer vision is always great until it isnt, DGT boards work, and work (semi) reliably. This software is great, but in order to replace DGT boards it would require a massive amount of testing.


Cost plays a factor too though. DGT boards are freakishly expensive. Sure for top tournaments it's not necessarily unreasonable, but for others were cost might be more of a factor a webcam and free software is way more appealing.


How much do those boards they use in the top tournaments cost?


> computer vision is always great until it isnt Though, as an AI scientist who always rags on ML, this is actually a pretty great usage because the problem space is pretty small, so if there are bugs it wouldn't be hard to fix.


Computer vision has come a loooong way in the last 5 years. We have cars driving themselves with just the input of cameras. There is no reason to believe this wouldn't work consistently if the model is trained right.


This is not going to be an issue for this application. It is definitely simple that this will work (with less error than a human writing down the moves).


It can be nearly perfect if made to start from the starting position and assume that only one piece is being moved at a time. Unless you're playing hyperbullet or something. Recognizing any position from the get-go is more error prone.


It works! Need to make it into something more user friendly / accessible but working on it! :)


Can you share the rough version as-is for people who want to explore?


I'm a computer engineering student and I certainly would like to know what references you used for learning to do this.


Hi! For a good theoretical introduction to machine learning I really like "the elements of statistical learning". There is a lot of online resources for learning about computer vision, machine learning and neural networks. We do a combination of a couple of different approaches. Neural networks for piece recognition, some more traditional computer vision for finding the board and tracking things in video. Best way to learn is to start coding on something you are interested in!


How well does the network perform with chess sets of different color schemes or sets with unique piece designs?


Very unique piece designs is definitely a problem. We're using a couple of different methods all at once to avoid scenarios where the network would have a hard time.


could you add a caliber option (for an app) where you can analize your own pieces and relate them to standar chess . Is just an idea, i dont know how could It work. (sorry for grammar, english is my second language) Excelent work, love it.


Yes I think that calibrating should be doable down the line. Nice idea!


could have "tagged" tournament pieces (officially approved for the app) that offer guaranteed detection


If players put the piece down real hard, you can add a "!" As an annotation


Brb building dgt board with force sensors


That's great, good job! I would definitely download it if it was an Android app. Can it process a recorded video from a game as well? I think that would be a very useful feature. Is it a private project or you're planning on releasing the code as open source?


Thank you! Processing an already recorded video is a cool feature, will definitely think about that. The plan is to build an android app as the next step!


Just hold the video up to your phone camera. Processing time is a little high though ;)


Yeah, my main thought was that I might want to record the game on a gopro on a tripod and then process the games later at home on my PC. So processing a recording would be nice, even if the input formats are limited.


How good is the ai at recognizing promotions?


That's awesome, are you planning on releasing it on android?


Yes that is what I'm thinking if there are others who would want to use it!:) It is a desktop app/julia-program right now.


Of course people want to use it, I remember playing some games at my university and couldn't study it after the game because my notation was garbage, couldn't figure out what did I play, with an app like that I could just play freely while my phone is recording and check the png after the game.


That’s exactly the same for me. So many 5 0 games played that I never can replay :)


Use it? With the right marketing, features, and a clean interface, you could throw it up on an app store for a couple of bucks and make a LOT of money.


I would even pay for it! If you put it on GitHub, I would be very interested to participate :)


please make we know when you release the program. This is great.


Thank you! I definitely will!


Random question... does your reddit name stand for Expected Value Minus?


Haha yes negative expected value. If you’d look at my chess games you’d realize that I win 99% of my games using luck


So glad that you choose the Evan's gambit


This is so cool and so genius and has a *ton* of practical value. Would take any live, irl chess content (educational, entertainment, or otherwise) to the next level. Amazing work, OP!!!!!!


Thank you for the kind words!


That's dooope. What libraries did you use?


Mainly the Julia packages Flux.jl, Chess.jl, ImageTransformations.jl and Mux.jl


Is there an app which reads in chess notation and creates for example a file which you can read into lichess?


Damnit. I literally had an idea to make this yesterday.


Very good! How can I subscribe to know when app is available, please?


Thanks! I’ll let you know when it’s ready!


This guy created this years ago : https://www.youtube.com/user/spiderchess32 http://webcamchess.fr/


This is really impressive! Does it work with various chess sets (different boards and pieces)?


Thank you! Yes it does. However really odd boards/pieces (like a starwars chess set) will be problematic.


That's fantastic, can't wait for the app :)


When I start playing chess, I remember that I searched for a program like this, and I didn't believe it didn't exist. But I left it aside because I thought I should have a reason to. looks like I was wrong. To give you an idea on how this is helpful: Here in Brazil, 2021 Brazilian championship was played, and unbelievably there were no dgts boards. In other words, the biggest championship in the country, it didn't have the most important thing for a tournament of this size. The streamers however wanted to broadcast the tournament and managed to get 2 cell phone cameras placed and streamed online, where the streamers had to pass what they were seeing onto a lichess board. Such a program would be very useful. I hope you stay on it. Congrats for the work.


That is a great story, thank you. Will definitely stay on it, I will let you know as soon as an app is available!


As long as the program could take a recording as an input, this would be great. I think that some clubs/organizations might not like having a phone recording the game, but they would be okay with any other camera on a tripod. Could you make it take an mp4, avi, etc. as an input?


Thank you for the input! That would also work. Started out using a web camera before switching to a phone.


One other cool idea: this could eventually morph into an alternative to investing in expensive DGT boards.


Very cool! That was the idea in the beginning. Played a lot of tournaments where only the top 3 boards were streamed (because of expensive DGT boards). Would be cool to play at a tournament where every game was live recorded :)


Dude, don't open source it or give it away for free. This interesting app could grant you a wealthy life.


This is amazing! Would be super useful for blitz/rapid OTB tournaments where players don’t take notation. Or for short time situations in classical


Sweet :D


I run my school's chess club and it would be quality to show chess tournament finals on the projector screen in real time


Very interested in this! Desktop or android app would be fantastic.


Oh nice .. So no more commentators complaining that they didn't get the PGN from a competitor website ..


Awesome work! Any details on the implementation? Just curious :)


Hi, I'm one of the developers behind this. It's all written in a relatively new language called Julia. We use a mix of traditional machine learning and neutral networks, mostly with the Flux.jl library.


Very interesting! Would you mind explaining how this works? How does the software recognize the pieces?


Hi, I'm one of the developers behind this. We use a mix of neural networks and other learning methods to piece together a prediction of the game


This is a fancy way of saying, we start with the starting position and then compare frames to see which square no longer has a piece on it. Then we can infer what move just happened. We have no idea what piece is where unless it got to where it by a series of moves from the starting position. If you knock over a couple pieces and put them back in slightly different locations, all bets are off. Source: am a developer.


So where does the ML take place? If it’s as simple as seeing that the pawn was on e2 a couple of frames ago and now its on e4 what is the need for ML? My guess is the ML is necessary to filter out and not process frames where the player is moving a piece?


Here's two stills laid on top of each other from the video. It doesn't take HAL to see what just happened and it can be done by simply checking for how much of the board is visible at each coordinate. Some people would call this "machine learning" but I'd call it a bunch of if statements and twiddling with how "filled" a square has to be to determine a piece is on it. I'm guessing nudging the camera slightly also causes the detection to break. [https://i.imgur.com/Wr95JTY.png](https://i.imgur.com/Wr95JTY.png)


That's cool! Have you like taken a million different pictures of chess pieces and trained the AI to recognize them or something?


You didn't really answer the question


Superb! Cant wait for the app


Thank you!


This is awesome. Does it only work for specific boards and camera angles?


We have trained on a few different boards, and the goal is that it should work on most traditional boards, but I wouldn't hold my breath for more obscure pieces like "Lord Of The Rings" sets. It seems to be quite robust to the viewing angle.


Thanks! No it works as long as the camera can see the board. Weird piece layouts can be a problem, want to make it more resilient later on.


Amazing! Everybody will want this!


Thank you!


this is incredible!


Something to sell to streamers? And Ytubers


How does it respond to nonsensical moves, like when [that street hustler tried to take 2 pieces in a single move](https://youtu.be/U5vnpOp0U_g), or [players knocking pieces all over the place in a time scramble](https://youtu.be/vNIpKPmJzYY)


It tries to recover from positions that doesn’t make sense. I’m really clumsy so we got a lot of training data with positions where I accidentally knock pieces over :)


Could be neat if you could have it report on positions it thought were confusing and maybe even preserve video of the previous X seconds, etc. so you could correct it manually if needed.


Makes sense. Will look into that!


This is a amazing! I hope to see it on android :) Remindme! 1 year


I will be messaging you in 1 year on [**2022-11-05 19:02:01 UTC**](http://www.wolframalpha.com/input/?i=2022-11-05%2019:02:01%20UTC%20To%20Local%20Time) to remind you of [**this link**](https://www.reddit.com/r/chess/comments/qneak9/made_a_program_that_in_real_time_creates_a_pgn/hjg9hdn/?context=3) [**1 OTHERS CLICKED THIS LINK**](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5Bhttps%3A%2F%2Fwww.reddit.com%2Fr%2Fchess%2Fcomments%2Fqneak9%2Fmade_a_program_that_in_real_time_creates_a_pgn%2Fhjg9hdn%2F%5D%0A%0ARemindMe%21%202022-11-05%2019%3A02%3A01%20UTC) to send a PM to also be reminded and to reduce spam. ^(Parent commenter can ) [^(delete this message to hide from others.)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Delete%20Comment&message=Delete%21%20qneak9) ***** |[^(Info)](https://www.reddit.com/r/RemindMeBot/comments/e1bko7/remindmebot_info_v21/)|[^(Custom)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5BLink%20or%20message%20inside%20square%20brackets%5D%0A%0ARemindMe%21%20Time%20period%20here)|[^(Your Reminders)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=List%20Of%20Reminders&message=MyReminders%21)|[^(Feedback)](https://www.reddit.com/message/compose/?to=Watchful1&subject=RemindMeBot%20Feedback)| |-|-|-|-|


Thank you:) hopefully sooner than that!


Can't wait!


Is it Open Source?


Is it actually able to distinguish the pieces, or does it get the information needed just from the board itself?


It can distinguish pieces from each other. Then it tracks states-board positions as well. We play a lot of 960 which works very well with that approach :)




That is awesome! Please share your progress and updates.


Remarkable stuff man I'm surprised this doesn't exist already!! If you can get it to mobile, that's huge! Changes live chess forever


RemindMe! 2 months


This is huge


Isn't it a computer graphic implementation, what kind of AI are you referring to? Since my studies I hate computer graphics, your implementation is awesome!




You might want to go into hiding. Some big companies are looking to hire hitmen about right now.


Add clock recognition (which side’s clock is ticking and times for each) and you have a full DGT replacement!


When would it make sense to check up on you? End of January?


Hopefully! I’ll post an update as soon as there is a beta available!


This is incredible. I'm going to keep an eye on your profile for updates


bro this is fucking amazing.


Chess24 could do with this to save the poor presenters from having to keep up with blitz games!


Man, this is genius. I imagine analysis, book reading etc would be amazing with such a tool.


What did you write this in? Is the source code available to look at? It sounds great!


Really cool project! Do you plan to release it under a free software license?


Dude this is fantastic! Please ping us when ready to try it out! Happy to help you beta test it if it helps!


Thank you! I definitely will let you know when it’s ready for testing!


Would love to have this app


Easy to do…


This is really great job. Waiting for the app. Also, just a thought: this can probably be modified to be used as a tool where people can play in realtime with physical chess sets remotely with their friends. That would really be awesome as it will give people a chance to get the real feel while enjoying all of the digital benefits.


Cool idea! We talked about building that, might be possible(without building a board), but not 100% sure.


Love this!!




Wow! This is a game changer. Really looking forward to this project.


Absolutely incredible work


This post has been parodied on r/AnarchyChess. Relevant r/AnarchyChess posts: [Made a device that in real time tracks your game from a physical board using your eyes and hands. Requires nothing more than a piece of pampers and a pencil.](https://www.reddit.com/r/AnarchyChess/comments/qnsiqo/made_a_device_that_in_real_time_tracks_your_game/) by abcdefgother [^(fmhall)](https://www.reddit.com/user/fmhall) ^| [^(github)](https://github.com/fmhall/relevant-post-bot)


DGT in shambles


This is awesome!! I would love to see where this goes. Remindme! 6 months


This is awesome dude. Good work.


If you need a backend developer at any point let me know. Id be happy to help.


and, i'm going to guess, your exact board and piece set.


Reset the counter


Counter is reset! There's been no reset for: 22 hours, 25 minutes and 39 seconds Congratulations on your first reset /u/Cornel-Westside! [Last reset](https://www.reddit.com/r/chess/comments/qmw1gs/still_cant_believe_i_found_this_idea_in_a_blitz/hje0wlw/) was on 2021-11-05 07:34:02 by /u/CautiousRice


oh wow thats very impressive good job man


I'm interested in your program. Anything you can share or at least document? Thanks


Is it possible to recognize multiple boards from a eide stance? For a or my chessclub that would be awesome and make the expensive DGT and horrible working DGT obsolete.


Man, now I have no reason to buy a DGT board.


Can it work off stills? There's a good game on the other sub to test it out with :-D


Very cool. Would be great to market to chess clubs