Domain Expansion

I've been watching the season 3 of Jujutsu Kaisen during the break. One of the coolest concepts in the show is the idea of a "Domain Expansion". It's essentially a powerful technique where the user creates a pocket dimension that guarantees their attacks will hit.

In the show, the users iconically perform a specific hand gesture to activate their Domain Expansion. I thought it would be fun to build a computer vision application that could recognize those hand gestures in real time using a webcam, and trigger some kind of on-screen overlay or effect when the gesture is detected. It would be a fun way to combine my curiosity in computer vision with my enjoyment of the show. I told my younger brother about it and he also found the project interesting, so I decided to invite him to collaborate.

The overall stack of this project is as simple as the concept itself. Python for the main application, OpenCV for video processing, and MediaPipe for hand tracking and gesture recognition. MediaPipe's hand tracking provides a lot of useful data points, like the positions of each finger and the angles between them. The main challenge is going to be accurately recognizing the specific hand gestures associated with the Domain Expansion technique.

The project should also be easy to demo and share, so I'm keeping that in mind as I build it out.

Implementing Satoru Gojo's Domain Expansion, "Unlimited Void", was quite simple because the gesture itself is simple. Only one hand needed and your middle finger to be wrapped behind your pointer. It was easy to set up a MediaPipe pipeline to recognize that gesture, and I ended up finishing it quickly while my brother was sleeping.

Then I went onto implementing Sukuna's Domain Expansion, "Malevolent Shrine", which was a lot more challenging. The gesture is more complex, requiring both hands and a specific way to position your fingers. MediaPipe's hand tracking can struggle with accurately recognizing that gesture, since theres 10 fingers trying to be tracked simultaneously.

At first, I was stumped, I had been playing around with different approaches to try to get it working, but nothing seemed to be accurate enough. I then set it aside, leaving it for the next day.

So today, I sat down with my brother and we brainstormed some solutions together. We noticed that the main issue was that MediaPipe was struggling to track all 10 fingers accurately, which made it hard to reliably detect the specific configuration needed for Sukuna's Domain Expansion. But what if we didn't need to track all 10 fingers? We could just focus on the pinky, ring and middle fingers, since those are the ones that are most critical for the gesture.

So we did, we set up a new MediaPipe pipeline that only tracked those three fingers, and we focused our gesture recognition logic on the positions of those fingers relative to each other. We made sure that we took into account more factors like fingertip distances and trying to find a triangle formed by the gesture. Soon enough, that pull request was merged, and we had a working implementation of Sukuna's Domain Expansion as well.

We have implemented 6 domain expansions so far:

• ❖Satoru Gojo's "Infinite Void"
• ❖Ryomen Sukuna's "Malevolent Shrine"
• ❖Mahito's "Self-Embodiment of Perfection"
• ❖Yuji Itadori's Unnamed Domain Expansion
• ❖Kinji Hakari "Idle Death Gamble"
• ❖Yuta Okkotsu's "Authentic Mutual Love"

Of course, there are many more domain expansions in the series, and we plan to implement as many as we can. But now, the foundation is there and the architecture is set up. The app detects the specific hand gestures for each domain expansion and displays the text. But it feels incomplete, we want to add some kind of visual effect or overlay that appears when the gesture is detected, to make it more immersive and fun to use.

But it has been a fun project to work on, and it's been a great way to learn more about computer vision and gesture recognition.