The Idea
I was meeting my grandfather, who is completely deaf, in the summer. I'd never had a true conversation with him before, and I wanted to find a way to communicate with him.
As I started searching for apps, every one I found had a problem. They didn't identify unknown speakers (people who haven't been enrolled in the app). I started to realize that there was no app that identified unknown speakers, so I decided to build my own. The app should identify unknown speakers, and also identify known speakers (people who have been enrolled in the app).
How It Works
The app uses Apple's speech recognition framework to transcribe spoken words into text. Then, it uses FluidAudio's framework to create speaker embeddings, which are unique representations of each speaker's voice. These embeddings are then compared against other embeddings in a multi-dimensional space to identify the speaker. If the speaker has been enrolled in the app, then the app has stored speaker embeddings to help identify the known speaker. If it recognizes a known speaker, it will display their name next to their speech bubble, otherwise it it will be displayed as "Speaker 1", "Speaker 2", etc.
Demo
Challenges
There were many challenges I faced while building this project. One of the very first challenges was that I didn't know how to code in Swift. Another challenge was getting the unknown speaker identification to work. It was a lot of trial and error.
What I Learned
I learned a lot while building this project. I learned a little bit of Swift, but more importantly, the process of building, then testing, then debugging, and then repeating that process again.
When I went to show my grandpa the app, I was able to have a real conversation with him for the first time.
Future Plans
I plan to continue working on this app, adding new features and improving the existing ones.