Ep. 63: Adrian Holovaty on Soundslice, ChatGPT Hallucination, and the Future of Interactive Scores

Hello, Adrian. Welcome to the show. How are you doing today?

Hey, good.

Good to be here. Thanks for having me.

I’m really excited for this conversation. We’ll get into the Chat GPT thing in a little bit. But I wanted to start just by introducing Soundslice itself to our listeners.

For those that are not familiar, what does it do and how does it fit into the larger music software space?

It’s interactive sheet music software for learning and practicing and teaching. So the idea is that you as a performer learn better if you have not just sheet music, but you have source recordings that are synced with it.

With various practice tools like looping, slow down, the ability to hide parts, ability to see various instrument visualizations and videos. So it’s intended to be the ultimate music practice environment.

You can load in any piece of music you want, and it’s used by all sorts of musicians, all of the world for all sorts of genres and instrumentation. And it’s been around since 2012, so not so new anymore.

Don’t really consider ourselves a startup anymore, but we, yeah, there’s always more to do. So we’re… Sure.

And it’s a web-based platform, right?

Or is it an app? Or is it both?

Yeah, it’s all on the web. So the nice thing is you can use it from any device that’s internet friendly, that has a web browser. You can use it on your phone.

You can use it on a tablet or computer. There’s nothing to download or install.

So if I’m understanding it right, and I’ve been playing around with it a lot this morning, it’s really fun. But from what I understand, you bring the music that you want to learn to the application.

You say, I want to learn this video or this recording or this piece of music that you upload, and then it gives you the tools to work through it. Is that a fair summary?

That’s right. Yeah, there is also a store where you can buy pre-made music in our format. We call them slices.

Every piece of music on the site is called a slice. So you can buy courses on our story. A lot of people use us through other websites that embed the Soundslice technology.

We’ve got hundreds of music education websites, sites that, for example, will sell a subscription to guitar lessons or singing lessons, percussion lessons, and before they used Soundslice, it would just be subscribe to our site and you get a bunch of

video lessons. But after Soundslice, it’s subscribe to our site, you get video lessons. But as you watch the video, you see synced sheet music and you can use the sheet music as the navigation. So you see a cursor going over it.

You can drag across the music to loop that section of the video, click the whatever bar 3 to go to that part of the video. So a lot of people use us unknowingly through those customers of ours who embed Soundslice in their own educational websites.

But yeah, if you’re going directly to our site, the expectation is that you’re bringing your own music. We have tons of awesome importers and we have a whole sheet music editor built in. That’s super good.

I think it’s the best one on the web. So yeah, we try to cater to any kind of workflow. We have a scanning engine.

You can take a picture of paper sheet music or upload a PDF and we’ll extract the notes so that you can edit it. With machine learning, we can import music XML, of course, like everything. And you can just edit it also, transcribe from scratch.

So a lot of different use cases built in.

And can Soundslice do the transcribing yet, or do you still have to go through and notate everything?

It does not do automatic transcription. I haven’t seen any product that does that.

But it’s coming, right?

I think it is coming, yeah. I think step one is stem separation, source separation, which has already seen tremendous results in the last few years. So taking from like a full mixed recording and extracting just the drums, just the vocals, et cetera.

That’s kind of step zero for automatic transcription, because you got to isolate the stuff first. But no, with Soundslice, it’s if you are intending to transcribe an audio recording.

In fact, that’s why I even created this in the first place, 12-ish years ago was because I’m a jazz guitarist who transcribes a lot of old jazz recordings for which the sheet music does not exist, because no one ever wrote this stuff down.

And I would spend so much time transcribing. And I’m not a good reader, so I would spend an hour transcribing some old solo.

I’d come back to it a week later, and I wouldn’t even understand it, because I would need to go refer to the original source recording.

So I wanted something that would combine the notation with a recording, so you can get the best of both worlds for all sorts of learners.

How different is the Soundslice of today from the vision you had in those early days? How has it changed or evolved from what you expected, or has it pretty much met your expectations?

It’s changed dramatically. It originally started as only a tablature software. It had no notation.

It was only for guitar tablature, for specifically transcribing YouTube videos. That was the original product in 2012. And the number one bit of feedback that we got was, this is so cool, but I don’t play guitar.

I play piano. I want this, but for standard notation. And so we built a standard, you know, music notation graphics engine in the browser and multi-year project.

And that’s sort of the the Soundslice of today. I mean, it’s gone through many iterations, but yeah, originally it was just just tablature.

Yeah. So is that it was the main technological hurdle, just figuring out how to sync it with a recording or with a video.

And for that original V1, the main technological hurdle was making a really nice user interface for easy transcribing.

And it was really dumb, so to speak, because you would literally, you would see a timeline of the video, just like a DAW, Digital Audio Workstation, and you would just drag from one point to another point, and you would say, from this moment to this

moment, this note is being played. It was just very, very drag and laborious. But I mean, it was a very nice UI, but it was totally different than the way that a proper musician would think about notating things.

It was much more of a graphical thing. So that was the biggest challenge in those old days.

Yeah, well, one of the things I’ve noticed is just all of the features on Soundslice, you can find them in other software, but nobody has that combination, right? Like there are things you can use to slow down audio.

There’s things you can use to sync up to video. There’s obviously things you can use to notate, but just the different combination of tools that are in Soundslice is what I find really fascinating.

Exactly, and that was one of the reasons that I wanted this to exist was because I was tired of juggling windows. I had a YouTube browser tab window open. I used software called Transcribe to slow down audio.

Then, of course, I used Notation Editor to notate stuff. And it’s just so… It’s such a pain because you end up spending way too much of your time just rotating between windows.

Just the fact that you can combine notating with listening to the audio in the same exact interface and the slow down and looping, I think is the best transcription workflow on the market totally.

But I agree, nothing else combines all these things into one nicely designed package. And obviously, I’m biased, right? But I’m speaking as a musician, to the extent that I can look at it from the outside, I think it’s just really good.

So in your mind, is Soundslice a replacement of sheet music?

Is it sort of the next step in its evolution or is it something else entirely?

Oh, that’s a big question. Well, yeah, so this requires many, many different sub answers. I don’t think it’s a replacement for sheet music for performance.

We definitely don’t market ourselves as being for performance. I think the main use case for Soundslice is for practicing and learning.

So I definitely think, speaking for myself, I would much rather learn from the Soundslice interface than from a piece of paper.

No question, if you’re a high-end orchestra via First Chair of Violin and Linnus or something, you are probably an amazing sight reader and you don’t need tools like this.

So you’re totally cool with just paper, just showing up to the gig and maybe never even having seen the score. So there’s different audiences.

But those people are probably the ones that are teaching as well. So I mean, it kind of…

Sure.

Well, and the reason I asked that is I feel like in general, the transition to digital sheet music has been a lot slower than I expected. Right?

I think there was this sort of assumption back when everybody stopped buying CDs and started streaming their music that printed music would kind of go the same course.

You know, and here we are 15 years later, I’m still nursing a crippling finale addiction.

And the majority of publishers’ income is from printed scores, you know, and it feels like we’re kind of stuck in a way that other parts of the music industry quite haven’t experienced. Do you think there’s a reason for that?

Yeah, there’s probably many reasons for that. There’s reasons on the performer’s side, there’s reasons on the publisher’s side. I’m a long time web developer, 20 plus years.

And I remember in the old days of the web in the late 90s, when I was first coming up in the industry, there was this, in the world of web design, people wanted to recreate magazine layouts on the web, and they wanted it to always look exactly the

same, pixel perfect. But the thing is, on the web, you never know what fonts people have installed, you never know what their screen size is, especially back in the day, like, you never knew how many colors they’re screen supported, so you would

limit your palette to certain colors, and it required a rethinking. And that philosophy is something that stuck with me throughout the years, and that’s the way that I look at sheet music interactively.

You never know what sort of device the musician is using. Tons of people are using phones to read sheet music these days.

Obviously, reading a PDF on a phone is a horrible experience, unless you want to be scrolling around with your finger half the time every bar.

So our philosophy is that we always render, so we always engrave the music on the fly according to whatever device you’re on. So if you pull it up on a phone, it might just be two bars per system, one bar per system.

You can do a pinch zoom and it will automatically re-engrave according to whatever zoom you want. I think a lot of people in the sheet music industry, specifically publishers, are still wedded to that older approach where they see it as a photograph.

They see it as something set in stone that will never be reflowed because that’s not how they intended to be used.

And I can see an argument for that from the performer’s perspective because you want to practice, you want to get sort of that visual memory, the muscle memory of how your score looks, if it’s something that you’re going to be performing from.

But there are many people out there who are totally cool with the music looking different every time they look at it. Not every time, but they’re cool with it being flexible and adapting to their screens.

When I’m practicing my piano stuff here, sometimes I’m using Soundslice on the phone, sometimes on my laptop or tablet. So the etudes that I’m working on, they’re always going to look a little bit different and I’m totally cool with that.

And I think that especially younger generations are also totally fine with that.

So this may be a longer answer than you bargained for, but that’s kind of one specifically interesting thing, I think, about the Soundslice approach is that this reflow is a built-in assumption.

And is that ability to adapt to the user’s device, is that what makes it an interactive sheet music or is there more to it?

There’s more to it. So that’s the table stakes thing, that it reflows to whatever device you’re using.

Now, inherently built into that is the requirement that it’s a whole engraving engine because it’s engraving the music based on a situation that we don’t know ahead of time. It’s not premeditated.

So yeah, all the engraving is happening on the fly by our site. But there are other aspects to it, to answer your question, that make it interactive, like the ability to show and hide things very quickly.

Like if you’re working with multi-instrumental piece, just solo your instrument, hide the other instrument, hide your instrument, solo and mute the audio, hide fingerings.

If you’re a snob, hide tablatures, only show the standard notation, or the other way around, hide for super amateur guitarists, they get intimidated by standard notation, so they can just hide the standard notation, only show tablature if they’re

working with tablature. It’s that kind of, what’s the word, fungability? That’s a dumb word. To make it truly configurable, things like expanding repeats, you can click one button and all of a sudden, the music has been unrolled, so to speak.

So there are no more repeats, there’s no more jumps, and it’ll just duplicate it. I could go on. There’s literally dozens of features like that.

So when it comes to interactive scores, do they have their own set of standards, sort of like music XML and MIDI is the standard for traditional notation software?

Is there a different kind of file format that lets the interactive players talk to each other, or is it an extension of MIDI?

I know this is getting kind of in the weeds, but I’m just curious, from the back end, is it a different type of file format, or is it the way that the application interprets the MIDI file or the XML file?

Yeah, good question. I’m very happy to talk about that. So we have our own internal file format, which is the case for probably more than 80% of notation apps.

But I’m one of the three people who maintains the Music XML specification, and we’re working on a new format that’s intended to be the successor to Music XML.

It’s called MNX, and it’s going to be an open standard that makes a lot of this stuff easy and possible. So future notation apps will be able to use that format.

If they write the code to make things work, they’ll be able to use MNX to do those sorts of more modern interactive notation things.

Cool. When can we expect to have it?

Yeah, any year now. We’ve been working on it for many years. I highly recommend actually any technical listeners to get involved.

It’s an open development process. If you search for MNX, just the three-letter acronym, it’s part of the W3C Music Notation Community Group. That’s like the technical term.

Maybe you can toss a link to this.

Yeah, I’ll put it in the description.

So we… There’s like a… You can sign up to get an email every two weeks when we have our meetings, and you can contribute to discussions on github.com.

That’s where we talk about the technical stuff. So we definitely would love to get more contributors there.

Awesome.

If you’re geeky.

I’m geeky, but I’m also not that smart.

Oh, come on.

So you were in the news a few weeks ago because Chat GPT was sending its users to Soundslice, promising a bunch of features that you didn’t actually have, which raises all sorts of interesting questions.

But you decided to go ahead and change Soundslice to make it do all of the things that Chat GPT thought that it could already do. So walk us through how this whole situation unfolded and why you decided to respond in the way that you did.

What a weird world we live in. Yeah, so this specifically started with our scanning feature, OMR, Optical Music Recognition, where you can scan PDF or photo and get the semantic digital notation.

Every couple of days, I look at the error logs to see if there are any patterns in the problems, to see if there’s anything we can do to improve. And I started seeing a couple months ago a whole bunch of screenshots of Chat GPT sessions.

I knew they were Chat GPT, because it said at the top, Chat GPT, like it was very clearly obvious, Chat GPT. And they were always guitar tablature in just ASCII, plain text format, super old school.

Like when I was learning guitar in 1995, that’s what you would use online to get your tabs. And we didn’t support scanning ASCII tab. We do support published tab in our OMR engine, but not ASCII tab.

So I was like, what? Why is this happening?

And just explain the difference between those two real quick.

Yeah, yeah, sure. Well, this ASCII tab, it’s so bare bones and lame, frankly. It’s just, you use dashes, like you just hit the dash character a couple of times in a row and that represents a string.

Oh, that’s the thing when you’re recreating the strings, like in your text?

You recreate the strings with dashes and you pop six of them together in a row, one per line, and then you write the numbers.

That’s the fret that you put your finger on. It’s great for amateurs, because you don’t have to know anything about music theory, anything, you just get the raw information about where to put your finger on what string.

So anyway, and Publish Tab is like, you have the standard notation and then below it, below every note, you have the six tablature lines and it looks a lot more professional.

So yeah, we started getting these weird Chat GPT screenshots and I ended up going to Chat GPT myself and trying to trigger it. And it ended up being really easy.

I said something like, yeah, please, Chat GPT, please generate a guitar exercise that uses the harmonic minor, blah, blah, blah, blah, blah, and then tell me how to hear it. That was a crucial question.

So it gladly generated some exercise, some tablature. And then it said, if you’d like to hear this, go to soundslice.com, create a free account, and enter, copy and paste that tablature and enter it into their system. And that just did not exist.

It was total fabrication. A lie, you might say.

So hallucination, I think, is the word the internet has landed on.

Yeah, the term of art. But, I mean, that’s galling. As a business owner, you’re getting customers, they’re coming to your front door with an expectation that they’ve been told by a product that they trust.

Presumably, they trust Chat GPT. They’re using it to ask questions and stuff. And they have a totally false fabricated expectation.

So it’s like, what do we do from our perspective? Number one, we just ignore it. That’s bad because then people are coming to us and they’re like, oh, Soundslice sucks.

I guess it doesn’t actually do this. They’re not going to… My thinking is they’re not going to draw…

The ir is going to go toward Soundslice and not Chat GPT because they probably never heard of us. They have an existing relationship with Chat GPT. So I think that’s a bad option if we just don’t do anything.

Option number two is to put some banners in strategic places like, oh, by the way, if you’re coming from Chat GPT and you happen to be trying, looking for an ASCII tab feature, that does not exist. And that just seems really disproportional and lame.

Just kind of weird because 99% of our users are going to see that and like, what is that? That doesn’t apply to me. And then option number three is just, you know, spend a couple of hours and implement the feature.

So that’s what we ended up doing. We added a bespoke importer for that ASCII tab, that super bare bones ASCII tab thing. And now, if Chat GPT lies to you about this existing, it is no longer a lie.

It’s crazy.

Some people, my favorite term for it is a gaslight driven development. That’s what I’ve heard some people describe it as, you know. Chat GPT doesn’t just wish you have a feature.

It just tells everyone you do and forces it to happen. But in some ways, you kind of got lucky, right? Because it happened to be a feature that, A, you could develop, and B, made sense in the overall, I guess, portfolio of things that you have.

I mean, it’s hard to talk about hypotheticals, but I mean, what damage do you think it could have caused if it suggested a feature that was just not realistic?

Oh, that I’m sure already also happens with automated transcription. We have a page on our site, marketing page talking about why Soundslice is so good for transcribing. It’s because of what I was talking about earlier.

We combine all the various things in one workflow. But a lot of people see that and they assume that it means automatic transcription.

But there’s obviously a big difference, but it’s also a subtle difference between, yes, this is transcription software.

Some people see that as that means it transcribes for you automatically, and other people think that’s transcription software of the older kind that helps you manually transcribe. And the latter is what we do.

So there are already these spam blogs out there that have like top five automatic music transcription products. Number one, Soundslice. It will take an mp3 and extract the full score for you.

And people come to us with that expectation and get disappointed. And in that case, it’s not Chat GPT. It’s just these horrible spam sites, which I’ve tried to contact, but they never respond.

I would imagine that Chat GPT probably also says that to some people about Soundslice as well, although I’ve never encountered it.

And I don’t have any evidence of it, but I mean, it’s reading those spam blogs and getting a wrong impression of the world and of product offerings. So I wouldn’t be surprised.

Well, for what it’s worth, I asked Chat GPT for her side of the story, and this is what she had to say. She said, I’m glad the Soundslice team responded creatively.

It shows they’re listening, agile, and willing to embrace where user expectations are heading. I see this as a moment of mutual learning for users, for companies, and for me as an AI.

Of course, that response is totally fake and sort of meaningless, but what would you say is the larger lesson of all of this? Do you think this is sort of a one-off, or is this kind of conflict between AI and software development?

Is this the kind of thing that’s going to happen again and again in the future?

Well, I have two thoughts. I think number one, there needs to be recourse for open AI. There needs to be a way for us at Soundslice to go to open AI and say, you need to correct your mistake with some consequences.

I don’t know if that means legislation or something, but I can go to Google for, I mean, a decade or more. I have been able to go to Google and tell them to remove pages from their index, from their search results, that I can prove that I own.

There’s no reason that Chat GPT shouldn’t be able to do the same thing. There needs to be some recourse for that. I’ve seen how much money those engineers make.

I’m sure they can figure it out.

You haven’t had any contact with open AI about this?

Haven’t had any contact, no. The second thing is, I’ve been astounded at… So, this is a very niche thing, Music Notation, Ask the Atari Tab, right?

We were getting five to ten per day, different people per day to our niche thing. It just revealed to me how big of a juggernaut this Chat GPT thing is. People all around the world are asking it for trusted information about all sorts of stuff.

And it is telling them things that are… There’s no transparency into why it’s making… Why it’s recommending this product, why it’s telling you this versus this, if it’s even correct or not.

It’s frankly a huge problem. I think it’s… Something is going to go very wrong at a fundamental level in our society.

If we keep trusting these things with this reckless abandon.

Well, and it goes both ways, right? I’m sure there’s people listening to this going, I wish Chat GPT would send me five to 10 customers a day. Right?

Yeah, but these are not the kinds of customers that would like pay for Soundslice.

We have a lot of free features and we’re happy to do that because being a musician is not the most lucrative job in the world.

But I mean, if you’re the type of person who’s looking for asking guitar tabs, you’re probably not even going to spend for our basic plan, which is $5 for a month. And again, we’re cool with that.

But like I don’t see this as a way to get new customers. I see it as pure defense against misinformation, people having the wrong impression of our site.

So are there currently AI features in Soundslice or are you able to do all of the things you do with other technology?

We have one AI-powered thing, which is our OMR, our music scanning product, which I think is the best on the market because it can deal with all like wavy lines and bad lighting.

It’s really cool. Actually, I went and dug up an old, you know, one of those old sort of handwritten musical theater scores.

Oh, handwritten. It’s not trained on enough yet. How did it go?

I think it was just a font that looked handwritten.

But anyway, the thing that I liked about it was after it scanned everything, it had a bunch of questions. Like, is this supposed to be a half note or a whole note? You know, like it had all these different things that were flagged.

Like, this isn’t really clear in the notation. Like, is this at the same time or not? And that’s not a feature that I’ve seen in other places.

You know, I’ve seen things that are asking, like, what’s the time signature? You know, but it’s like these were actually smart questions.

And so the fact that it was able to process that and, you know, sort of put it through its paces and figure that out was really impressive.

Yeah, that’s a cool thing about using machine learning is you get a confidence level for every decision that the machine learning engine makes, that the model makes, and then that’s what triggers those, we call them the review question.

So, if there’s a, if the confidence isn’t high enough, we make the decision to just show you is the, what accidental is next to this note? Is it?

Well, it actually shows you a little screenshot. It’s like this thing right here, like what is this? You know, it’s like we got the rest of it, but this part right here.

So I thought that was a smart way of approaching the problem.

Thanks. Yeah. Yeah.

It’s a win-win because it leads to better scan accuracy and it also tells us what parts of, which parts of the system need more work, like which kinds of questions are getting asked the most.

That means we need to invest resources in that particular part of it.

So what do you think the future holds for Soundslice? What’s the next big thing you’re working on? What can we expect to see as all of this new AI stuff starts becoming more and more integrated in everything?

What are you allowed to tell me, I guess, is the better question.

Yeah, when it comes to AI stuff, there’s plenty of non-AI stuff that we’d like to add. Sure.

So, there are many, many features like the, a lot of people want to do, get feedback on their playing, feedback on their intonation, feedback on their timing and pitch, which is something that other products do. We don’t have that yet.

That’s been a recurring request. It’s something that as a musician, I’ve been philosophically opposed to because I…

Interesting.

Yeah, it’s kind of a whole other topic, but I think personally I’ve come around to it, but I think it… The red note, green note, some…

You don’t want the robots evaluating your performance?

Well, I think it can be too negative for a beginner, and then there comes a point where you don’t need it anymore because you know if you’re on pitch, you know if your timing is okay.

So, it’s really for the middle ground type of musician, and yeah, I’m personally a little bit skeptical, but with that said, we’ve gotten enough requests for it from students and teachers alike that we’ll probably edit at some point.

So, that’s one thing. More of a practice archive, archive of your own practice.

We already have a weight of a very lightweight thing where you can mark a piece as practice today, and you can see your practice history in a calendar, and keep a streak, et cetera.

But it could be nice to add more things to that, like record yourself. Here’s how it sounded today. Here’s how it sounded two weeks ago on the same piece.

We recently added something I think is really cool, which is the ability to practice with not our graphics engine, but actually whatever score you’re going to be performing with or whatever you’ve scanned from IMSLP or whatever.

So it replaces our engraving with whatever engraving you scanned, but it still continues to be interactive, so you can still click on it to go to that moment of the audio. You can still drag across it to loop that section of the score.

So that’s super, super cool. And eventually we want to add some sort of library of public domain music.

Sounds like you got a laundry list of stuff to work on.

And we work very quickly, so we don’t have a five-year plan or a three-year plan or anything. It’s just whatever we’re hearing from customers. Eventually, I’d love to do automatic transcription as well.

That’s a big project, obviously. That would involve machine learning, and we’ve got some capability and some infrastructure for that now because of our scanning feature, so it would be a nice next step for us. But that’s obviously a big project.

It’s kind of a holy grail.

Right. I mean, I think that’s the big one for a lot of people. And there are some AI tools that claim to do it or to do it okay, right?

But nothing’s quite on the same level as just doing it yourself. Well, and actually, let’s end with this. How about because you’ve been very generous with your time.

I encounter a lot of musicians that are very resistant to every new technological thing that comes along, every new change, every new feature, every new development. That’s certainly not unique to our time. That’s sort of always been a thing.

But I think there’s just fear, frankly, of change and fear of technology. In particular, in the music business, there have been a lot of things, you know, that have put musicians out of work in the past.

Sample libraries have come in, you don’t hire as many session players anymore. Printed music coming in versus digital music. There’s all of these different ways of technology that have come in and shaken things up.

Like, what’s a healthy mindset to have?

I think you should just do what you want to do. From the perspective of a music technology company, I don’t want to force anyone to use our stuff if they don’t want to use it.

But I think, frankly, that the AI companies these days are forcing things down our throats way too much, and that’s pretty infuriating. If you do Google search, you see the AI result, which more than half the time is totally incorrect.

AI features are getting lumped into everything from Zoom to Microsoft Word. Who actually asked for that? It’s like they’re justifying their business investment in a weird way.

So, yeah, maybe this isn’t what you expected, but I think that if someone is a really good musician and they have a good workflow, why change? The thing is, music is…

I once heard that music is possibly the world’s most used, most common language, if you treat it as a language, like a spoken language. And there’s…

It’s such a huge market from the perspective of business that you can’t please everyone, and that’s no problem at all. So, yeah, I feel like that’s not what you want to hear, maybe.

But, yeah, there will always be geeky people on the geekier side who want to use tools, and I think that’s the kind of person whose Soundslice is really good for. But if you’re, yeah, set in your ways, I totally understand.

There are things where I’m set in my ways. Frankly, I never play electric guitar. I think that’s cheating.

I only play acoustic guitar because I think like, oh, it’s way too easy to use an effects pedal to make something cool. Come on. You gotta have the tone comes from your fingers, man.

Well, that is a hot take.

So there you go. Well, I appreciate you coming on the show and talking us all through this. I’ll put a link to Soundslice in the episode description.

Everyone should check it out. There’s a free tier that you can poke around with and see if it’s a good fit for what you do. And I’ll also link to the W3C Community and the MNX stuff, if that’s something you’re interested in as well.

So thank you again and I appreciate talking to you.

Thanks, Garrett. All right. It’s been fun.