If you can give ChatGPT the transcript and it can say “yes that’s about ____”, then that means it’s certainly possible for them to do the same. I would expect that anything trained specifically for that should only get better from there, although obviously they’re not going to throw ChatGPT-sized compute at it.
although obviously they’re not going to throw ChatGPT-sized compute at it.
I’m not entirely sure whether and what more fundamental distinctions between embeddings and LLMs may exist, but smaller LLMs really struggle with comprehension if things are phrased in an unexpected way, and embeddings use comparatively very few resources. Maybe a circumvention training tool could work like this: a writing game where the goal is to produce text about a topic such that the embedding fails to associate it with that topic, but a more powerful LLM succeeds (the idea being that maybe a human would be able to tell also). The biggest advantage these systems have is probably just the way people do not get direct feedback about how their work is being interpreted.
If you can give ChatGPT the transcript and it can say “yes that’s about ____”, then that means it’s certainly possible for them to do the same. I would expect that anything trained specifically for that should only get better from there, although obviously they’re not going to throw ChatGPT-sized compute at it.
I’m not entirely sure whether and what more fundamental distinctions between embeddings and LLMs may exist, but smaller LLMs really struggle with comprehension if things are phrased in an unexpected way, and embeddings use comparatively very few resources. Maybe a circumvention training tool could work like this: a writing game where the goal is to produce text about a topic such that the embedding fails to associate it with that topic, but a more powerful LLM succeeds (the idea being that maybe a human would be able to tell also). The biggest advantage these systems have is probably just the way people do not get direct feedback about how their work is being interpreted.