I'm not a ML engineer and only have surface level knowledge of models, but I’ve been wondering, would it be possible to train models in a way to be able to embed a system prompt in a non-textual format?
Ideally, something that’s lightweight (like cheaper than fine-tuning) and also harder to manipulate using regular text prompts?
Text is just one representation. The model uses tensors (think multi-layered matrices in the context of ML that handle language and you're not too far off; in laymans terms, 'hard maths') to actually represent the inputs when they're being processed.
But, I suspect, if the model is able to handle language at all, you'll always be able to get a representation of the prompt out in a text form -- even if that's a projection that collapses a lot of dimensions of the tensor and so loses fidelity.
> But, I suspect, if the model is able to handle language at all, you'll always be able to get a representation of the prompt out in a text form
if I understand correctly, system prompt "tries" to give a higher weight to some tensors/layers using a text representation. (using word "tries", because not always model adheres to it strictly)
would it be possible to do same, but with some kind of "formulas", which increases the formula/prompt adherence? (if you can share keywords for me to search and read relevant papers, that would be also a great help)
Iirc there have been studies that were able to do some amount of LLM debugging and identify certain weights corresponding to certain behaviors.
Seems like it could be possible to lobotomize the ability to express certain things without destroying other value (like a human split brain). Of course possible doesn’t mean tractable.
Ideally, something that’s lightweight (like cheaper than fine-tuning) and also harder to manipulate using regular text prompts?