am trying to store a bytea as a value in hstore
You cannot safely store bytea in hstore without some kind of encoding, because hstore is stored as text in the current text encoding, but bytea has no text encoding and may also contain null bytes that are not legal in text strings.
So you need to encode the bytea as hex, base64, or some other form that makes it into valid text in the current encoding.
The simplest way is to cast it to text, but I recommend instead using encode and decode to/from base64. This encoding is more compact than the text or hex encodings used for bytea's text representation in PostgreSQL, and it's also independent of the value of the bytea_output setting.
e.g.
UPDATE users
SET store = store
|| hstore('key1', encode( pgp_pub_encrypt('testval',dearmor('xxxx'))), 'base64')
WHERE ... ;