In general, I experienced lucid dreams very rarely and It was when I was anxious and stressed, leading me to not go completely unconscious as I slept: there was a part of me that was "aware"/conscious and another part of the brain conjuring up the dream. Thereby, I found myself in the situation of myself being aware of being inside of the dream and consciously living it.
While it was very cool, it was a signal that I was very stressed and unable to let go completely. I don't even remember those nights being restful since, as I said, I would not sleep completely
Anyway, to achieve what you said you wanted, I would suggest trying the "Inclusive Meditation" I wrote about in this post: https://ancient-forums.com/threads/exclusive-and-inclusive-meditation-differences.302637/
With this technique, you can differentiate your "pure awareness" from other parts of the brain. You can see as an aware spectator: automatic thoughts, visions and I even travelled consciously with my mind. I even use the technique to astral communicate with the Gods (They usually speak through visions you have to decipher, in my experienced but sometimes full-on Telepathy) and deceased loved ones. This technique is similar to lucid dreams without messing up your nightly sleep.