I’m finding this to be the biggest hurdle. In terms of improvisation and composing, a lot of what is holding us back (this could possibly apply to repertoire as well) is that we’re not strong with our fretboard knowledge. Ear training might play into this as well. But one of the biggest things is (again like the chord building analogy I used earlier) that we just don’t know all the shapes and figures to really crank out non stop lines. There’s a few ways to improve this.
I think the biggest vocabulary builder is simply to sit there and noodle for hours, but be adventurous, try to sound jazzy, try to sound funky, try to sound like Marty Friedman and then Scott Henderson and then Joe Pass.
Second I would stop learning scales and start learning chords and arpeggios, the latter especially. this is not only for the purpose of finding all the notes, but these two train your ears to find guide tones and melodic lines much easier. Arpeggios in particular are a bit more acrobatic in the way they can be played on the fingerboard and provide some great glue for your lines. Again, if you think about modal arpeggios, you should have a perfect selection of notes to choose from.
At that, I’d like to add that the reasoning behind a good lot of the traditional alternate picking exercises is not just right hand work, but also to build left hand vocabulary and to break up some usual shapes into workable patterns.
Lastly…here we go with the reading again…one of the best ways to expand your vocabulary is to read. Pick up whatever book you decide to use and read through it, slowly if you have to and just surrender and allow it to teach you something, allow it to take you somewhere sonically. I highly recommend “Jazz Rock Patterns” by David Chesky as a great resource to improve your fretboard knowledge, sight reading, and your ears. The other thing with improving your sight reading is that trains you to quickly call to mind the notes and shapes on the fretboard.