It's called a Cue mix, headphone mix or Foldback.
It's not simple to set up and not worth the effort when recording yourself. Worth it for recording bands though.
You need an interface with several outputs, (8) then a mixer with several line inputs (6 needed), and a TRS cable for each.
Plug output 3 of the interface into input 1 of the mixer.
Pan it left
plug output 4 of the interface into input 2 of the mixer.
Pan it right
etc.
interface output 1-2 is for your monitors in the control room
All your tracks by default go to output 1-2
In the software you make a PRE-FADER send on every track going to the interface outputs 3-8.
Choose groups of tracks to be controlled by each pair of faders.
Following along?
so lets say output 7-8 from the interface (faders 5-6) will carry your FX returns (reverb, delay whatever you need in the cans)
output 5-6 (faders 3-4) is drums and bass
output 3-4 (faders 1-2) is guitars
and shit we're out of options for where the vocals go
I told you it's complicated.