It is possible to build just about anything given the time and resources. If you want something like a raspberry pi, you could start with that products schematic as a starting point and begin to make the modification that you desire. With a working schematic you can design a PCB and have it manufactured. Then comes assembly. Given these high level steps, all you need are the knowledge and skills (possibly some equipment) and you can do it.
Figure out where you skills/knowledge is weak and these are the things that you can learn about.
Just a FYI, it will cost you much more than the cost of a raspberry pi to build one.
Its not an SBC the same as the Raspberry Pi, but its a fun activity to do is build your own Arduino "uno"
Its called Shrimp It.
You probably already know this, but the Uno is just a break out of the ATMEGA328, with some nicely made programming and power circuits added to the PCB. Its a fun activity to build those your self so you know how they work, this method can also be used for using the Uno design in your own projects.
Phil's proposal is something that can be done with some skills and dedication.
Building a SBC like the Pi is an advanced topic on theory (what's involved in creating a computer?), hardware (all CPUs, GPUs, peripherals, drivers, glue logic must work together, PCB with timing and signal integrity constraints), software (needs a compatible Linux ?), lab (do you own a fast enough logic analyzer and oscilloscope to troubleshoot, can you solder fine pitch surface mount) and purchasing (can a private person purchase the CPU and other components?).
