Terraria server hosting: vanilla, TShock or tModLoader?
Terraria is one of the lightest games to host, and also one where worlds last for years. That combination makes your choice of server software more important than the hardware.
Vanilla, TShock or tModLoader
The standard server works fine for a group of friends but offers no administration. If you want permissions, commands, grief protection or a ban list, you use TShock — the de facto standard for public Terraria servers.
If you want mods instead, that is tModLoader. Note that the two do not combine. Decide which direction you want up front, because switching means rebuilding your setup.
Your world is a single file
A Terraria world is one .wld file. That makes backups simple — and losing it painful, because there is no middle ground. Keep a copy aside periodically, especially if you have been building the same world for months.
How many players
Terraria is aimed at smaller groups; eight to sixteen players is typical. More is technically possible, but the world gets busy and chaotic quickly.
How much RAM
2 GB is ample for vanilla or TShock with a normal group. Heavy modpacks through tModLoader ask more: budget 4 GB.