To calculate how many players your server can handle, find the amount of RAM your server has, in megabytes (MB). This can be found in your game panel. Divide that amount by 80. That will give you a good estimate of how many players your server can handle. If the number has a decimal, be sure to round down. For instance, if a server has 1024 MB (1GB), it can handle 12 players.
That number is the recommended amount for a Vanilla server. If you are running mods or plugins, you will need to account for those. Players have been seen to use up to 140 MB of RAM each. To calculate the minimum number of slots your server could run, divide the amount of RAM you have by 140. For a 1024 MB server, this would give us 7 player slots.
The closer players are to each other, the less RAM each individual player will use, and therefore the more player slots you can have. If 20% of the players are playing close to each other, we can multiply the number of slots by 1.2 to get a new estimate. Therefore, a 1024 MB server could have 16 slots (rounded up), as long as some of the players are near to each other. We will say this is the maximum number of player slots for the RAM on a server.
This is a list of minimum, recommended, and maximum slots per server:
256 MB - 1 Min. - 3 Rec. - 4 Max.
512 MB - 3 Min. - 6 Rec. - 8 Max.
768 MB - 5 Min. - 9 Rec. - 12 Max.
1024 MB (1 GB) - 7 Min. - 12 Rec. - 16 Max.
2048 MB (2 GB) - 14 Min. - 25 Rec. - 31 Max.
3072 MB (3 GB) - 21 Min. - 38 Rec. - 47 Max.
4096 MB (4 GB) - 29 Min. - 51 Rec. - 62 Max.
For another way to calculate the recommended amount, visit http://canihostaminecraftserver.com, enter 10 for both upload and download speed, and then the amount of RAM you have. This number is not entirely accurate, as it assumes you are hosting the server from your house, but it is easy to use.
As always, if you need help, please don't hesitate to open a support ticket. Be sure to include as much information as possible, including configuration and log files, if needed.