So you wanna be a tabletop game designer, eh?
Cool. I'm no pro but I can share what I've learned in the last 5 years as a hobby designer.
Before I get into it, I’m going to address something that’s on the minds of all new designers: Kickstarter. If you’re planning on using it - don’t. Or at least not yet. Start out operating on the assumption that you’ll license your design to a publisher. Why? This way you’ll avoid doing silly things like investing money on artwork or marketing early on. I also personally think licensing your game is a better deal for new designers. I’m more than happy to discuss the merits of both, but that’s another discussion for another day. I'm not saying you shouldn't self-publish, I'm just saying that you shouldn't make that decision until you've got a publishable game and you've fully researched the investment it will take to do so - and if you're just getting started, you're still far away from that point.
With that out of the way, here are some tips for designing your first game:
Start with a hook. It could be a thematic or mechanical hook, just as long as it answers the question: “why would somebody play this over the millions of other games out there?”. Find it early and it should help guide your design decisions.
Fail Faster. Get your ideas out of your head and into a testable prototype as soon as possible (and watch this video: https://www.youtube.com/watch?v=rDjrOaoHz9s)
MVP: Minimum Viable Prototype. You’ve probably got a million awesome ideas for your game, but include them all and you’ll end up with a convoluted mess - if you ever finish it at all. Include only the absolutely necessary mechanics in your early versions and make sure they work. It’s going to be a lot easier to add stuff than to remove stuff down the road.
Rapid iteration. Make your early prototypes quick and dirty. Your game is going to go through many, many, many changes. Make sure you can make new versions quickly and throw them out afterward without remorse.
Play similar games. There are other games out there similar to whatever you are designing. Find those games, play them, and learn from them. Avoid the pitfalls they ran into and steal the ideas that worked. Also, give careful thought to how you’re going to make your game stand apart from them.
Four phase playtesting. Who do you playtest with? It usually depends on how far along you are.
Phase 1: Solo playtesting. Your early versions WILL suck. Please don’t inflict them on other humans.
Phase 2: Play with friends. Friends are willing to play your crappy looking prototypes, but will often provide biased feedback.
Phase 3: Play with strangers. Strangers are unfamiliar with you and your game and will give you less biased feedback. They usually appreciate a nicer prototype, though, and may need to be plied with beer or candy.
Phase 4: Blind playtests. Only relevant at the very, very end of your design process. This is when you give somebody the rules and the game to see if they can figure it out without your assistance.
Interact with other designers. You can learn a lot just from talking to other designers, playing their games, and letting them play yours. There are plenty of design communities on the internet and in meatspace - find them and become a part of them.
https://thenounproject.com/ - Free icons for prototypes.
https://www.thegamecrafter.com/ - Order nice prototypes (for Phase 3+ playtesting or publisher submissions).
https://tabletopia.com/ - Amazing resource for virtual prototyping and playtesting.
http://unpub.net/ - Find playtesting events near you. Great way to get your games tested and to mingle with other designers.
https://www.reddit.com/r/tabletopgamedesign/ - Nice community for tabletop game designers.